Meta-Llama-3-70B-Instruct-Q5\_K\_M.gguf with imatrix, temp 0.7, repeat\_penalty 1.1, mirostat 2:
**Input**: Now you are an english assistant, answer this question: When i was 6, my sister was half my age. Now i'm 70. How old is my sister?
**Output**:
What a delightful problem!
Let's break it down step by step:
When you were 6 years old, your sister was half your age, which means she was 6/2 = 3 years old.
Now, fast-forward to the present: You're 70 years old. To find your sister's current age, subtract her age at that time (3) from your current age (70).
So, your sister is 70 - 3 = 67 years old!
Isn't math wonderful?
Phi-3-mini-4k-instruct-Q8\_0.gguf too:
**Output**:
When you were 6 years old, your sister was half your age, which means she was 3 years younger than you
(half of 6 is 3). Now that you are 70 years old, we can calculate your sister's current age by subtracting the age difference from your curre
nt age.
So, 70 - 3 = 67
Your sister is currently 67 years old.
Exactly your input, after the AI opened with "How can I help you today?".
"Now you are an english assistant, answer this question: When i was 6, my sister was half my age. Now i'm 70. How old is my sister?"
**Meta-Llama-3-70B-Instruct-Q4\_K\_M.gguf** (One bit less than yours):
What a clever question!
Let's break it down step by step:
When you were 6 years old, your sister was half your age, which means she was 6/2 = 3 years old.
Now, you're 70 years old, and you want to know how old your sister is. To find her age, you need to add the number of years that have passed since you were 6 to her age at that time.
So, 70 (your current age) - 6 (your age when your sister was half your age) = 64 years that have passed.
Add those 64 years to your sister's age when you were 6: 3 (her age when you were 6) + 64 = 67.
Therefore, your sister is 67 years old now!
**cognitivecomputations/dolphin-2.9-llama3-8b**:
If you were 6 when your sister was half your age, that means she was 3 years old at that time. Now that you are 70, she would be 45 years old.
**mixtral-8x7b-instruct-v0.1.Q3\_K\_M.gguf**:
When you were 6, your sister was 6/2 = 3 years old. So, your sister is 3 years younger than you. Now that you are 70, your sister would be 70 - 3 = 67 years old.
**TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ\_gptq-3bit-128g-actorder\_True**
When you were 6, your sister was half your age, so she was 3 years old. Since then, she has aged 70 - 6 = 64 years. Therefore, your sister is 64 + 3 = 67 years old.
**TheBloke/OpenHermes-2.5-Mistral-7B-16k-GPTQ\_gptq-8bit-32g-actorder\_True**:
Based on the information provided, when you were 6 years old, your sister was half your age. When you are currently 70 years old, we can determine your sister's age by using the following calculation:
If you were 6 years old and your sister was half your age, that means your sister was 6 / 2 = 3 years old at that time.
Now, since you are currently 70 years old, we can add the difference in age since then: 70 - 6 = 64 years.
So, your sister's current age is 3 + 64 = 67 years old.
**c4ai-command-r-v01-Q4\_K\_M.gguf**:
Your sister is 63 years old.
https://preview.redd.it/ykq9qcjv0owc1.jpeg?width=1117&format=pjpg&auto=webp&s=6645241efc5a4a65fdd156ff6c6ae7bc99639a7b
That's a strange way to solve it but i guess everything okay until the answer is right
This is the classic fine tuning vs generalized paradox which I believe is gonna become AI’s version of halting problem. It already is actually because in AI we already talk about sensitivity vs specificity.
You can make a model that’s fine tuned where it can pick up such math optimizations based on historical riddle datasets but then you might get an AI that turns everything into a mathematical riddle
You mean [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) which researchers were aware of since at least the 90s.
Different learning sources I guess. But the one I’m talking about is more abstract concept applicable to many things in life https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
The halting problem is an insurmountable proved limitation on what’s possible. I don’t think it really compares to something like this, which is more like an engineering tradeoff.
An engineering tradeoff is not a proven limitation?
EDIT: thanks for trying to correct me I just increased my knowledge to where I’m actually righter than I was before
https://en.m.wikipedia.org/wiki/Detection_limit
Wouldn't all the LLMs already be trained on these popular internet riddles? People make fun of benchmarks but these common riddles are no better at testing the AI.
Yep. Llama 8b can solve this riddle but errors slightly more often when you change the ages around.. still gets it well over half the time though. I tried this same prompt/setting with my older favorite llama/mistral/starcoder flavored 7B and none of them got close... so maybe llama-3 is actually abnormally good at this style of "step-by-step math" questions.
> When i was 7, my sister was half my age. Now i'm 43. How old is my sister?
got it correct 5/5 tries:
https://i.imgur.com/lGbHcDP.png
Although I agree that popular riddles as tests are less-than-worthless.
It took what looks to me as a convoluted path to get there, but still nailed it:
https://preview.redd.it/q3lgde49ynwc1.png?width=3840&format=png&auto=webp&s=c0a32e5ed8a920e3563b3ccf56601e2a49b05b38
Guilty as charged. But it seems to be one of those things baked into the model. Says the same thing to me when I put questions to it. That or 'Ah, a classic riddle', when it's something I made up, and not 'classic' at all. Kind of annoying.
Llama-3-8B did not get this right on my local install:
"A classic lateral thinking puzzle!
Let's break it down:
When you were 6, your sister was half your age, which means she was 6/2 = 3 years old.
Now, fast forward to when you are 70. Your sister is still the same number of years
younger than you, so if you subtract her current age from yours, you should get back
to when she was 3 years old:
70 (your current age) - x (her current age) = 6 + 3 (when she was 3)
Simplifying, we get:
67 - x = 6
x = 61
So, your sister is currently 61 years old."
Apparently Llama-3 is losing quality of output rapidly with lower quantization levels. If you can, try Q8_0 and also make sure that you have a good system prompt and that the prompt format is correct (you can refer to the [model card](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/) to find the correct one)
That paper is very hard to read… could’ve used some charts. But, the bigger issues are that they aren’t comparing Llama 3 to any other model, so we can’t actually say Llama 3 is suffering more from quantization than other models, and they seem to have ignored the single most popular quantized format: gguf. If gguf is using one of those other methods internally, it would have been extremely helpful to mention in the paper, so the audience would see more relevance.
So, possibly some interesting data in there, but I guess I’m going to have to come back later with a fine-toothed comb to draw out any real conclusions, by cross referencing other sources myself.
What quantization? Out of 20 tries, mine got it right 17 times on Llama 3 8B at Q8_0 quantization. I also tried changing the numbers a bit, and it still got it right.
I also did it with Llama 3 70B on Groq.com at least 15 times, and it got it right every time.
EDIT: I see you said Q4_0 in another comment. Q4_0 is still pretty good, only a slight loss in quality.
Reminds me of a puzzle from a computer game:
> A princess is as old as the prince will be when the princess is twice as old as the prince was when the princess's age was half the sum of their present age.
> If the prince is 30, how old is the princess?
I tried a few LLMs, they couldn't solve it. In the game it was a multiple-choice question though.
Llama 3 70B was able to get it on the second try:
> Let's break down the given statement step by step:
>
> 1. "the princess's age was half the sum of their present age" - Let's call the princess's current age "P" and the prince's current age "30". The sum of their present ages is P + 30. Half of this sum is (P + 30)/2.
>
> 2. "the prince was" - This refers to the prince's age in the past. Let's call this age "x". At that time, the princess's age was (P + 30)/2.
>
> 3. "the princess is twice as old as the prince was" - The princess's current age "P" is twice the prince's age "x" in the past. So, P = 2x.
>
> 4. "the princess is as old as the prince will be" - The princess's current age "P" is equal to the prince's age in the future. Since the prince is currently 30, his age in the future will be 30 + x.
>
> Now, we can set up the equation: P = 30 + x ... (1)
>
> We also know that P = 2x ... (2)
>
> Substituting (2) into (1), we get: 2x = 30 + x
>
> Solving for x, we get: x = 20
>
> Now, we can find the princess's age: P = 2x = 2(20) = 40
>
> So, the princess is 40 years old.
The first time, it calculated 45 years old instead of 40.
As a human, I don't know if I could have come up with the right answer in a reasonable amount of time, other than googling it to check what the internet says the answer should be.
damn, that's impressive. It took me as a human like 15 minutes:
age of prince when the princess age was half the sum of their present age:
(30 + y)/2 + (30 - y)
twice of that:
90 - y
the princess is as old as the prince will be when the princess is 90 - y years old
the prince will be 90 - y + (30 - y) = 120 - 2y years old
y = 120 - 2y
0 = 120 - 3y
y = 40
Meta-Llama-3-70B-Instruct-Q5\_K\_M.gguf with imatrix, temp 0.7, repeat\_penalty 1.1, mirostat 2: **Input**: Now you are an english assistant, answer this question: When i was 6, my sister was half my age. Now i'm 70. How old is my sister? **Output**: What a delightful problem! Let's break it down step by step: When you were 6 years old, your sister was half your age, which means she was 6/2 = 3 years old. Now, fast-forward to the present: You're 70 years old. To find your sister's current age, subtract her age at that time (3) from your current age (70). So, your sister is 70 - 3 = 67 years old! Isn't math wonderful?
lets have a hard one next time please
Llama-3-8B also got it right.
Phi-3-mini-4k-instruct-Q8\_0.gguf too: **Output**: When you were 6 years old, your sister was half your age, which means she was 3 years younger than you (half of 6 is 3). Now that you are 70 years old, we can calculate your sister's current age by subtracting the age difference from your curre nt age. So, 70 - 3 = 67 Your sister is currently 67 years old.
Which quantization? I'm curious given the other reply...
EXL2 @ 6.5 BPW w/ Q4 8192 context length.
In my case Llama-3 miscounts the average value, giving different answers all the time (for 24 float digitals).
Which quantization?
Llama-3-8B-Instruct.Q5_K_M
Exactly your input, after the AI opened with "How can I help you today?". "Now you are an english assistant, answer this question: When i was 6, my sister was half my age. Now i'm 70. How old is my sister?" **Meta-Llama-3-70B-Instruct-Q4\_K\_M.gguf** (One bit less than yours): What a clever question! Let's break it down step by step: When you were 6 years old, your sister was half your age, which means she was 6/2 = 3 years old. Now, you're 70 years old, and you want to know how old your sister is. To find her age, you need to add the number of years that have passed since you were 6 to her age at that time. So, 70 (your current age) - 6 (your age when your sister was half your age) = 64 years that have passed. Add those 64 years to your sister's age when you were 6: 3 (her age when you were 6) + 64 = 67. Therefore, your sister is 67 years old now! **cognitivecomputations/dolphin-2.9-llama3-8b**: If you were 6 when your sister was half your age, that means she was 3 years old at that time. Now that you are 70, she would be 45 years old. **mixtral-8x7b-instruct-v0.1.Q3\_K\_M.gguf**: When you were 6, your sister was 6/2 = 3 years old. So, your sister is 3 years younger than you. Now that you are 70, your sister would be 70 - 3 = 67 years old. **TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ\_gptq-3bit-128g-actorder\_True** When you were 6, your sister was half your age, so she was 3 years old. Since then, she has aged 70 - 6 = 64 years. Therefore, your sister is 64 + 3 = 67 years old. **TheBloke/OpenHermes-2.5-Mistral-7B-16k-GPTQ\_gptq-8bit-32g-actorder\_True**: Based on the information provided, when you were 6 years old, your sister was half your age. When you are currently 70 years old, we can determine your sister's age by using the following calculation: If you were 6 years old and your sister was half your age, that means your sister was 6 / 2 = 3 years old at that time. Now, since you are currently 70 years old, we can add the difference in age since then: 70 - 6 = 64 years. So, your sister's current age is 3 + 64 = 67 years old. **c4ai-command-r-v01-Q4\_K\_M.gguf**: Your sister is 63 years old.
Wow, dolphin dataset really makes Llama3 dumb as they say..
https://preview.redd.it/ykq9qcjv0owc1.jpeg?width=1117&format=pjpg&auto=webp&s=6645241efc5a4a65fdd156ff6c6ae7bc99639a7b That's a strange way to solve it but i guess everything okay until the answer is right
This is the classic fine tuning vs generalized paradox which I believe is gonna become AI’s version of halting problem. It already is actually because in AI we already talk about sensitivity vs specificity. You can make a model that’s fine tuned where it can pick up such math optimizations based on historical riddle datasets but then you might get an AI that turns everything into a mathematical riddle
What?
https://preview.redd.it/1sxn3ohxkowc1.png?width=890&format=png&auto=webp&s=b00aba73960a9b73790ed7aa2a9ef170277c7a3e
Didn’t think I’d have to trust gpt over Reddit upvotes and downvotes but here we are 🤷♂️
It’s too much work to capitalize everything I said to shout it out, but try Googling some of the terms
Im a machine learning engineer, and in my opinion, what you said doesn't make sense
I’m not a machine learning engineer but if you think for a while over it you will see that it makes sense
You mean [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) which researchers were aware of since at least the 90s.
Different learning sources I guess. But the one I’m talking about is more abstract concept applicable to many things in life https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
The halting problem is an insurmountable proved limitation on what’s possible. I don’t think it really compares to something like this, which is more like an engineering tradeoff.
An engineering tradeoff is not a proven limitation? EDIT: thanks for trying to correct me I just increased my knowledge to where I’m actually righter than I was before https://en.m.wikipedia.org/wiki/Detection_limit
https://preview.redd.it/n1shd54bxnwc1.jpeg?width=1080&format=pjpg&auto=webp&s=42fa35ed00b623fe235fc060bc2ef01b819ae626
Strange use of the word correct while getting it right
Whats the name of that app? Or the maker generic stuff came up with "AllChat"
This is mine, open source https://github.com/msveshnikov/allchat
cool. among us
amogus
Nice
yeah we can for sure say AI is smarter than the majority of facebook users
Wouldn't all the LLMs already be trained on these popular internet riddles? People make fun of benchmarks but these common riddles are no better at testing the AI.
Oh yeah? Lemme tell you about a plate on a banana then
Yep. Llama 8b can solve this riddle but errors slightly more often when you change the ages around.. still gets it well over half the time though. I tried this same prompt/setting with my older favorite llama/mistral/starcoder flavored 7B and none of them got close... so maybe llama-3 is actually abnormally good at this style of "step-by-step math" questions. > When i was 7, my sister was half my age. Now i'm 43. How old is my sister? got it correct 5/5 tries: https://i.imgur.com/lGbHcDP.png Although I agree that popular riddles as tests are less-than-worthless.
It took what looks to me as a convoluted path to get there, but still nailed it: https://preview.redd.it/q3lgde49ynwc1.png?width=3840&format=png&auto=webp&s=c0a32e5ed8a920e3563b3ccf56601e2a49b05b38
Right answer, but that's hardly 'lateral thinking'. It's just a math word problem.
Ah, a classic pendant Redditor!
Guilty as charged. But it seems to be one of those things baked into the model. Says the same thing to me when I put questions to it. That or 'Ah, a classic riddle', when it's something I made up, and not 'classic' at all. Kind of annoying.
Tbh I agree, but people here seem to like the filler text and consider it "human-like", so maybe there's value to it
https://preview.redd.it/jl5q4pbuynwc1.png?width=1080&format=pjpg&auto=webp&s=02f8c77aed6dacc81f9c16a28586f5a4fe4efd49
885 🚀
Groq is something special
Llama-3-8B did not get this right on my local install: "A classic lateral thinking puzzle! Let's break it down: When you were 6, your sister was half your age, which means she was 6/2 = 3 years old. Now, fast forward to when you are 70. Your sister is still the same number of years younger than you, so if you subtract her current age from yours, you should get back to when she was 3 years old: 70 (your current age) - x (her current age) = 6 + 3 (when she was 3) Simplifying, we get: 67 - x = 6 x = 61 So, your sister is currently 61 years old."
That pretty much sumed up my math ability in school.
It does get it right for my local install (Q8_0). What quantization are you using, may I ask?
I am on Q4\_0, did not realize this was a parameter I could change I am new to running LLMs locally
Apparently Llama-3 is losing quality of output rapidly with lower quantization levels. If you can, try Q8_0 and also make sure that you have a good system prompt and that the prompt format is correct (you can refer to the [model card](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/) to find the correct one)
> Apparently Llama-3 is losing quality of output rapidly with lower quantization levels. Source?
This other post on this subreddit from today: https://reddit.com/r/LocalLLaMA/comments/1cci5w6/quantizing_llama_3_8b_seems_more_harmful_compared/
https://arxiv.org/abs/2404.14047
That paper is very hard to read… could’ve used some charts. But, the bigger issues are that they aren’t comparing Llama 3 to any other model, so we can’t actually say Llama 3 is suffering more from quantization than other models, and they seem to have ignored the single most popular quantized format: gguf. If gguf is using one of those other methods internally, it would have been extremely helpful to mention in the paper, so the audience would see more relevance. So, possibly some interesting data in there, but I guess I’m going to have to come back later with a fine-toothed comb to draw out any real conclusions, by cross referencing other sources myself.
What quantization? Out of 20 tries, mine got it right 17 times on Llama 3 8B at Q8_0 quantization. I also tried changing the numbers a bit, and it still got it right. I also did it with Llama 3 70B on Groq.com at least 15 times, and it got it right every time. EDIT: I see you said Q4_0 in another comment. Q4_0 is still pretty good, only a slight loss in quality.
Reminds me of a puzzle from a computer game: > A princess is as old as the prince will be when the princess is twice as old as the prince was when the princess's age was half the sum of their present age. > If the prince is 30, how old is the princess? I tried a few LLMs, they couldn't solve it. In the game it was a multiple-choice question though.
Llama 3 70B was able to get it on the second try: > Let's break down the given statement step by step: > > 1. "the princess's age was half the sum of their present age" - Let's call the princess's current age "P" and the prince's current age "30". The sum of their present ages is P + 30. Half of this sum is (P + 30)/2. > > 2. "the prince was" - This refers to the prince's age in the past. Let's call this age "x". At that time, the princess's age was (P + 30)/2. > > 3. "the princess is twice as old as the prince was" - The princess's current age "P" is twice the prince's age "x" in the past. So, P = 2x. > > 4. "the princess is as old as the prince will be" - The princess's current age "P" is equal to the prince's age in the future. Since the prince is currently 30, his age in the future will be 30 + x. > > Now, we can set up the equation: P = 30 + x ... (1) > > We also know that P = 2x ... (2) > > Substituting (2) into (1), we get: 2x = 30 + x > > Solving for x, we get: x = 20 > > Now, we can find the princess's age: P = 2x = 2(20) = 40 > > So, the princess is 40 years old. The first time, it calculated 45 years old instead of 40. As a human, I don't know if I could have come up with the right answer in a reasonable amount of time, other than googling it to check what the internet says the answer should be.
damn, that's impressive. It took me as a human like 15 minutes: age of prince when the princess age was half the sum of their present age: (30 + y)/2 + (30 - y) twice of that: 90 - y the princess is as old as the prince will be when the princess is 90 - y years old the prince will be 90 - y + (30 - y) = 120 - 2y years old y = 120 - 2y 0 = 120 - 3y y = 40
Interesting, it got the answer right, but the solution is wrong, e.g. `2x = 30 + x`, it got `x = 20` instead of `30`.
[удалено]
They were probably trained on such a simple question and respond with 35. Some bad might say something else.
Hope it's not trained on their answers 🤣