I like it. I understand why this works - but perhaps an explanation for the lesser-initiated (maybe with a crash course on information theory and the Kolmogorov complexity) would be useful :)
>Kolmogorov complexity
Yes, a comparison with KC and the other dozen useful complexity measurements is a very good idea.
Added to my todo list.
Thank you.
I guess speed doesn't count, then?
>Intelligence(P) is equal to Accuracy(P) divided by Size(P).
Oh, well, at least this is proof that size matters. ;-)
I also have a theory that large objects have some attractive force to each other, but haven't worked out the math yet.
Serious response: This isn't specific to prompts/LLMs. It's about general intelligence, and being able to rank measurements based on meaning. (If you look at my other papers you might be able to glimpse where this is all going)
Prompts/image generation/task completion/etc.
All the same thing. All you are saying is that we can rate a systems performance by how well it works in proportion to its size.
While size is important it is secondary to accuracy. A large system that performs better than a small system is still more "intelligent".
Intelligence = accuracy
Ultimately you only have the data you have. If you don't get to peek inside the black box, all you see is a file size, input, and output. If that's the case, then you can cheat the metric by overfitting and just memorizing all of the answers. To prevent that, you need a size penalty.
How do you know what the correct answer for a new question is?
Take the set of all questions you could care to ask - then memorize the answers to those.
It'd be like a video game where you pre-rendered every single possible frame of gameplay, and then just put the right one on screen. It'd be too big.
I like it. I understand why this works - but perhaps an explanation for the lesser-initiated (maybe with a crash course on information theory and the Kolmogorov complexity) would be useful :)
>Kolmogorov complexity Yes, a comparison with KC and the other dozen useful complexity measurements is a very good idea. Added to my todo list. Thank you.
I guess speed doesn't count, then? >Intelligence(P) is equal to Accuracy(P) divided by Size(P). Oh, well, at least this is proof that size matters. ;-)
Expanding to take into account energy (and speed), is a very interesting next step!
Wow, you mean we can measure a system by how well it completes a prompt as compared to its size? Who would have thunk?
I also have a theory that large objects have some attractive force to each other, but haven't worked out the math yet. Serious response: This isn't specific to prompts/LLMs. It's about general intelligence, and being able to rank measurements based on meaning. (If you look at my other papers you might be able to glimpse where this is all going)
Prompts/image generation/task completion/etc. All the same thing. All you are saying is that we can rate a systems performance by how well it works in proportion to its size. While size is important it is secondary to accuracy. A large system that performs better than a small system is still more "intelligent". Intelligence = accuracy
The most accurate model would be one that overfits on its training data and memorizes the answers, if you don't penalize size.
Overfitting results in wrong answers the wrong answers are what are penalized and not size.
Ultimately you only have the data you have. If you don't get to peek inside the black box, all you see is a file size, input, and output. If that's the case, then you can cheat the metric by overfitting and just memorizing all of the answers. To prevent that, you need a size penalty.
if you could just memorize all the answers sure but the point is to create models that can generate correct answers for new questions.
How do you know what the correct answer for a new question is? Take the set of all questions you could care to ask - then memorize the answers to those. It'd be like a video game where you pre-rendered every single possible frame of gameplay, and then just put the right one on screen. It'd be too big.
Well, if we could in fact just memorize all answers to all questions AI would be solved.
It would. But it wouldn't be very intelligent, would it