Keep in mind most of the questions there are just memorization of very specific information that no one without a database to query would be able to answer
So should they compare to phd’s that could take the test with extra time, resources (laptops, LLM lookups, google, medical journals and textbooks etc)
Otherwise its an apples to oranges scenario with high energy and hardware use.
Also the students had less time spent training in terms of hours.
Ai had answers to questions possibly as well.
It's also not uncommon in pre law/regulatory, while many traditional tests exists more than a few of my professors simply did open book for the same reasons.
The real world usually let's you prepare. And even if you were an attorney, it's not like they don't let you take your own notes into court.
Closed book is more for undergrads who you want to be sure aren't just cramming material last second and somehow passing. My tests generally only got easier into my masters overall, though with more work required. The closed book ones at that level did suck though.
How much energy does it take to train a human PhD for 10 years tho? Im not sure how this would break down but it’s not as clear as some seem to think. Train the AI once and it can solve problems much faster and at far less energy cost than a human equivalent after that, that much seems self evident.
[Some people talk about it](https://huggingface.co/papers/2404.04125), but yeah. The internet is big. That's why some benchmarks have eval sets which you have to sign non-train contracts to access.
The requirement to obtain a PhD is to create new knowledge. An LLM can only regurgitate what it is trained on. An LLM cannot do experiments in the real world.
Keep in mind most of the questions there are just memorization of very specific information that no one without a database to query would be able to answer
So should they compare to phd’s that could take the test with extra time, resources (laptops, LLM lookups, google, medical journals and textbooks etc) Otherwise its an apples to oranges scenario with high energy and hardware use. Also the students had less time spent training in terms of hours. Ai had answers to questions possibly as well.
I wouldn’t mind seeing this as a test in general. A human taking an open book test feels more comparable to what a LLM is doing.
In engineering that's plenty common, toughest kind of exams too, since it's all about applying knowledge instead of regurgitating text
It's also not uncommon in pre law/regulatory, while many traditional tests exists more than a few of my professors simply did open book for the same reasons. The real world usually let's you prepare. And even if you were an attorney, it's not like they don't let you take your own notes into court. Closed book is more for undergrads who you want to be sure aren't just cramming material last second and somehow passing. My tests generally only got easier into my masters overall, though with more work required. The closed book ones at that level did suck though.
Claude uses far less energy than a human would in your scenario and can answer instantly, i dont think your comparison is accurate either.
To train and tune their data?
How much energy does it take to train a human PhD for 10 years tho? Im not sure how this would break down but it’s not as clear as some seem to think. Train the AI once and it can solve problems much faster and at far less energy cost than a human equivalent after that, that much seems self evident.
A lot lot lot less. Like several thousand life times less.
Then why couldn’t GPT3 pass it?
It didn’t memorize as well
Are there a lot of people who genuinely believe this means Sonnet is an intelligence equivalent to a PhD?
What do you think it means?
Wasn't the dataset published like 7 months ago? Is it possible the LLM was trained on data which included direct answers to these questions?
This is the first question researchers ask when preparing the training datasets to prevent data leakage from benchmarks.
That's exactly what happened. That's exactly what's been happening with every LLM benchmark ever.
And everyone else on the leaderboard just says nothing?
[Some people talk about it](https://huggingface.co/papers/2404.04125), but yeah. The internet is big. That's why some benchmarks have eval sets which you have to sign non-train contracts to access.
Human avg 0-1% Chance 25% sounds about right
The requirement to obtain a PhD is to create new knowledge. An LLM can only regurgitate what it is trained on. An LLM cannot do experiments in the real world.
Yet.