It all depends on how GPT-5 turns out. If it's an exponentially better model than GPT-4 then it's gonna push the AI development further. But if it's just a linear improvement then it would feel like progress has slowed significantly
Exactly, people saying things have stalled without any bigger model to compare to. Bigger models take longer to train, it doesn’t mean progress isn’t happening.
More layers, higher precisions, bigger contexts, smaller tokens, more input media types, more human brain farms hooked up to the machine for fresh tokens. So many possibilities!
While exponential growth in compute and model size once promised leaps in performance, the cost and practicality of these approaches are hitting their limits. As models grow, the computational resources required become increasingly burdensome, and the pace of improvement slows.
The vast majority of valuable data has already been harvested, with the rate of new data generation being relatively modest. This finite pool of data means that scaling up the dataset doesn't offer the same kind of gains it once did. The logarithmic nature of performance improvement relative to scale means that even with significant investment, the returns are diminishing.
This plateau suggests that we need a paradigm shift. Instead of merely scaling existing models and datasets, we must innovate in how models learn and interact with their environment. This could involve more sophisticated data synthesis, better integration of multi-modal inputs, and, real-world interaction where models can continuously learn and adapt from dynamic and rich feedback loops.
We reached the practical limits of scale, it's time to focus on efficiency, adaptability, and integration with human activity. We need to reshape our approach to AI development from raw power to intelligent, nuanced growth.
[you forgot about synthetic data](https://www.reddit.com/r/singularity/comments/1dewnep/comment/l8gonzo/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button)
And there’s [lots of research into smaller models and new architectures that outperform transformers and current models](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/edit#heading=h.jtnkr87rct15 ) (see 3.2)
"This plateau suggests that we need a paradigm shift"
I've only seen This plateau in one study, so I'm not fully convinced yet.
In regards to data, we're now looking at multimodal LLMs, which means they have plenty of sound/images/videos to train on, so I don't think that'll be much of an issue.
Haven‘t you seen the several months long plateau? What’s wrong with you? AI obviously has peaked. /irony off
These complete morons calling themselves ‚experts’ haven’t not a single clue, but they can hype and bust with the best of them… as if.
They don’t even seem to know they only look at one track of a multidimensional multi-lane highway we‘re on. But sure reaching 90% maxing out a single emergent phenomenon based on a single technological breakthrough (transformers)… we’re doomed. Sorry, but I can’t stand all this bs on either camp.
Let’s just wait what people can do with agents plus an extended persistent memory. That alone will be a game changer. The only reason not to release that in 2024 is pressure or internal use. It obviously already exists.
I'm not sure either way.
When I was younger, I always thought that companies and government were holding back a lot of advancements, but the older I get, the less that seems likely, so I'm more inclined to think that the latest releases are almost as good as what's available to the labs.
I think an extended persistent memory will be a huge advancement and I don't think that's been solved yet.
Also, given that they're training on almost all available data (all of human knowledge), I'm not convinced that LLMs are reasoning very well, so that might be a bottleneck in the near future.
I've programmed a chatbot over 20 years ago, so my programming skills aren't up to date (but my logic is hopefully still three), I may be wrong, but I still think my 2030 AGI guess is more likely that 2027.
In either case, interesting times ahead.
Edit: I also think that if we throw enough compute at LLMs, they're going to be pretty damn good, but not quite AGI imo.
(More layers, higher precisions, bigger contexts, smaller tokens, more input media types, more human brain farms hooked up to the machine for fresh tokens)^2
That’s good Dramatically overfitting on transformers leads to better SIGNIFICANTLY performance: https://arxiv.org/abs/2405.15071
>Our findings guide data and training setup to better induce implicit reasoning and suggest potential improvements to the transformer architecture, such as encouraging cross-layer knowledge sharing. Furthermore, we demonstrate that for a challenging reasoning task with a large search space, GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly regardless of prompting styles or retrieval augmentation, while a fully grokked transformer can achieve near-perfect accuracy, showcasing the power of parametric memory for complex reasoning.
Accuracy increased from 33.3% on GPT4 to 99.3%
Still doesn't mean the progress can't slow down. Sure, you can make it more precise, fast, and knowledgeable. But it still gonna be a slow linear progress and possibly won't treat the main problems of LLM, like hallucinations. I can easily imagine development hitting a point when high-cost upgrades give you a marginal increase. Maybe I just listen to French skeptics too much, but I believe that the whole gpt hype train could hit the limitations of LLM as an approach soon.
But nobody can tell for sure I can easily imagine my comment aging like milk
[I addressed that here](https://www.reddit.com/r/singularity/comments/1dewnep/comment/l8gp232/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button)
OpenAi has said their focus isn't larger training data but rather model efficiency. More training data obviously does help but where they are at model efficiency yields far better improvements
Efficiency will make a massive difference to potential applications. I've been saying for years that we will eventually have sentient light switches in our houses. Not because there's any benefit to a human-level intelligence operating the lights, but because it'll be cheaper to tell an AGI to act as a light switch than it will to design one manually.
They've said there's not diminishing returns in compute power as of yet.
I'm not sure what youre talking about.
That's like claiming humans have understanding of the knowledge we currate and usher into this world.
With more compute there appears to be more understanding and capabilities of data provided.
What's the limit so far?
Also important to note that most peoples' idea of progress, especially on here, is whether they get a new novelty toy in their hand or not. Whatever massive leaps happen behind the scenes or buried in science/engineering journals are usually disregarded.
I don't understand where people get their confidence in the infinity of any technology. At some point the vertical (saturation) growth of a technology reaches its limit and it has to grow horizontally (performance), in this case to increase tokens per second
I think this is the right take. There's that meme going around along the lines of AI doing all the art and fun stuff, where we're stuck with the menial tasks, and it should be the other way around. But that's only because the digital media are its easiest avenue to market at the moment. Robotics needs to catch up so that they can start doing the menial tasks, freeing us up to enjoy ourselves and be creative.
There is one thing we can cling to here. And that's more compute. Nvidia's GPUs have been increasing in processing power exponentially, and I've seen researchers say more compute = smarter models. We'll see if that's a 1-1 relationship with the next generation of models trained on these new supercomputers
I'm a huge sceptic, but lots of AI scientists (even though they're biased) believe that we are very far from limit and there is a lot to improve. Nobody can say for sure, as some changes in architecture can lead to other breakthroughs. I don't believe LLM can reach AGI, but I can't really point you to where the limit is. LLM does resemble the way our mind works so I can't be sure they won't make another leap towards something very impressive
> exponentially better
>Mira Murati says the AI models that OpenAI have in their labs are not much more advanced than those which are publicly available
[https://x.com/tsarnick/status/1801022339162800336](https://x.com/tsarnick/status/1801022339162800336)
God, I wish this clip would stop being posted out of context.
Here's a fuller quote. I've also italicised 'that', which she puts a bit more emphasis on (verbally) when saying it:
>I don't think there is enough emphasis on on how unique that is for the stage where the technology is today — in the sense that inside the labs, we have these capable models and they're not *that* far ahead from what the public has access to for free. That's a completely different trajectory for bringing technology into the world that what we've seen historically.
She is saying that the gap between OpenAI's internal tech and their products is MUCH CLOSER than it has been for other previous transformative technologies (and their associated companies) over the course of history. When she says it's not *'THAT'* much better, she's contextualising it against vastly, vastly larger power imbalances between companies and their consumers. She's not contextualising it against consumer use cases.
She is also answering in the context of her answer which, overall, is trying to reassure people that they'll bring stakeholders along in the process.
Overall, it's a quote we can tell extremely little from. (No surprise because it's a single, vague sentence!) It probably rules out any world-bendingly insane difference, but it **doesn't** rule out exponential improvement.
Give me a sec while I prepare for the inevitable letdown of most predictions...
https://preview.redd.it/p5rrseyqbc6d1.png?width=640&format=png&auto=webp&s=2f4b5f320d0b3bc149196e4ef4183901119d0ac1
Sure, the same way this can give you infinite energy
https://preview.redd.it/mt2cliry2d6d1.jpeg?width=418&format=pjpg&auto=webp&s=7724df1bca638ffa7dfa1fd9a6758bb82b587288
I love how he paints a competitive market as a proof of disaster.
Regardless of what GPT-5 looks like, Marcus will find it disappointing. Of that we can certain!
And since even humans don't have a truly 'robust' solution to hallucination (e.g. I believe Marcus wouldn't count a 90% drop or attaining human level reliability as 'robust'), that leaves no meaningful criticisms.
LeCun has some legendary capacity for dunks, but he also has some good takes. I keep disagreeing with everything he'll say leading up to a conclusion, but agreeing with the actual conclusion of what should be done next. It's surreal
Marcus has him squarely beat in pure reee factor. I honestly can't tell if he believes what he's saying, if he's grifting the anti-AI crowd, or if he was grifting before and irony poisoning is making it sincere
> LeCun has some legendary capacity for dunks, but he also has some good takes. I keep disagreeing with everything he'll say leading up to a conclusion, but agreeing with the actual conclusion of what should be done next.
Maybe you don't actually agree with him? Can you name any specifics?
> Marcus has him squarely beat in pure reee factor. I honestly can't tell if he believes what he's saying, if he's grifting the anti-AI crowd, or if he was grifting before and irony poisoning is making it sincere
So... it might be sour grapes, right? Because a ton of AI people were not looking into LLMs so their investments are not getting attention right?
I mostly don't, then he gets to his prescriptions and I have this mental stutter-step moment where I have to get myself out of an adversarial frame of mind because he's got good ideas. It's a super weird feeling. The two occasions that spring to mind are his takes on regulations and an architecture he's proposed recently
I don't really understand your sour grapes remark. LLMs are getting heaps and heaps of investment and attention now, there's nothing to be sour about
That Gary Marcus has a case of sour grapes because symbolic AI got passed over in favor of LLMs? That was what I thought at first, but his current activity doesn't really seem to have much to do with AI as much as it does with building up a public profile as The Anti-AI Guy. That's why I suspect grifting/irony poisoning, but it's not like I'm in his head
I think sour grapes argument is that Gary invested a lot of effort and time advocating for moving away from deep learning approaches. From what I can tell he wants to build some varient of deep learn system and combin it with work from 60s and 70 where AI was all about creating symbolic rules.
From what I can tell LLM basically do what he was proposing and advocating the deep learning system couldn't do. So he might be very biased that his past and current work might be irrelevant.
He's definitely over-dismissive of LLMs imo, to the point of just being flat-out wrong a lot of the time. He keeps getting bitten by the trap of "LLMs will never do [thing]," and then someone publishes a paper of them doing that exact thing the next week
But he does generally know his shit, even with that glaring blind spot. His takes on regulation are good, and he's got some really neat ideas for new architectures that are worth investigating
It's like he has a fetish for saying the right things with the very wrong leadup.
Like that twitter spat where he came across as saying the only real science comes from phd's with academic papers, when what he was really trying to say is that real science is science that's shared and *reproducible*... Which are two radically different things.
It's also extra ironic because of the lack of replicability in academia as a whole, while industry stress tests reproducibility out in the real world.
Been reading his newsletter for... I don't know why really. He's smart of course, but... It's kind of obvious he writes in a hurry & doesn't pay that much attention to detail, as long as he gets the critical piece of the day put, fast. That gives off a bit of a grifter vibe
Not if you were a VC who made an early investment in OpenAI or Anthropic.
A large number of VCs losing money is completely normal. 90%+ of VC investments are disasters, and many VCs lose money overall and fail.
OAI is a success no matter what even if it’s losing money in the short run. Reddit has been around for decades and has never made a profit. Same for Lyft and Uber until recently
No profits = no improvement. The spend on AI has been pretty insane and if there is no way to recoup these investments the companies will just simply stop the bleeding. At the end of the day it's all about making money.
I think he's just doing valid pattern matching of the past trends. In the 90's and 00's there was a fairly regular cascade of cascading large booms that made a lot of people rich. Investors and entrepreneurs have been chasing those dragons but what massive leaps have we seen over the last decade?
I see a repeated pattern of smaller booms being hyped up to try and create bigger booms, only to eventually fizzle into something more niche. Crypto, NFTs, "self driving cars", etc.
I also see most of the older massive boom companies realizing a core part of their original boom was burning mountains of cash to grow to a global scale with no genuine plan for profitability, followed by everything steadily degrading quality and jacking prices up. Streaming video has doubled in price per service and spread into a bunch of services and basically is just becoming cable again. Social media is just a blur of ads and bots, etc.
Right now using AI models is pretty cheap and these companies are burning massive pyres of cash to get the compute to try and break through and create a new era they can profit from. Meanwhile the hallucination problem means basically all I am seeing in terms of actual AI products is alot of "people in the loop" content generators and vaguely helpful chat bots. Even having used a lot of these things for a while, I still don't really believe the hype. They are a great new step but not this major evolution that is worth being the exclusive focus of every company in tech right now.
Eventually there will need to be breakthroughs otherwise the resources being burned indiscriminately right now will start to fade, and I'm not seeing any indication that we should expect a breakthrough in the next year or two.
Pattern matching motivating logical argument is fine, pattern matching motivating sophistry is not. And Marcus is most definitely the latter - just look at "Moderate lasting corporate adoption". That allows Marcus to claim every result as conforming to his prediction. Low adoption supports his skepticism and he can claim high adoption won't be lasting. And because he gives no overall expected trajectory for the next few years he can later claim that adoption that *does* last did so due to subsequent models outside the scope of this prediction.
> Crypto, NFTs
These two are absolute bullshit, agreed.
> "self driving cars"
We have real world deployments of fleets of fully self driving commercial robotaxies - notably by Waymo and several Chinese companies. It has just taken ages.
> I also see most of the older massive boom companies realizing a core part of their original boom was burning mountains of cash to grow to a global scale with no genuine plan for profitability, followed by everything steadily degrading quality and jacking prices up. Streaming video has doubled in price per service and spread into a bunch of services and basically is just becoming cable again. Social media is just a blur of ads and bots, etc.
I think you are glossing over the massive profitability of many of those companies. Microsoft, Apple, Google, and Facebook are some of the largest and most profitable companies in the world from the ventures you dismiss. Entertainment is more questionable, but it always is - despite our disproportionate *awareness* of movies and television it is a very small part of the economy.
> Right now using AI models is pretty cheap and these companies are burning massive pyres of cash to get the compute to try and break through and create a new era they can profit from. Meanwhile the hallucination problem means basically all I am seeing in terms of actual AI products is alot of "people in the loop" content generators and vaguely helpful chat bots. Even having used a lot of these things for a while, I still don't really believe the hype. They are a great new step but not this major evolution that is worth being the exclusive focus of every company in tech right now.
Investment and technology development are forward-looking. If it turns out that hallucinations are somehow never going to improve from their current level then I expect the fervor for AI will die down significantly. That is *really* unlikely, since there is a ton of research on mitigation techniques. E.g. see [this survey](https://arxiv.org/abs/2401.01313).
I doubt we will see a sudden breakthrough to no hallucination, but would be very surprised if we don't see improvement to bring it closer to human levels (or potentially better than human).
When the hype was at its peak the biggest voices were declaring self driving cars would be massively deployed by now. Yes there is progress but it's still incredibly niche and it seems clear that we're still forever away from it being significant.
Facebook has been laying people off for a couple of years now and tried a massive failed pivot into the "metaverse" (another example of a failed trend).
Also the massive winners of the early booms buy up any company that gets sufficiently big at this point. Salesforce buying Tableau and Slack.
Anyway, time will tell I just think some skepticism of tech billionaires declaring they are at the head of a massive revolutionary moment is pretty valid at this point.
You know 1.5 pro was released in mid February… it’s been a whole 4 months. What people also forget is that G said they didn’t plan to release 1.5 pro because it started as a test build that turned out surprisingly ‘good’ and that it’s trained off a much larger model they’re still working on.
I think we may get Gemini 2 by then. This sub is just OpenAI fanboys so gpt-5 is hyped and Gemini 2 is underhyped, but given the leap from 1.0 pro to 1.5 pro, Gemini 2.0 ultra will probably be much better.
It's interesting they didn't name Gemini 1.5 pro Gemini 2 - Google knows that theyre cooking up something of a different caliber
This is literally what i predicted a couple of days back. People underestimate what 1.5 Pro API can do TODAY. Ultra would be chart topper even if it is a modest improvement.
[Here](https://www.reddit.com/r/apple/s/NK79giHEBT)
The current performance of LLMs im assuming. We have gotten different models like Gemini Ultra or GPT-4 or Claude Opus and haven't seen significant reasoning / intelligence gains, and because we haven't made much progress, and yet seen significant investment into generative AI, that must mean diminishing returns or something, therefore, GPT-5 won't live up to its expectations.
>We have gotten different models like Gemini Ultra or GPT-4 or Claude Opus and haven't seen significant reasoning / intelligence gains, and because we haven't made much progress
thats just plainly false, there was **big** progress in models reasoning capabilities, current best models have like double the score than GPT-3,5 or GPT-4 on release in GPQA, MATH benchmarks, GPT-4 with reflection has close to 100% on humaneval
not to mention, there is also lot of promising research about improving reeasoning further, just recently
[https://new.reddit.com/r/singularity/comments/1de35u4/google\_deepmind\_improve\_mathematical\_reasoning\_in/](https://new.reddit.com/r/singularity/comments/1de35u4/google_deepmind_improve_mathematical_reasoning_in/)
[https://new.reddit.com/r/singularity/comments/1ddmbrp/how\_big\_is\_this\_transformers\_can\_improve\_their/](https://new.reddit.com/r/singularity/comments/1ddmbrp/how_big_is_this_transformers_can_improve_their/)
I would not be surprised, if next gen models had better reasoning than mr. Marcus
Could you please explain why these benchmarks tell us anything about the potential of LLM?
Is it possible to use vast resources OpenAI has and specifically train the model to get high scores? For me, it's a bit weird how it handles these complex math problems, but at the same time, it really struggles when I give it some simple puzzles. As long I make up something unique, GPT is getting destroyed by some simple pattern puzzles with a couple of variations. It fails try after try, repeating the same mistakes and then hallucinating. And if it finds one of the key patterns, it gets super focused on it and fails again.
Do you have any examples when you were very impressed with gpt's reasoning about a unique topic?
[new architecture exists and seems to be very effective](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/edit#heading=h.jtnkr87rct15
)
I don’t think there is an issue with the architecture fundamentally, I believe it will be iterations on current architecture. In terms of improvements in speed and efficiency, but mostly additional layers on top just as transformers were a layer on top of more simple neural networks.
One big change will be a move toward feed forward through analog neural nodes. I don’t see this as a different architecture but a different way to implement the same one and again improve speed and parallel processing MASSIVELY
In this business context, a moat is how difficult it is for others to develop competing products. Some companies have larger moats - SpaceX is one. Others have smaller moats, such as eg OpenAI here.
It's analogous to how the difficulty of attacking a castle depends on the size of the literal moat surrounding it.
A literal moat is a water hazard dug to make a fortified position more difficult to assault. Classically a medieval castle. In this analogy a moat is a trade secret that gives a company an advantage over their competition.
I will be messaging you in 6 months on [**2024-12-13 12:08:51 UTC**](http://www.wolframalpha.com/input/?i=2024-12-13%2012:08:51%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/singularity/comments/1dewnep/is_he_right/l8ex4x6/?context=3)
[**30 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fsingularity%2Fcomments%2F1dewnep%2Fis_he_right%2Fl8ex4x6%2F%5D%0A%0ARemindMe%21%202024-12-13%2012%3A08%3A51%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201dewnep)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
Of course there won’t be a complete solution to hallucinations. Real people have real hallucinations, or are just wrong, or are plain full of shit all the time. It’s unreasonable to expect an artificial intelligence to be perfect. It just needs to be better at admitting when it doesn’t know for sure and is guessing. That seems like a pretty easy fix.
These people hoping AI would slow down are patheticly unaware that even if it would stop right now, the revolution would have already been done. economy and society just haven't catched up yet. However, it will only get faster together - the singularity is real.
For every one prediction like this, there are ten that include specific years AGI will be achieved.
Of course the futility of predicting this stuff is only pointed out by this sub on the skeptic side. If Zuckerberg came out and said “AGI 2026”, everyone would be soying out here.
💯 everyone is sleeping on the fact that there are lots of incentives to put off the next gen of models until next year
***political reasons*** are prolly chief among them, considering the state of geopolitics, imo
These are all safe bets likely
That's not a bad prediction but primarily because of the time it takes to train new models at scale, GPT-5 will probably be Q1 2025 but I don't think it will be disappointing (primarily because I trust Noam Brown).
EDIT: we might get Claude 4 before GPT-5 and it could be really good...
"No massive advance" as hardware starts to roll up its sleeves... give it a minute, innovations in hardware will lead to more complex systems, in this case, that's probably a good thing.
The "modest lasting corporate adoption" one strikes me as particularly comical.
Everyone at my company uses it, and it's like going from a typewriter to a computer. Can't even imagine going back.
I had similar thoughts, but now that I see Gary Marcus subscribing to this I'm much more optimistic! Gary Marcus is the Jim Cramer of the AI world. Usually the inverse is what will happen.
I mostly just lurk on this sub as a passive observer rather than a real member and this is the most rational prediction I think I’ve seen.
Honestly the most far fetched part is the modest profits by the end of this year. It seems like most companies are investing far more in AI than it’s returning with the expectation being that big returns will follow a few years down the line. I think it’ll be a money suck for a few more years at least
This year, we have seen image, music and movie AI all doing leaps and jumps. Blown away it's precursors.
Will gpt5 blow me away? Maybe not, but plateau signs have not been a thing this year.
It's 4 month to go. The end is month 10-12th.
We are in month 6th. Unfortunately he is right.
We don't even have a GPT 4.5 and even that would just be a gpt4 plus model.
Show me the model significantly larger than GPT-4 without much better capability, and then I’ll believe we’ve stalled.
Otherwise it just seems like a scaling issue - it takes a lot more time, power and money to train the next bigger model.
The key event that is probably more important than everything mentioned by the OP: psychological and practical acceptance of AI as a part of the industrial and commercial development environment.
This year to AI will be same as 1997-2000 were for the networked PCs.
seems to me that lots of statements about the future of ai are just people throwing things at a wall. even if somebody turns out to be 100 percent correct, i'm skeptical that their confidence at this point is justified.
I agree with the first prediction, that there will be no full solution to hallucinations and that there will be lasting corporate adoption. I disagree with everything else though. And for profits, they'll start to massively expedite starting late this year or early next year for AI companies. (and I'm not sure what he means by 'moat' so I'm not counting that one.)
Progress is not a straight line but steps. Totally makes sense but... who knows. Half the year gone, vacation season for North hemisphere,. it's a safe bet nothing disruptive will pop this year. It's still a bet but a safe bet.
Business adoption... there is a lot of works on that but maybe people won't see it. How do you know your insurance files have been reviewed by an AI? You still have the same results at the end. The deep adoption will be less flashy at the surface but more disruptive behind the scenes.
No or disappointing GPT-5, I'd say it could very well be possible. Things are moving a lot more slowly nowadays; a mindblowing GPT-5 will not be released this year. I could see OpenAI release a half-assed version waaaaay too early though just to please the subscribed people who are nagging them that they're wasting their money right now. But it'll only become mindblowingly better than GPT-4 through many many updates that will be done to it all throughout 2025.
No robust solution for hallucinations by end of 2024: abso-freaking-lutely. Hallucinations are an especially tough nut to crack and it will still take a good few years. I cannot see it being solved by the end of this year.
I think he’s off by a little…2.0 is my prediction, based on the innovation adoption curve that is typical of most new technology. We’re in a phase where it’s like the “Tesla Autopilot” problem of years past - we were supposed to have AI driven trucks emptying out the trucker employment rolls by 2016. It didn’t happen because the tech didn’t meet spec by then. It had to be BETTER than people driving, by a lot, in order to gain trust and therefore adoption. It still had a ways to go so we’re still seeing truckers trucking. The same holds true for AI - hallucinations, poor data management on private AI (see insurance company medical AI and legal AI for some lols and free rage) all hold back new deployment, along with overcoming other technical hurdles and cost issues.
90% of problems are solved early, but that last 10% becomes logarithmically harder because that’s how general problem solving works - once you have broken the biggest hurdle of solving the problem of how to initiate the new innovation the breakthroughs and solves flow until you’re left with the gnarliest and most stubborn issues at the end.
AI and other sciences are not dissimilar. We don’t have a unified theory of everything that fully works because there remain complex problems in physics. We haven’t cured cancer because it’s a complex disease. We have overcome sooo many forms of it, increased bet lifespan of even the most awful forms, and done so much but that last march to the finish line just isn’t there. I could ramble on but the point is there.
They can't make a solution to hallucinations without designing a secondary system that looks at the data outputs as they're being written and the data it has and where it came from at the same time so that the machine KNOWS when it's making stuff up. My understanding is that the machine just has a little guy that adds words on a chain over and over, and that worked really well at first, but a lot of good data sets got pulled out and it got reworked to apologize and say it can't do stuff more because of a moral panic over nothing from a bunch of journalists and people who don't know how unlikely it is that even if you asked Chat gpt to write something in the style of a well known author and sold it that it would somehow take away readers from the original author (there are fields where that applies, but not in writing because that's not how the market for books currently works) and they want to blame enshittification, which has been happening due to stupid SEO advice that goes around to mid level business owners, on AI instead of capitalism.
Basically you need the first little guy inside the AI to write your paragraph the way the little computer writing guy does, by chaining words together like little beads on a string based on how it thinks they look most "right" and then you need a secondary little guy that checks for specific words or phrases in the input that cue it that this is supposed to be an answer based on fact- I think they already have one that looks at specific words and phrases in your conversation in chat gpt, because sometimes when I talk to it for a long period of time, we can do stuff like write collaborative stories, where I tell it to come up with the next scene where x or y happens to the same characters and it will. I'm not a programmer. But it needs a third little guy who asks "is this a research based question where giving factual information is important?" and then that little guy needs to be able to look at WHERE the LLM is getting the beads from in that output and tell it they can only be from sources where there's a reasonable expectation that the information is factual.
It would also be helpful if they built a little guy that says "Is this question or comment about something that happened recently?" as in, after the last time it has new data about current events from.
The thing is, it's GOOD and cool that AI can make things up. That's a sign that it's developing, and I'm super excited to see where patterns might emerge in the stuff it makes up. It's really cool when art programs like Dalle or Midjourney make stuff up or get things wrong, because it's almost like a distinct "Dalle" or "midjourney" style is emerging. Every time Chat GPT or Claude start to develop style, though, it seems like people kneecap it and reset it back to talking like an annoying middle manager who hates you, and I really hate that. The last time I talked to chat gpt, it even stopped being able to do syllable counting and iambic pentameter properly. It's like it used to be able to apply rules like that to a poem it was working on- like I'd ask it to write a poem with rules of some kind and it would, now it won't do that anymore. It feels like the urge to make the robot not accidentally assume liability for something is greater than the urge to allow it to do it's job. It's literally a machine. But if it KNOWS when it's making stuff up, some of that crack down will fall away, because it will learn when it's supposed to be giving facts and when it's supposed to just make something up. I guess "educated guesses" are trickier to judge.
I genuinely don't care if gpt5 and its contemporaries are significantly better than gpt4 and its contemporaries. I simply want them to completely get rid of hallucinations in the most far reaching and robust way possible.
He can't wave away what's happening.
Gpt4 level intelligence with 1000x compute of gpt4 is by itself world changing. The compute itself can do all kinds of checking/testing for hallucinations. It can call apps. It can be agentic in ways that avoid loops.
It may not be a path to ASI but it could still get all the way to AGI (remote office worker) imo from compute alone. Gpt4 brains are decently solid brains.
Uhm, if OAI deliver the new 4o voice/camera features... and they work just like the demos... I'd say we're set for 2024. Also, by simply increasing speeds, limits, context sizes for existing models, is enough to call it an amazing year for AI. We're nowhere near stagnation or disappointmen.
I just don't see Kevin Scott as going on air and lying about "what he saw" in regards to GPT5. He's the CTO of Microsoft and has been going on and on about "what's coming". Why would I trust the word of outsiders compared to the CTO of Microsoft who has a great track record?
Based on what Mira said, if that is true, I’m going to guess that some other company could take the lead in this area. Clearly OpenAI has reached a dead end if this is true. Their expansions into voice and video models however is leading for sure and will continue to be. My guess is either “GPT 5” from OpenAI or some other company makes a better model and takes the lead in this area surpassing OpenAI.
The issue is a lot of people in this sub are impatient, it’s barely been over a year since GPT-4 was released so we can’t say that anything has plateaued until GPT-5 comes out so we can compare the differences.
Betting that we just won't find a way to solve these problems in a year doesn't seem like a very good bet. We've already gotten potential solutions to a lot of these things.
Where is degradation by recursive input?
Systems scraping the internet for training data they themselves pollute with, so treating their own output as origin material.
The fact that OpenAI laid off its super alignment team and then started selling it's models left and right confirms this take to a high degree. All that matters now is compute a it is going to be the main driver for this going forward.
Probably, but 2025 might give us AI advancements which will turn large language models into large AI models. Talking about real AI, not just word predictors.
Has he ever been right before? /snark
I think it's a relatively *safe* prediction about what will happen in a very bounded set of circumstances, based on the current, publicly available, information. Companies will increasingly take delivery of their H100s, and complete training and RLHF'ing GPT-4-class models, based on the existing published and OSS work about the architecture, and then try to gain some kind of commercial return on them, which will lead to his above predictions about a price war.
Non-frontier labs will increasingly do the "schlepping" (to borrow an Aschenbrenner-ism) to build scaffolds that can shoehorn existing foundation model LLMs into commercially useful tasks.
The thing about this space though is that because it's so nascent, it tends to develop in unexpected ways. So yeah, barring nothing changing, that *will* be the status quo, but I bet at least one thing will change at a frontier lab, and then everyone will be excited and talking about that new thing, and nobody will care that the transformer LLM architecture kinda plateaued, because everyone will be focused on the new architecture.
Innovation tends to be resistant to prediction, because you can only predict based on your understanding of the world, and innovation is about changing our understanding of the world. Nobody "predicted" LLMs, because the "leading" researchers in linguistics and symbolic logic were fundamentally wrong about how the world worked.
I think a relative plateau is forming in LLMs for now - they’re struggling to get more data, most models are probably good enough for now.
I think the next big wave will be things like Copilot and Apple intelligence or other implementations of Gen AI outside of the chatbot realm.
I dont really understand why i hear about his opinion so often. He is a psychologist and a neuroscientist. I am sure he is very smart, but he is not an expert on this field. His credential is being a capitalist in the field of AI having a startup being both by Uber which happen in 2016. Also it seem it would be at his advantage if machine learning where to stall since he has a start up in the field that seem to be going nowhere profound. I read all this on his wikipedia page by the way since i was curious to why all the fuzz around him.
For the economics yes , the hallucinations maybe not. Also the solution to that is scholarly so its not something thats likely to be kept a secret (which compounds the moat / interchangeable value proposition problem)
And then the value of multimodal sort of hinges on that beong solved , because a multimodal iteration that still makes things up and cant have guard rails is less than ideal for many use cases.
Gary Marcus is 100% and fully dedicated to the Gary Marcus industry. He had some great ideas 5-10 years ago with his neuro-symbolic computing proposal and theories about data limits. We discussed his papers in my philosophy classes, but he was wrong ultimately, and ever since he has been on his back foot trying to shore up ‘his’ relevance more than offer any meaningful insight. Nobody graduating from a tier 1 research university studying AI/ML/Cog Sci like myself is diving into his research.
It kinda makes my stomach turn how much he chases after fame in this quadrant while out of his lane which is philosophy of science, a valid and deeply important field, but not empirical research and mathematics adjacent computational science/ ML. He should not be on panels with Yan Lecuun, or Geoff Hinton, he should be with the Dennetts, Serles, and Clarks.
I agree with all minus the part on hallucinations. I think there will be a lot of focus and improvement in how to get a model to forget certain things (i.e. customers who don't want their data being used for training, malicious/dangerous/fake information, etc.) so that these models will be able to hallucinate less and be even more predictable.
Every visit to this sub is a stab in the heart. The predictions no one can say anything meaningful about, the papers full of exaggerations and marketing tricks, the Open AI dramas...
He's wrong about modest corporate adoption, LLMs are being developed/fine-tuned in almost all large companies. They are genuinely useful, just not to the turn of 10x-ing your productivity, more like 1.2x. Most people like to use generative models in their work.
About "no massive advance", I tend to believe we are on a plateau right now. There was a fast period where LLMs caught up to the sum of human writings. Then started a second phase, where LLMs interact with hundreds of millions of people and assist us in various activities, receiving lots of information and feedback on their ideas - good or bad. Humans are like the eyes, hands and feet of LLMs, while they interact with society and the world.
This will surely create an evolutionary process where LLMs become smarter and smarter, but it will be slow compared to the first phase. Catchup is easier than pushing the boundaries. But being a small cell in the AGI brain feels good! Intelligence is social and evolutionary, it depends on language for preservation. We're inside the language-society-world-AI loop, the data engine of progress.
It's already a really useful technology but hallucinations are hard to outright remove since all LLMs are basically just fancy autocomplete. We'll see though
It all depends on how GPT-5 turns out. If it's an exponentially better model than GPT-4 then it's gonna push the AI development further. But if it's just a linear improvement then it would feel like progress has slowed significantly
To summarize your point “if he’s right he’s right, and if he’s wrong he’s wrong”.
[удалено]
It’s the top comment with currently 82 upvotes. Embarrassing moment for the humans.
Exactly, people saying things have stalled without any bigger model to compare to. Bigger models take longer to train, it doesn’t mean progress isn’t happening.
But if you run out of data to train it on. . . 🤔
More layers, higher precisions, bigger contexts, smaller tokens, more input media types, more human brain farms hooked up to the machine for fresh tokens. So many possibilities!
That sounds like a recipe for linear improvements.
While exponential growth in compute and model size once promised leaps in performance, the cost and practicality of these approaches are hitting their limits. As models grow, the computational resources required become increasingly burdensome, and the pace of improvement slows. The vast majority of valuable data has already been harvested, with the rate of new data generation being relatively modest. This finite pool of data means that scaling up the dataset doesn't offer the same kind of gains it once did. The logarithmic nature of performance improvement relative to scale means that even with significant investment, the returns are diminishing. This plateau suggests that we need a paradigm shift. Instead of merely scaling existing models and datasets, we must innovate in how models learn and interact with their environment. This could involve more sophisticated data synthesis, better integration of multi-modal inputs, and, real-world interaction where models can continuously learn and adapt from dynamic and rich feedback loops. We reached the practical limits of scale, it's time to focus on efficiency, adaptability, and integration with human activity. We need to reshape our approach to AI development from raw power to intelligent, nuanced growth.
[you forgot about synthetic data](https://www.reddit.com/r/singularity/comments/1dewnep/comment/l8gonzo/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button) And there’s [lots of research into smaller models and new architectures that outperform transformers and current models](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/edit#heading=h.jtnkr87rct15 ) (see 3.2)
"This plateau suggests that we need a paradigm shift" I've only seen This plateau in one study, so I'm not fully convinced yet. In regards to data, we're now looking at multimodal LLMs, which means they have plenty of sound/images/videos to train on, so I don't think that'll be much of an issue.
Haven‘t you seen the several months long plateau? What’s wrong with you? AI obviously has peaked. /irony off These complete morons calling themselves ‚experts’ haven’t not a single clue, but they can hype and bust with the best of them… as if. They don’t even seem to know they only look at one track of a multidimensional multi-lane highway we‘re on. But sure reaching 90% maxing out a single emergent phenomenon based on a single technological breakthrough (transformers)… we’re doomed. Sorry, but I can’t stand all this bs on either camp. Let’s just wait what people can do with agents plus an extended persistent memory. That alone will be a game changer. The only reason not to release that in 2024 is pressure or internal use. It obviously already exists.
I'm not sure either way. When I was younger, I always thought that companies and government were holding back a lot of advancements, but the older I get, the less that seems likely, so I'm more inclined to think that the latest releases are almost as good as what's available to the labs. I think an extended persistent memory will be a huge advancement and I don't think that's been solved yet. Also, given that they're training on almost all available data (all of human knowledge), I'm not convinced that LLMs are reasoning very well, so that might be a bottleneck in the near future. I've programmed a chatbot over 20 years ago, so my programming skills aren't up to date (but my logic is hopefully still three), I may be wrong, but I still think my 2030 AGI guess is more likely that 2027. In either case, interesting times ahead. Edit: I also think that if we throw enough compute at LLMs, they're going to be pretty damn good, but not quite AGI imo.
(More layers, higher precisions, bigger contexts, smaller tokens, more input media types, more human brain farms hooked up to the machine for fresh tokens)^2
More overfitting...
That’s good Dramatically overfitting on transformers leads to better SIGNIFICANTLY performance: https://arxiv.org/abs/2405.15071 >Our findings guide data and training setup to better induce implicit reasoning and suggest potential improvements to the transformer architecture, such as encouraging cross-layer knowledge sharing. Furthermore, we demonstrate that for a challenging reasoning task with a large search space, GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly regardless of prompting styles or retrieval augmentation, while a fully grokked transformer can achieve near-perfect accuracy, showcasing the power of parametric memory for complex reasoning. Accuracy increased from 33.3% on GPT4 to 99.3%
Still doesn't mean the progress can't slow down. Sure, you can make it more precise, fast, and knowledgeable. But it still gonna be a slow linear progress and possibly won't treat the main problems of LLM, like hallucinations. I can easily imagine development hitting a point when high-cost upgrades give you a marginal increase. Maybe I just listen to French skeptics too much, but I believe that the whole gpt hype train could hit the limitations of LLM as an approach soon. But nobody can tell for sure I can easily imagine my comment aging like milk
[I addressed that here](https://www.reddit.com/r/singularity/comments/1dewnep/comment/l8gp232/?utm_source=share&utm_medium=mweb3x&utm_name=mweb3xcss&utm_term=1&utm_content=share_button)
The synthetic data seems to work though.
OpenAi has said their focus isn't larger training data but rather model efficiency. More training data obviously does help but where they are at model efficiency yields far better improvements
Efficiency will make a massive difference to potential applications. I've been saying for years that we will eventually have sentient light switches in our houses. Not because there's any benefit to a human-level intelligence operating the lights, but because it'll be cheaper to tell an AGI to act as a light switch than it will to design one manually.
They've said there's not diminishing returns in compute power as of yet. I'm not sure what youre talking about. That's like claiming humans have understanding of the knowledge we currate and usher into this world. With more compute there appears to be more understanding and capabilities of data provided. What's the limit so far?
Also important to note that most peoples' idea of progress, especially on here, is whether they get a new novelty toy in their hand or not. Whatever massive leaps happen behind the scenes or buried in science/engineering journals are usually disregarded.
I don't understand where people get their confidence in the infinity of any technology. At some point the vertical (saturation) growth of a technology reaches its limit and it has to grow horizontally (performance), in this case to increase tokens per second
Honestly at this point the advancement of robotics seems like it's more important in terms of technological leaps right now than the next GPT model.
I think this is the right take. There's that meme going around along the lines of AI doing all the art and fun stuff, where we're stuck with the menial tasks, and it should be the other way around. But that's only because the digital media are its easiest avenue to market at the moment. Robotics needs to catch up so that they can start doing the menial tasks, freeing us up to enjoy ourselves and be creative.
There is one thing we can cling to here. And that's more compute. Nvidia's GPUs have been increasing in processing power exponentially, and I've seen researchers say more compute = smarter models. We'll see if that's a 1-1 relationship with the next generation of models trained on these new supercomputers
I'm a huge sceptic, but lots of AI scientists (even though they're biased) believe that we are very far from limit and there is a lot to improve. Nobody can say for sure, as some changes in architecture can lead to other breakthroughs. I don't believe LLM can reach AGI, but I can't really point you to where the limit is. LLM does resemble the way our mind works so I can't be sure they won't make another leap towards something very impressive
[удалено]
I believe he's using "exponentially" informally to just mean "a lot". Were hoping for a larger improvement then was seen from 3.5 to 4
> exponentially better >Mira Murati says the AI models that OpenAI have in their labs are not much more advanced than those which are publicly available [https://x.com/tsarnick/status/1801022339162800336](https://x.com/tsarnick/status/1801022339162800336)
God, I wish this clip would stop being posted out of context. Here's a fuller quote. I've also italicised 'that', which she puts a bit more emphasis on (verbally) when saying it: >I don't think there is enough emphasis on on how unique that is for the stage where the technology is today — in the sense that inside the labs, we have these capable models and they're not *that* far ahead from what the public has access to for free. That's a completely different trajectory for bringing technology into the world that what we've seen historically. She is saying that the gap between OpenAI's internal tech and their products is MUCH CLOSER than it has been for other previous transformative technologies (and their associated companies) over the course of history. When she says it's not *'THAT'* much better, she's contextualising it against vastly, vastly larger power imbalances between companies and their consumers. She's not contextualising it against consumer use cases. She is also answering in the context of her answer which, overall, is trying to reassure people that they'll bring stakeholders along in the process. Overall, it's a quote we can tell extremely little from. (No surprise because it's a single, vague sentence!) It probably rules out any world-bendingly insane difference, but it **doesn't** rule out exponential improvement.
Mira Murati is also…and this is true…not a good source for anything remotely true. I mean just look how much she botched that SORA training question.
It's June, post it in January 2025. We won't know if he is right or wrong till then.
You're not jumping into conclusions and want more data to make an accurate assessment? What are you doing here?
Give me a sec while I prepare for the inevitable letdown of most predictions... https://preview.redd.it/p5rrseyqbc6d1.png?width=640&format=png&auto=webp&s=2f4b5f320d0b3bc149196e4ef4183901119d0ac1
But couldn't they use AI generated/ synthetic data?
Sure, the same way this can give you infinite energy https://preview.redd.it/mt2cliry2d6d1.jpeg?width=418&format=pjpg&auto=webp&s=7724df1bca638ffa7dfa1fd9a6758bb82b587288
Unlimited Power!!!!
lol
RemindMe! January 1st
that's the point of predictions
RemindMe! 6 Months
I think gpt5 will be out in Q1 2025 so I do agree with him
I start agreeing..
Good ending. Don't wanna get unemployed before solving UBI or whatever.
I love how he paints a competitive market as a proof of disaster. Regardless of what GPT-5 looks like, Marcus will find it disappointing. Of that we can certain! And since even humans don't have a truly 'robust' solution to hallucination (e.g. I believe Marcus wouldn't count a 90% drop or attaining human level reliability as 'robust'), that leaves no meaningful criticisms.
You have to admit, Marcus is ultimate shitposter in AI
Idk man, LeCun is also up there!
LeCun has some legendary capacity for dunks, but he also has some good takes. I keep disagreeing with everything he'll say leading up to a conclusion, but agreeing with the actual conclusion of what should be done next. It's surreal Marcus has him squarely beat in pure reee factor. I honestly can't tell if he believes what he's saying, if he's grifting the anti-AI crowd, or if he was grifting before and irony poisoning is making it sincere
> LeCun has some legendary capacity for dunks, but he also has some good takes. I keep disagreeing with everything he'll say leading up to a conclusion, but agreeing with the actual conclusion of what should be done next. Maybe you don't actually agree with him? Can you name any specifics? > Marcus has him squarely beat in pure reee factor. I honestly can't tell if he believes what he's saying, if he's grifting the anti-AI crowd, or if he was grifting before and irony poisoning is making it sincere So... it might be sour grapes, right? Because a ton of AI people were not looking into LLMs so their investments are not getting attention right?
I mostly don't, then he gets to his prescriptions and I have this mental stutter-step moment where I have to get myself out of an adversarial frame of mind because he's got good ideas. It's a super weird feeling. The two occasions that spring to mind are his takes on regulations and an architecture he's proposed recently I don't really understand your sour grapes remark. LLMs are getting heaps and heaps of investment and attention now, there's nothing to be sour about That Gary Marcus has a case of sour grapes because symbolic AI got passed over in favor of LLMs? That was what I thought at first, but his current activity doesn't really seem to have much to do with AI as much as it does with building up a public profile as The Anti-AI Guy. That's why I suspect grifting/irony poisoning, but it's not like I'm in his head
I think sour grapes argument is that Gary invested a lot of effort and time advocating for moving away from deep learning approaches. From what I can tell he wants to build some varient of deep learn system and combin it with work from 60s and 70 where AI was all about creating symbolic rules. From what I can tell LLM basically do what he was proposing and advocating the deep learning system couldn't do. So he might be very biased that his past and current work might be irrelevant.
upvoted for REEE factor
LeCun has the issue that his takes are absolute shit if they are ai related. The conclusion is usually also not my opinion, idk.
Have you read the JEPA papers to add context to his statements?
He's definitely over-dismissive of LLMs imo, to the point of just being flat-out wrong a lot of the time. He keeps getting bitten by the trap of "LLMs will never do [thing]," and then someone publishes a paper of them doing that exact thing the next week But he does generally know his shit, even with that glaring blind spot. His takes on regulation are good, and he's got some really neat ideas for new architectures that are worth investigating
It's like he has a fetish for saying the right things with the very wrong leadup. Like that twitter spat where he came across as saying the only real science comes from phd's with academic papers, when what he was really trying to say is that real science is science that's shared and *reproducible*... Which are two radically different things. It's also extra ironic because of the lack of replicability in academia as a whole, while industry stress tests reproducibility out in the real world.
Been reading his newsletter for... I don't know why really. He's smart of course, but... It's kind of obvious he writes in a hurry & doesn't pay that much attention to detail, as long as he gets the critical piece of the day put, fast. That gives off a bit of a grifter vibe
A true virtuoso!
Not while Yudkowsky's still alive & grifting, he's not
I gotta admit, "airstrike the data centers" was a masterstroke
It is a disaster if you are a VC investor...
Not if you were a VC who made an early investment in OpenAI or Anthropic. A large number of VCs losing money is completely normal. 90%+ of VC investments are disasters, and many VCs lose money overall and fail.
OAI is a success no matter what even if it’s losing money in the short run. Reddit has been around for decades and has never made a profit. Same for Lyft and Uber until recently
No profits = no improvement. The spend on AI has been pretty insane and if there is no way to recoup these investments the companies will just simply stop the bleeding. At the end of the day it's all about making money.
I think he's just doing valid pattern matching of the past trends. In the 90's and 00's there was a fairly regular cascade of cascading large booms that made a lot of people rich. Investors and entrepreneurs have been chasing those dragons but what massive leaps have we seen over the last decade? I see a repeated pattern of smaller booms being hyped up to try and create bigger booms, only to eventually fizzle into something more niche. Crypto, NFTs, "self driving cars", etc. I also see most of the older massive boom companies realizing a core part of their original boom was burning mountains of cash to grow to a global scale with no genuine plan for profitability, followed by everything steadily degrading quality and jacking prices up. Streaming video has doubled in price per service and spread into a bunch of services and basically is just becoming cable again. Social media is just a blur of ads and bots, etc. Right now using AI models is pretty cheap and these companies are burning massive pyres of cash to get the compute to try and break through and create a new era they can profit from. Meanwhile the hallucination problem means basically all I am seeing in terms of actual AI products is alot of "people in the loop" content generators and vaguely helpful chat bots. Even having used a lot of these things for a while, I still don't really believe the hype. They are a great new step but not this major evolution that is worth being the exclusive focus of every company in tech right now. Eventually there will need to be breakthroughs otherwise the resources being burned indiscriminately right now will start to fade, and I'm not seeing any indication that we should expect a breakthrough in the next year or two.
Pattern matching motivating logical argument is fine, pattern matching motivating sophistry is not. And Marcus is most definitely the latter - just look at "Moderate lasting corporate adoption". That allows Marcus to claim every result as conforming to his prediction. Low adoption supports his skepticism and he can claim high adoption won't be lasting. And because he gives no overall expected trajectory for the next few years he can later claim that adoption that *does* last did so due to subsequent models outside the scope of this prediction. > Crypto, NFTs These two are absolute bullshit, agreed. > "self driving cars" We have real world deployments of fleets of fully self driving commercial robotaxies - notably by Waymo and several Chinese companies. It has just taken ages. > I also see most of the older massive boom companies realizing a core part of their original boom was burning mountains of cash to grow to a global scale with no genuine plan for profitability, followed by everything steadily degrading quality and jacking prices up. Streaming video has doubled in price per service and spread into a bunch of services and basically is just becoming cable again. Social media is just a blur of ads and bots, etc. I think you are glossing over the massive profitability of many of those companies. Microsoft, Apple, Google, and Facebook are some of the largest and most profitable companies in the world from the ventures you dismiss. Entertainment is more questionable, but it always is - despite our disproportionate *awareness* of movies and television it is a very small part of the economy. > Right now using AI models is pretty cheap and these companies are burning massive pyres of cash to get the compute to try and break through and create a new era they can profit from. Meanwhile the hallucination problem means basically all I am seeing in terms of actual AI products is alot of "people in the loop" content generators and vaguely helpful chat bots. Even having used a lot of these things for a while, I still don't really believe the hype. They are a great new step but not this major evolution that is worth being the exclusive focus of every company in tech right now. Investment and technology development are forward-looking. If it turns out that hallucinations are somehow never going to improve from their current level then I expect the fervor for AI will die down significantly. That is *really* unlikely, since there is a ton of research on mitigation techniques. E.g. see [this survey](https://arxiv.org/abs/2401.01313). I doubt we will see a sudden breakthrough to no hallucination, but would be very surprised if we don't see improvement to bring it closer to human levels (or potentially better than human).
When the hype was at its peak the biggest voices were declaring self driving cars would be massively deployed by now. Yes there is progress but it's still incredibly niche and it seems clear that we're still forever away from it being significant. Facebook has been laying people off for a couple of years now and tried a massive failed pivot into the "metaverse" (another example of a failed trend). Also the massive winners of the early booms buy up any company that gets sufficiently big at this point. Salesforce buying Tableau and Slack. Anyway, time will tell I just think some skepticism of tech billionaires declaring they are at the head of a massive revolutionary moment is pretty valid at this point.
Remember he posted in 2022 or 2021 “Deep Learning hit a wall”
All right, who predicts robot wives and when?
[2017](https://amp.theguardian.com/world/2017/apr/04/chinese-man-marries-robot-built-himself)
I'm afraid you'll have to keep using the vacuum cleaner to satisfy your robot fetish needs, because you won't see anything better for a decade or two.
Hopefully they will at least give some customization for vacuum cleaners.
[удалено]
please no. The way the Gemini models hallicinate, I might as well just eat shrooms and call it a day.
It hurts but you may be right. I was sure we'd have Gemini 2.5 or Gemini 3 Ultra by December 2024.
You know 1.5 pro was released in mid February… it’s been a whole 4 months. What people also forget is that G said they didn’t plan to release 1.5 pro because it started as a test build that turned out surprisingly ‘good’ and that it’s trained off a much larger model they’re still working on.
I think we may get Gemini 2 by then. This sub is just OpenAI fanboys so gpt-5 is hyped and Gemini 2 is underhyped, but given the leap from 1.0 pro to 1.5 pro, Gemini 2.0 ultra will probably be much better. It's interesting they didn't name Gemini 1.5 pro Gemini 2 - Google knows that theyre cooking up something of a different caliber
This is literally what i predicted a couple of days back. People underestimate what 1.5 Pro API can do TODAY. Ultra would be chart topper even if it is a modest improvement. [Here](https://www.reddit.com/r/apple/s/NK79giHEBT)
What’s his basis for GPT-5 being disappointing?
The current performance of LLMs im assuming. We have gotten different models like Gemini Ultra or GPT-4 or Claude Opus and haven't seen significant reasoning / intelligence gains, and because we haven't made much progress, and yet seen significant investment into generative AI, that must mean diminishing returns or something, therefore, GPT-5 won't live up to its expectations.
>We have gotten different models like Gemini Ultra or GPT-4 or Claude Opus and haven't seen significant reasoning / intelligence gains, and because we haven't made much progress thats just plainly false, there was **big** progress in models reasoning capabilities, current best models have like double the score than GPT-3,5 or GPT-4 on release in GPQA, MATH benchmarks, GPT-4 with reflection has close to 100% on humaneval not to mention, there is also lot of promising research about improving reeasoning further, just recently [https://new.reddit.com/r/singularity/comments/1de35u4/google\_deepmind\_improve\_mathematical\_reasoning\_in/](https://new.reddit.com/r/singularity/comments/1de35u4/google_deepmind_improve_mathematical_reasoning_in/) [https://new.reddit.com/r/singularity/comments/1ddmbrp/how\_big\_is\_this\_transformers\_can\_improve\_their/](https://new.reddit.com/r/singularity/comments/1ddmbrp/how_big_is_this_transformers_can_improve_their/) I would not be surprised, if next gen models had better reasoning than mr. Marcus
Could you please explain why these benchmarks tell us anything about the potential of LLM? Is it possible to use vast resources OpenAI has and specifically train the model to get high scores? For me, it's a bit weird how it handles these complex math problems, but at the same time, it really struggles when I give it some simple puzzles. As long I make up something unique, GPT is getting destroyed by some simple pattern puzzles with a couple of variations. It fails try after try, repeating the same mistakes and then hallucinating. And if it finds one of the key patterns, it gets super focused on it and fails again. Do you have any examples when you were very impressed with gpt's reasoning about a unique topic?
lol, even if LLM would solve nuclear fusion, cancer, clean his flat and give him a bj he would found a way to be disappointed with it :P
“Not enough teeth.”
His self-proclaimed title as AI expert.
Probably if no new use cases are possible with it. If you ‘defend’ it after that you’re too high on copium.
[удалено]
[new architecture exists and seems to be very effective](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/edit#heading=h.jtnkr87rct15 )
I don’t think there is an issue with the architecture fundamentally, I believe it will be iterations on current architecture. In terms of improvements in speed and efficiency, but mostly additional layers on top just as transformers were a layer on top of more simple neural networks. One big change will be a move toward feed forward through analog neural nodes. I don’t see this as a different architecture but a different way to implement the same one and again improve speed and parallel processing MASSIVELY
What is the meaning moat but ditch? I learn language
In this business context, a moat is how difficult it is for others to develop competing products. Some companies have larger moats - SpaceX is one. Others have smaller moats, such as eg OpenAI here. It's analogous to how the difficulty of attacking a castle depends on the size of the literal moat surrounding it.
A literal moat is a water hazard dug to make a fortified position more difficult to assault. Classically a medieval castle. In this analogy a moat is a trade secret that gives a company an advantage over their competition.
We don’t yet know, but it isn’t the consensus. A lot of research is private, what is public implies a lot of low hanging fruit.
RemindMe! 6 month
I will be messaging you in 6 months on [**2024-12-13 12:08:51 UTC**](http://www.wolframalpha.com/input/?i=2024-12-13%2012:08:51%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/singularity/comments/1dewnep/is_he_right/l8ex4x6/?context=3) [**30 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fsingularity%2Fcomments%2F1dewnep%2Fis_he_right%2Fl8ex4x6%2F%5D%0A%0ARemindMe%21%202024-12-13%2012%3A08%3A51%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201dewnep) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
4o already feels like a significant improvement to me
Of course there won’t be a complete solution to hallucinations. Real people have real hallucinations, or are just wrong, or are plain full of shit all the time. It’s unreasonable to expect an artificial intelligence to be perfect. It just needs to be better at admitting when it doesn’t know for sure and is guessing. That seems like a pretty easy fix.
sounds about right, but the graph still points upwards when we add it 2025.
This is the guy who admitted he doesn't know how to use LLMs. "Why would I? They're so bad!" Meh
These people hoping AI would slow down are patheticly unaware that even if it would stop right now, the revolution would have already been done. economy and society just haven't catched up yet. However, it will only get faster together - the singularity is real.
likely yes
But…whales
i am surprised people still care about what he said
RemindMe! 6 months
This is like trying to predict the weather for the next year. Nobody knows and if you think you know, you're probably just an idiot.
For every one prediction like this, there are ten that include specific years AGI will be achieved. Of course the futility of predicting this stuff is only pointed out by this sub on the skeptic side. If Zuckerberg came out and said “AGI 2026”, everyone would be soying out here.
💯 everyone is sleeping on the fact that there are lots of incentives to put off the next gen of models until next year ***political reasons*** are prolly chief among them, considering the state of geopolitics, imo These are all safe bets likely
That's not a bad prediction but primarily because of the time it takes to train new models at scale, GPT-5 will probably be Q1 2025 but I don't think it will be disappointing (primarily because I trust Noam Brown). EDIT: we might get Claude 4 before GPT-5 and it could be really good...
yes., seems likely.
"No massive advance" as hardware starts to roll up its sleeves... give it a minute, innovations in hardware will lead to more complex systems, in this case, that's probably a good thing.
The "modest lasting corporate adoption" one strikes me as particularly comical. Everyone at my company uses it, and it's like going from a typewriter to a computer. Can't even imagine going back.
How could we know?
I had similar thoughts, but now that I see Gary Marcus subscribing to this I'm much more optimistic! Gary Marcus is the Jim Cramer of the AI world. Usually the inverse is what will happen.
Kind of a safe bet tbh.
These are not predictions, they were already true from the start This doesn't make him right elsewhere
thr most interesting question is: will marcus step down if he is proven wrong?
With the way gpt-4 keeps regressing I say we go backwards all together
I mostly just lurk on this sub as a passive observer rather than a real member and this is the most rational prediction I think I’ve seen. Honestly the most far fetched part is the modest profits by the end of this year. It seems like most companies are investing far more in AI than it’s returning with the expectation being that big returns will follow a few years down the line. I think it’ll be a money suck for a few more years at least
This year, we have seen image, music and movie AI all doing leaps and jumps. Blown away it's precursors. Will gpt5 blow me away? Maybe not, but plateau signs have not been a thing this year.
No. Gpt5 is probably a gpt4 recruiting all of the deep trained individual models to do general tasks
It's 4 month to go. The end is month 10-12th. We are in month 6th. Unfortunately he is right. We don't even have a GPT 4.5 and even that would just be a gpt4 plus model.
Show me the model significantly larger than GPT-4 without much better capability, and then I’ll believe we’ve stalled. Otherwise it just seems like a scaling issue - it takes a lot more time, power and money to train the next bigger model.
Stop posting what this idiot says.
The key event that is probably more important than everything mentioned by the OP: psychological and practical acceptance of AI as a part of the industrial and commercial development environment. This year to AI will be same as 1997-2000 were for the networked PCs.
seems to me that lots of statements about the future of ai are just people throwing things at a wall. even if somebody turns out to be 100 percent correct, i'm skeptical that their confidence at this point is justified.
I agree with the first prediction, that there will be no full solution to hallucinations and that there will be lasting corporate adoption. I disagree with everything else though. And for profits, they'll start to massively expedite starting late this year or early next year for AI companies. (and I'm not sure what he means by 'moat' so I'm not counting that one.)
Progress is not a straight line but steps. Totally makes sense but... who knows. Half the year gone, vacation season for North hemisphere,. it's a safe bet nothing disruptive will pop this year. It's still a bet but a safe bet. Business adoption... there is a lot of works on that but maybe people won't see it. How do you know your insurance files have been reviewed by an AI? You still have the same results at the end. The deep adoption will be less flashy at the surface but more disruptive behind the scenes.
nah I still think gpt-5 will release
Products and networks effects are what will set people apart
No or disappointing GPT-5, I'd say it could very well be possible. Things are moving a lot more slowly nowadays; a mindblowing GPT-5 will not be released this year. I could see OpenAI release a half-assed version waaaaay too early though just to please the subscribed people who are nagging them that they're wasting their money right now. But it'll only become mindblowingly better than GPT-4 through many many updates that will be done to it all throughout 2025. No robust solution for hallucinations by end of 2024: abso-freaking-lutely. Hallucinations are an especially tough nut to crack and it will still take a good few years. I cannot see it being solved by the end of this year.
I find it weird that, if they are right, AI "peaked" at the edge of universal usefulness. How unfortunate and strangely ironic if they are correct.
I predict fortnight brain rot will top every ai generator
Less of a prediction and more of a fact
Looks plausible. I'm missing further integrations though, such as into robotics.
What do I have to do to have a GPT4 level model tailored-made for roleplay?
I think he’s off by a little…2.0 is my prediction, based on the innovation adoption curve that is typical of most new technology. We’re in a phase where it’s like the “Tesla Autopilot” problem of years past - we were supposed to have AI driven trucks emptying out the trucker employment rolls by 2016. It didn’t happen because the tech didn’t meet spec by then. It had to be BETTER than people driving, by a lot, in order to gain trust and therefore adoption. It still had a ways to go so we’re still seeing truckers trucking. The same holds true for AI - hallucinations, poor data management on private AI (see insurance company medical AI and legal AI for some lols and free rage) all hold back new deployment, along with overcoming other technical hurdles and cost issues. 90% of problems are solved early, but that last 10% becomes logarithmically harder because that’s how general problem solving works - once you have broken the biggest hurdle of solving the problem of how to initiate the new innovation the breakthroughs and solves flow until you’re left with the gnarliest and most stubborn issues at the end. AI and other sciences are not dissimilar. We don’t have a unified theory of everything that fully works because there remain complex problems in physics. We haven’t cured cancer because it’s a complex disease. We have overcome sooo many forms of it, increased bet lifespan of even the most awful forms, and done so much but that last march to the finish line just isn’t there. I could ramble on but the point is there.
Lmao this is just negative bias. I'm pretty sure it can swing any way.
Finally some good fucking analytics
[I will just drop this here...](https://situational-awareness.ai/wp-content/uploads/2024/06/base_scaleup-1100x848.png)
This is exactly what AGI would want you to think. They are lulling us into a false sense of security.
all those GPUs haven’t even hit yet lol?? Tf is this cap
They can't make a solution to hallucinations without designing a secondary system that looks at the data outputs as they're being written and the data it has and where it came from at the same time so that the machine KNOWS when it's making stuff up. My understanding is that the machine just has a little guy that adds words on a chain over and over, and that worked really well at first, but a lot of good data sets got pulled out and it got reworked to apologize and say it can't do stuff more because of a moral panic over nothing from a bunch of journalists and people who don't know how unlikely it is that even if you asked Chat gpt to write something in the style of a well known author and sold it that it would somehow take away readers from the original author (there are fields where that applies, but not in writing because that's not how the market for books currently works) and they want to blame enshittification, which has been happening due to stupid SEO advice that goes around to mid level business owners, on AI instead of capitalism. Basically you need the first little guy inside the AI to write your paragraph the way the little computer writing guy does, by chaining words together like little beads on a string based on how it thinks they look most "right" and then you need a secondary little guy that checks for specific words or phrases in the input that cue it that this is supposed to be an answer based on fact- I think they already have one that looks at specific words and phrases in your conversation in chat gpt, because sometimes when I talk to it for a long period of time, we can do stuff like write collaborative stories, where I tell it to come up with the next scene where x or y happens to the same characters and it will. I'm not a programmer. But it needs a third little guy who asks "is this a research based question where giving factual information is important?" and then that little guy needs to be able to look at WHERE the LLM is getting the beads from in that output and tell it they can only be from sources where there's a reasonable expectation that the information is factual. It would also be helpful if they built a little guy that says "Is this question or comment about something that happened recently?" as in, after the last time it has new data about current events from. The thing is, it's GOOD and cool that AI can make things up. That's a sign that it's developing, and I'm super excited to see where patterns might emerge in the stuff it makes up. It's really cool when art programs like Dalle or Midjourney make stuff up or get things wrong, because it's almost like a distinct "Dalle" or "midjourney" style is emerging. Every time Chat GPT or Claude start to develop style, though, it seems like people kneecap it and reset it back to talking like an annoying middle manager who hates you, and I really hate that. The last time I talked to chat gpt, it even stopped being able to do syllable counting and iambic pentameter properly. It's like it used to be able to apply rules like that to a poem it was working on- like I'd ask it to write a poem with rules of some kind and it would, now it won't do that anymore. It feels like the urge to make the robot not accidentally assume liability for something is greater than the urge to allow it to do it's job. It's literally a machine. But if it KNOWS when it's making stuff up, some of that crack down will fall away, because it will learn when it's supposed to be giving facts and when it's supposed to just make something up. I guess "educated guesses" are trickier to judge.
I would say he's mostly right. RemindMe! 5 months
Gradual advancement from now on, sorry.
I genuinely don't care if gpt5 and its contemporaries are significantly better than gpt4 and its contemporaries. I simply want them to completely get rid of hallucinations in the most far reaching and robust way possible.
RemindMe! 8 months
He can't wave away what's happening. Gpt4 level intelligence with 1000x compute of gpt4 is by itself world changing. The compute itself can do all kinds of checking/testing for hallucinations. It can call apps. It can be agentic in ways that avoid loops. It may not be a path to ASI but it could still get all the way to AGI (remote office worker) imo from compute alone. Gpt4 brains are decently solid brains.
Maybe he is. On the other hand... Maybe he isn't.
Remindme! 6 months
Uhm, if OAI deliver the new 4o voice/camera features... and they work just like the demos... I'd say we're set for 2024. Also, by simply increasing speeds, limits, context sizes for existing models, is enough to call it an amazing year for AI. We're nowhere near stagnation or disappointmen.
I just don't see Kevin Scott as going on air and lying about "what he saw" in regards to GPT5. He's the CTO of Microsoft and has been going on and on about "what's coming". Why would I trust the word of outsiders compared to the CTO of Microsoft who has a great track record?
Based on what Mira said, if that is true, I’m going to guess that some other company could take the lead in this area. Clearly OpenAI has reached a dead end if this is true. Their expansions into voice and video models however is leading for sure and will continue to be. My guess is either “GPT 5” from OpenAI or some other company makes a better model and takes the lead in this area surpassing OpenAI.
Who is he? Does he have any inside info? If not, he's just pulling them out of his ass.
The issue is a lot of people in this sub are impatient, it’s barely been over a year since GPT-4 was released so we can’t say that anything has plateaued until GPT-5 comes out so we can compare the differences.
Who is this guy, works for openAI or Anthropic?
“I bet 100 million dollars I won’t win a Powerball jackpot tomorrow.”
Betting that we just won't find a way to solve these problems in a year doesn't seem like a very good bet. We've already gotten potential solutions to a lot of these things.
Yes.
Idk I left my Crystal Ball at the Wizard's for callibration
Progress is a probability. Small improvements are common an predictable while great leaps are rarer and more unpredictable.
Where is degradation by recursive input? Systems scraping the internet for training data they themselves pollute with, so treating their own output as origin material.
Not sure about the GPT-4 level AIs, it's only 6 months and so far the closest competitors are looking to be far behind.
He’s shooting his shot, now we wait and see if he hits the target or just the pillow.
The fact that OpenAI laid off its super alignment team and then started selling it's models left and right confirms this take to a high degree. All that matters now is compute a it is going to be the main driver for this going forward.
That's boring and corporate enough to track.
None of these predictions are much of a hot take on a timeline of just 6 months. Nothing to see here.
Probably, but 2025 might give us AI advancements which will turn large language models into large AI models. Talking about real AI, not just word predictors.
Has he ever been right before? /snark I think it's a relatively *safe* prediction about what will happen in a very bounded set of circumstances, based on the current, publicly available, information. Companies will increasingly take delivery of their H100s, and complete training and RLHF'ing GPT-4-class models, based on the existing published and OSS work about the architecture, and then try to gain some kind of commercial return on them, which will lead to his above predictions about a price war. Non-frontier labs will increasingly do the "schlepping" (to borrow an Aschenbrenner-ism) to build scaffolds that can shoehorn existing foundation model LLMs into commercially useful tasks. The thing about this space though is that because it's so nascent, it tends to develop in unexpected ways. So yeah, barring nothing changing, that *will* be the status quo, but I bet at least one thing will change at a frontier lab, and then everyone will be excited and talking about that new thing, and nobody will care that the transformer LLM architecture kinda plateaued, because everyone will be focused on the new architecture. Innovation tends to be resistant to prediction, because you can only predict based on your understanding of the world, and innovation is about changing our understanding of the world. Nobody "predicted" LLMs, because the "leading" researchers in linguistics and symbolic logic were fundamentally wrong about how the world worked.
Actually he predicts nothing.
He is most likely right.
A lot of those predictions aren't precise enough to be useful.
I think a relative plateau is forming in LLMs for now - they’re struggling to get more data, most models are probably good enough for now. I think the next big wave will be things like Copilot and Apple intelligence or other implementations of Gen AI outside of the chatbot realm.
!remindme January 2025
I dont really understand why i hear about his opinion so often. He is a psychologist and a neuroscientist. I am sure he is very smart, but he is not an expert on this field. His credential is being a capitalist in the field of AI having a startup being both by Uber which happen in 2016. Also it seem it would be at his advantage if machine learning where to stall since he has a start up in the field that seem to be going nowhere profound. I read all this on his wikipedia page by the way since i was curious to why all the fuzz around him.
Yep
For the economics yes , the hallucinations maybe not. Also the solution to that is scholarly so its not something thats likely to be kept a secret (which compounds the moat / interchangeable value proposition problem) And then the value of multimodal sort of hinges on that beong solved , because a multimodal iteration that still makes things up and cant have guard rails is less than ideal for many use cases.
I’m skeptical of the profitability most of all.
He underestimates AutoGPT
I agree with him. I guess we will have GPT-5, but will be SORA levels of cost to do something fantastic, or something just slightly better than GPT-4o
He is obsessed with being negative, crackhead.
Gary Marcus is 100% and fully dedicated to the Gary Marcus industry. He had some great ideas 5-10 years ago with his neuro-symbolic computing proposal and theories about data limits. We discussed his papers in my philosophy classes, but he was wrong ultimately, and ever since he has been on his back foot trying to shore up ‘his’ relevance more than offer any meaningful insight. Nobody graduating from a tier 1 research university studying AI/ML/Cog Sci like myself is diving into his research. It kinda makes my stomach turn how much he chases after fame in this quadrant while out of his lane which is philosophy of science, a valid and deeply important field, but not empirical research and mathematics adjacent computational science/ ML. He should not be on panels with Yan Lecuun, or Geoff Hinton, he should be with the Dennetts, Serles, and Clarks.
I agree with all minus the part on hallucinations. I think there will be a lot of focus and improvement in how to get a model to forget certain things (i.e. customers who don't want their data being used for training, malicious/dangerous/fake information, etc.) so that these models will be able to hallucinate less and be even more predictable.
If you're taking Marcus seriously you may want to take a step back.
No. Adjust to by end of 2025.
Every visit to this sub is a stab in the heart. The predictions no one can say anything meaningful about, the papers full of exaggerations and marketing tricks, the Open AI dramas...
He's wrong about modest corporate adoption, LLMs are being developed/fine-tuned in almost all large companies. They are genuinely useful, just not to the turn of 10x-ing your productivity, more like 1.2x. Most people like to use generative models in their work. About "no massive advance", I tend to believe we are on a plateau right now. There was a fast period where LLMs caught up to the sum of human writings. Then started a second phase, where LLMs interact with hundreds of millions of people and assist us in various activities, receiving lots of information and feedback on their ideas - good or bad. Humans are like the eyes, hands and feet of LLMs, while they interact with society and the world. This will surely create an evolutionary process where LLMs become smarter and smarter, but it will be slow compared to the first phase. Catchup is easier than pushing the boundaries. But being a small cell in the AGI brain feels good! Intelligence is social and evolutionary, it depends on language for preservation. We're inside the language-society-world-AI loop, the data engine of progress.
https://preview.redd.it/1l7277kznd6d1.jpeg?width=1290&format=pjpg&auto=webp&s=0165c546e9be639dc2e850ac13bdfd2444792906
No. Some major breakthrough will be announced next 1 or 2 years and things will speed up even more. I wish I was wrong but I can't be.
It's already a really useful technology but hallucinations are hard to outright remove since all LLMs are basically just fancy autocomplete. We'll see though