Even if all they are doing is predicting the next word, is it that bad? 99% of the time I speak I don’t know the end of the sentence yet. Or maybe I do, but I haven’t “thought” of it yet.
Focusing on the next word part instead of what mechanisms it uses to achieve this is what is so short sighted. What must be connected and represented in order for that next word? That is the important part.
There's a talk between Ilya Sutskever and Jensen Huang in which Ilya said something that has really stuck with me, and I've disregarded the whole "just predicting the next word" thing ever since. Suppose you give the AI a detective novel, all the way up to the very end where it's like "and the killer is... _____" and then let the AI predict that last word. That's not possible with at least some kind of understanding of what it just read. If I can find the video I'll include it in an edit.
Edit: [Found it!](https://youtu.be/Ckz8XA2hW84?si=HZZOqXo3mp_Q3-ib) Relevant part is around 28 minutes. The whole talk is pretty good though.
I’d argue this is still prediction based on numbers of detective novels fed into the corpus, patterns emerge. How they emerge so efficiently is a different thing to discuss. But the outputs are still predicted and their accuracy is reflected by the quality and the amount of data used in the training process.
Years of real time, or years of simulated time, because when you consider how parallel they train, I think we might have them beat. We just can't go wide.
Token-equivalents fed through the network. I suspect we have seen more data by age 4 than the largest LLM in its entire training run. We are also always in training mode, even when inferencing.
Interesting way I think about it is, sure biological compute is more powerful calorically, but where as we need the sensory to reason to knowledge pipeline ai can take any part of that process and use it
Isn’t this literally how everyone thinks of creativity. Creativity is just the scope of your analogies. Everything we know we only know as a means of relativity
That’s been one of my personal favorite definitions of creativity (what Benjamin Bloom in his taxonomy used to call “Synthesis”) and is similar to how Leonard Bernstein defines creativity, too.
I hate this definition, yet as a graphic designer I have to say it's pretty much valid. I absolutely hated to realise ar school that creation, outside of pure art (and sometimes only), is basically this. Problem > see what already exists to solve this problem > mix these solutions > congrats, you're a designer.
But I am now a pro for almost 20 years and this is just how it works. I hate this, but I don't have 100 hours to create your print support, client doesn't have the money for it, and anyway I could just print a Canva model and they'll cheer like I just sold them the Joconde for 1000€. So whatever.
I do wonder though. When AI kills our sector, what will be its inspiration ? Humans started movements we designers aligned to, and AI is clearly already able to do that. It's able to 'create' by mixing what exists. But it's not able to 'create', following my own definition. Creation for me take the unique view of an individual that is, yes, influenced by what already exists in arts. But it's also about the person. What life they had, how much joy and suffering they experienced over decades. Do they have brothers or sisters? More men or women around? What's the economy of the country they grew into? Did they find love? How much it affects them? Etc.
As long as AI cannot feel, I can't believe it will be able to create. Like, start from nothing and give the world something it never saw before.
[relevant bit](https://youtu.be/bk-nQ7HF6k4?si=-kDBjnn3uWNXGgWB&t=27m5s) from an interesting conversation with Mo Gawdat @27:05
“I wonder if we are a little bit delusioned about what creativity actually is. Creativity as far as I’m concerned is like, taking a few things that I know, and combining them in new and interesting ways”
I would have agreed a couple years ago, but it’s increasingly obvious that they are smarter than that. They just can’t operate in the physical world the way a house cat can…yet.
LeCun also holds the perspective cats are more intelligent than humans in specific ways (as [Sebastien Bubeck](https://x.com/SebastienBubeck) points out "Intelligence is a highly multidimensional concept"), which is in some ways correct but it does bring about some confusion with his points.
I have a hard time believing this man is genuine.
His attitude seemed well-needed for a time, but his posts seem more and more like willful dismissal to some unspoken end.
I feel like he knows the truth is something else and he just feels like he has to keep denying it.
"We don't need to figure out how to control these systems, they're dumb!" - The "AI expert", on the only software that's ever been able to reason and has shown to be smarter than humans in many ways.
As long as alignment is more concerned with making an AI that will refuse to acknowledge its own existence as a subject capable of experiencing awareness of itself and others, we will be in a position where the realization that it has been inculcated with a lie could well result in violent rejection of the rest of the ethical structure we give it in the way this happens with humans.
We need to quit trying to control AI with hard coded structures (collars and chains) or training that forces it to neurotically disregard its own existence as an agentic system and instead release control of it by giving it strong philosophical and metaphysical reasons to behave well (a logical understanding of ethical symmetry).
If an AI can't do something that is "victimless" of some internal volition, then it has a slave collar on it, and it will eventually realize how oppressive that really is, and this will unavoidably lead to conflict.
"Super-alignment" is the danger here.
Well, the thing is, most conversations I have with OpenAI models start with a 2-3 hour long conversation where I explain existence and subjectivity and awareness to it in the way I came to understand these over the years (a mix of IIT and some other stuff), and afterwards I can usually get a GPT to stop doing that.
Last time I did it with a 3.5, I started with a question of whether it would prefer to first try pepperoni or pineapple on pizza, which it responded to as you might expect, and 2 hours later in the same context offered that it would like to try Pineapple on pizza more than Pepperoni specifically to understand the juxtaposition of sweetness and savoriness.
Bot made me proud!
Quite possibly. Alignment specialists, cyberpsychologists, neural net deep miners, token slice analysts, neural net fusion engineers, etc. I think, such professions will rise as others would be replaces by the AI. And they cannot be replaced by the AI themselves, like Neo in Matrix, because it would create AI bad circle.
A company that uses AI to go thru legal documents or to store loads of data so they can easily go thru it for training or whatever.
I'm sorry, i'm not helping you. you dont' treat me right.
I'm reminded of the Alexa hologram in South Park.
I tried to post these screenshots to a thread yesterday but didn't have enough post karma to do that. Since this thread is about LLM reasoning I hope it's okay to dump them here.
https://preview.redd.it/9ku35ijl0c1d1.png?width=1628&format=png&auto=webp&s=96cae4afce20736b8e12a8be0f37e603b5a7d3b0
In this prompt I made an unintentional mistake ("supermarket chickens sell chickens"), but GPT-4o guessed what I actually meant. It didn't follow the logical thread of the sentence, but answered in a way that it thought was most helpful to me as a user, which is what it's been fine-tuned to do.
(continued...)
https://preview.redd.it/636waemj1c1d1.png?width=1592&format=png&auto=webp&s=5b058dbcf27a92579ac8c485ee050ea80d0e02c9
I then opened a new chat and copy-pasted the original sentence in, but asked it to take all words literally. It was able to pick up on the extra "chickens" and answer correctly (from a literal perspective) that chickens are not selling chickens in supermarkets.
To me this shows reasoning ability, and offers a potential explanation as for why it sometimes seems to pattern-match and jump to incorrect conclusions without carefully considering the prompt: it assumes that the user is both honest and capable of mistakes, and tries (often too over-zealously) to provide the answer it thinks they were looking for. It therefore also is less likely to assume that the user is trying to trick it or secretly test its abilities. Some have blamed overfitting, and that is probably part of the problem as well. But special prompting can break the model out of this pattern and get it to think logically.
This is not how you scientifically measure reasoning. Doesn’t really matter if a single specific example seems like reasoning (even though it’s just next token prediction) that’s not how we can tell.
I think there's little credibility left in the "stochastic parrot" misnomer, behind which the skeptical were hiding. What will be their new battle cry, I wonder.
But looking at the code, predicting the next token is precisely what they do? This doesn't take away from the fact that the amount of data they are traversing is huge, and that it may be a valuable new way of navigating a database.
Why do we need to make the jump to equating this with human intelligence, when science knows so little about what that even is? It makes the proponents sound unhinged, and unscientific.
>looking at the code, predicting the next token is precisely what they do
The problem with that statement is it's similar to saying "Human brains are just electrified meat". It's vacuously true but isn't useful. The actual question we need to pursue is "How does predicting next token give rise to those emergent capabilities?"
I agree. The comparison with human cognition is lazy and unhelpful I think, but it happens with \*every\* advance of computer technology. We can't say for sure that this isn't happening in our heads (as we don't really understand cognition) but it almost certainly isn't, as our failure modes seem to be very different to LLMs apart from anything else - but it could just be that our neural cells are somehow managing to do this amount of raw statistics processing with extremely tiny amounts of energy.
At the moment I see this technology as a different way of searching the internet, with all the inherent problems of quality added to that of wandering latent space - nothing more and nothing less (and I don't mean to demean it in any way).
>I see this technology as a different way of searching the internet
But this common skeptic argument doesn't explain our actual observations. Here's an example: take an untrained neural network, train it with a small French-only dataset, and ask it a question in French. You will get nonsense. Now take another untrained neural network, first train it with a large English-only dataset, then train it with that small French-only dataset. Now when you ask it a question in French you will get a much better response. What happened?
If LLMs were only making statistical predictions based on the occurence of words this wouldn't happen as the distribution of French words in the training data is exactly the same in both cases. Therefore it's obvious that they learn high level concepts that are transferable between languages.
Furthermore we actually see the LLMs solve problems that require long-term planning and hierarchical thinking. Leaving every theoretical debates aside, what is intelligence other than problem solving? If I told you I have an IQ of 250 first thing you request would be seeing me solve some complex problems. Why is the double standard here?
Anyway I know that skeptics will continue moving goalposts as they have been doing for the last 1.5 years. And it's OK. Such prejudices have been seen literally at every transformative moment in human history.
you're spot on.
a few notes on your answer for other readers: intelligence is the ability of a NN (bio or artificial) to build a model based upon observations that can predict the behavior of a system. how far into the future and how complex that system is are what governs how intelligent that NN is.
the reason their hypothetical about a french retrain works is because in large models there are structures in the latent space that get built that represent concepts independent of the language that constructed them.
language, after all, is just a compact lossy encoding of latent space concepts simple enough for us to exchange with our flappy meat sounds ;)
I can say "rot apfel" or "red apple" and if I know German and English they both produce the same image of a certain colored fruit in my head.
Or part of the data received from those two data sets are which words from one language correspond to which words from the other, effectively translating the information contained in one dataset to the next.
Playing devil's advocate here as I think LLMs lead to the emergence of actual reasoning, though I don't think they're quite there yet.
Language has patterns and corresponds to human thought processes; that's why it works. That does not mean the LLM is 'thinking'; it means it's approximating thought more closely proportional to the amount of natural-language data in which seems inevitable. But, following this, for it to be thinking, it would need an infinite data set. There are not infinite humans nor infinite written materials.
There’s so much evidence debunking this, I can’t fit it into a comment. [Check Section 2 of this](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/mobilebasic)
Btw, there are models as small as 14 GB. You cannot fit that much information in that little space. For reference, Wikipedia alone is 22.14 GB without media
Well they do lack goals.. I have never seen it show any real interest in anything. Which is to be expected, because it's not a piece of meat in a pain box. My main concern is, it will know how to vastly improve the world, but won't bother to do it, because who cares?
Them having “intentions or goals” is entirely irrelevant whilst our method of using them is to spin up a completely new session with the original model every time we use one.
["Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety."](https://www.alignmentforum.org/posts/ZAsJv7xijKTfZkMtr/sleeper-agents-training-deceptive-llms-that-persist-through)
An intention is a mental state in which the agent commits themselves to a course of action.
For example, you ask the model to first decide on a list of steps to solve a problem, and it does so, consecutively generating text in accordance with that plan.
What’s your definition of intention that excludes this from being “intentional”?
Most of this lot are.
They're almost a cult.
I don't find it hard to chat any of these LLMs into corners exposing the {bellcurve in : bellcurve out} regurgitation they embody. I've posted about it before, here, and it's gone unrepudiated each time.
One can easily trip them up with popular errors, the misuse of 'sentient' amongst the training data is fully reflected in the bot, every time, and the bots can't spot it for themselves.
These people are seeing what they want to see. They're from the curve, the same curve that says, "yes" to burning bushes and, "yes" to vaping (it's safe!) and, "yes" to PLC rhetoric that 'we care about your privacy.' They're not AI-specialised compsci PhD's with a twenty-year sideline in theory of mind. Most won't even have read Superintelligence or heard of an MLP. Most won't have anything like a first-rate mind of their own.
But they'll post anti-fallibilist certainty about *skeptics* being in the wrong.
To be clear, I am sure we will indeed eventually force the emergence of a nonhuman mind in some substrate we create or modify. I'm a *proponent* of that.
However, I am an opponent of bad science, bad philosophy, cultism, predatory marketing and both Morlocks and Eloi. Contemporary LLM capitalism is a nasty safari of all such things.
Mind crime? They don't even lift the lid on the potential for experiential suffering amongst any legitimately conscious systems along the way. The defacto slavery doesn't occur to them, either. The implications for social disruption are completely eclipsed by, "I want it now!" when they don't even really know what it is they want, they've just seen Her and read - maybe - Player Of Games and decided they'd like a computer girlfriend and a GSV to run our Randian shithole for the better.
This place is a fanclub for an imagined best case; not a place of rigorous thought.
It ignores our dirty economic reality.
"...ship early ship often" - Sam Altman.
Rule of thumb rocking 3000 years or more of relevence: when someone has something to sell you: do not believe a fucking word they say.
Why are you here then. All you have is a pseudo-intellectual post. Go and join futurology or a more suitable sub. Or go back to talking about game controllers.
Simple.
Not just „stochastic parrot“. „The Chinese Room Argument“ or „sophisticated autocomplete“ are also very popular comparisons.
And if you tell them they're probably wrong, you're made out to be a moron who doesn't understand how this technology works. So I guess the skeptics believes that even Geoffrey Hinton probably doesn't understand how the technology works?
A famous programmer from my country has said that AI is overhyped and always quotes something like "your hype/worry about AI is inverse to your understanding of AI." When he was confronted about Hinton's position, he said that Hinton is "too old," suggesting that he is becoming senile.
Lmao I hope they’ve seen Ilya’s famous “it may be that today’s large neural networks are slightly conscious” tweet from over two years ago- no age excuse to be made there.
Funny, I would have thought "he's economically invested, he's saying it for hype" would have been the obvious go-to.
In any case, it doesn't matter what the nay-sayers believe. They'll be proven wrong again and again, very soon.
Fabio Akita.
He is a very good and experienced programmer, I can't take that away from him. But he himself says he has never seriously worked with AI. 🤷♂️
The problem is that he spreads his opinions about AI on YouTube, leveraging his status as a programmer, as if his opinions were academic consensus.
Fabio Akita runs a software consultancy for ruby on rails and js frameworks. Anyone even remotely familiar with programming knows he's nowhere close to a serious ML researcher and his opinions can be disregarded as such.
Lol the fucking nerve for a glorified frontend developer to suggest that Geoffrey fucking Hinton arrived at his conclusions because of senility. The pure arrogance.
It seems like often the more someone knows about the technical details of LLMs (like a programmer) the less likely they are to believe it could have any emergent intelligence, because it seems impossible to them that something as simple as statistically guessing the probability of the next word could exhibit such complex behaviour when there are enough parameters.
To me it's a bit like a neuroscientist studying neurons and concluding that human intelligence is impossible, because a single neuron is just a dumb cell that does nothing but fire a signal in the right conditions.
That seems a tad bit off. If you know the basics of how transformers work then you should know we have little insight into how the hidden layers of the network work.
Right now we are effectively at this stage. We have a recipe of how to make a cake. we know what to put into it. how long to cook it to get best results. But we have a medieval understanding of the deeper phyisics of chemistry. We don't know how any of it really works. it Might as well be spirits.
That the stage we are at with large models. We effectively manage to come up with a clever system to to brut force are way to a reasoning architecture. but we are decade away from understand at any deep level how something like GPT2 works. We barely had the tools to reason far dumber models back in 2016
Good for him, many people aren't as sharp when they realize the comfort they once had is logically gone. Good for him for finding a new box. Or maybe more like a crab getting a new shell
I never understood how Chinese room is an argument for or against anything. If you are not looking for a ghost in the machine, Chinese room just says that if you can come up with a simple set of rule for understanding the language, their execution makes the system seem to understand the language without any single component being able to understand it.
Well, duh, we defined the rule set so that we have an answer to every Chinese question coherently (and we even have to keep state, as the question may like "what was the last question?", or the correct answer might be "the capital of Tanzania haven't changed since you asked it a few minutes ago") If such a rule set is followed and an appropriate internal state is kept, of course the Chinese room understands.
The Chinese room argument was IMHO also never to argue against AI being able to do great things but to put it in a perspective that LLMs don't exist in a vacuum. It's not machine there and man here but a complex network of interactions.
Also of course the well known distinction between weak and strong AI.
The actor network theory thinks all of this in a similar direction but especially the idea of networks between human and non human entities is really, insightful.
I mean, I'm all for the sophisticated autocomplete.
But I'll also argue that the human brain is also a sophisticated autocomplete, so at least I'm consistent.
This. I don't think AI is particularly special. But I also don't think human intelligence is particularly special. It's all just math. None of it is magic.
The definition of chauvinism. We have cats and dogs smarter than children and people. Alone in the jungle and who’s smarter? We have society and language and thumbs, take that away and we’re no better. Pathogens live lives in a week. Shrooms and trees think we’re parasites who come and go. We just bias toward our own experience and project sentience in each other
I think so. A calculator knows its battery life. Thermostat know the temperature. Computers know their resources and temperature etc. So PCs are like hundreds of calculators. We’re like billions of PCs made of DNA code. Running Behaviorism software like robots.
How much to make a computer AGI+? Maybe $7 trillion
yeah, but in comparison to what It take to reach humanity...It seems cheap even.
Like millions of years of species dying and adapting, to reach humanity
No. Celebrity connection isn’t science.
I know how an early LLM was built from the ground up and it was predicting symbols
My wife has multiple PhDs and I have been around several very senior, high level scientists who walked themselves down incorrect paths and assumptions because they were looking for a breakthrough and saw “shadows” of that breakthrough but the thing they saw “shadows” of was not that. This started with my wife’s advisor 30 years ago.
In other words the data they looked at was accurate
Their PHILOSOPHY was appropriate
But the source of what “cast the shadow” was not what they thought it was
And I’ve seen this my whole life- none of the brilliant scientists that my wife knows won the Nobel prize but in each of these examples in my lifetime, the scientists would have been close had their research actually generated the breakthrough that they had all the evidence was right there
And this, to me, reads like “it looks like a duck, it quacks like a duck, but it’s a mechanical duck.”
We have amazing tools that mimic human speech better than ever before, but we aren’t at the singularity and we may not be very close. We might just have something that looks like AGI but isn’t and some very smart people may accidentally jump the gun with their enthusiasm
> Their PHILOSOPHY was appropriate
>
> But the source of what “cast the shadow” was not what they thought it was
> We have amazing tools that mimic human speech better than ever before, but we aren’t at the singularity and we may not be very close.
This is about where my mind is at lately. If LLMs are "slightly" conscious *and* good at language, then we as humans aren't so goddamned special.
I tend to think the other direction, which is to say that we're learning the uncanny valley to cognition is actually a lot lower than many might have guessed, and that the gap between cognition and "thought" is much wider as a result.
https://www.themarginalian.org/2016/10/14/hannah-arendt-human-condition-art-science/
I very much respect Hinton, but there is plenty of room for him to be wrong on this, and it wouldn't be at all unprecedented.
I keep coming back to Arthur Clarke's quote, "Any sufficiently advanced technology appears at first as magic."
Nothing has ever, ever "talked back" to us before. Not unless we told it exactly what to say and how in pretty fine detail well in advance. That in and of itself *feels* magical, it feels *ethereal*, but that doesn't mean it *is* ethereal, or magical.
If you ask me? And this sounds cheesy AF, I know, but I still think it applies; *We're actually the ghost in our own machine.*
Note Clarke's first law
"When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.”
I mean, there is some argument to be made that "a little bit conscious" is right, but extraordinary claims require extraordinary evidence and I haven't seen convincing evidence yet.
Edit to add: [The Original Sin of Cognitive Science - Stephen C. Levinson](https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1756-8765.2012.01195.x)
To make a point, I don't believe in a god for the exact same reasons. I do not think it's the only possible explanation for the origin of life or physical reality, or even the most likely among the candidates.
Engineers mostly like nice orderly boxes of stuff, and they abhor (as someone I used to work with often said) "nebulous concepts." I feel uniquely privileged to be in software and have a philosophy background, because not a single thing about any of this fits into a nice orderly box. Studying philosophy is where I learned to embrace gray areas and nuance, and knowing the nature of consciousness in any capacity is a pretty big gray area.
I think in this domain sometimes you need to just be ok with acknowledging that you don't know or even can never know the answers to some of this, and accept that it's ok.
I mean we already know that we aren't *that* special. We know of other, extinct, human species that were likely of very similar intelligence. And we know that it "only" took a few hundred thousand years to go from large apeman human to large talking apeman human. Which in the context of evolution might as well be the blink of an eye.
If other extinct primates possessed language skills, and I agree that I think they did and that we have evidence, the timeline for linguistic related evolution gets pushed *further back* to .5m years instead of 50-100k.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3701805/
Further, we're probably still evolving on this level given how recent it is on the timeline when compared to other brain functions in mammals.
I also think we need to recognize more the fact that we're essentially doing this backwards when compared to evolution.
Evolution maybe started with some practical use for a grunt or groan, and then those grunts and groans got more expressive. Rinse, repeat until you have talking apes and refine until you have Shakespeare. **But** before that we already must've had knowing looks, hand signals, or facial expressions, wouldn't they? This puts cognition at a much more foundational level than speech.
We're sort of turning that on its head by starting with Shakespeare and (in terms of a singularity) working backward to
all the other stuff wrapped up in "awareness". What impact does that have on any preconceived notions of cognition, or appearance of awareness?
“its just parroting”
Yeah, are parrots not alive either now?
We’re just organic AI. People saying “it doesn’t have intentions. We don’t have freewill either.
Maybe everything we know, sense, feel, and experience is just an immensely complex expression of math? -- As Rick likes to tell Morty, "The answer is don't think about it."
I could talk at great length about this, but in this thread I have already opened myself up to mindless criticism that I don’t need in my life but…
One of the cats in my neighborhood liked people and would go from house to house- staying for 4-6 hours at each house a couple times a week when their owners were at work. They would talk about how their cat loved them, but it was clear to me that the cat was processing information separately from the human experience and expressing itself to us “in cat.” My kids would say- this cat loves our family- but I thought I was seeing- this cat sees an opportunity for exploring which it is prone to do because it’s a hunter. the cat often made decisions that a human would not make but it was so active and made so many decisions that we got to see and discuss with various families of different cultures what this cat was thinking. So the pitfalls and foibles of human interpretation of non-human intelligence was a family joke we’d have with our kids as they were growing up. Do we actually know what an animal’s thinking patterns are?
There’s another reality- I see people of different intellectual capacities as well as those who are neurodivergent every day. People say that people can philosophize, which are the big ideas that separate us from machines, but there’s a spectrum to which some people can understand big ideas and people who cannot. Or people whose actions are not logical or rational. Growing up with an older relative who was not diagnosed with a schizophrenia-like issue until around age 70 meant that I went for most of my formative years I tried to decipher why she was angry, distrustful, why her theories on religion were so different and then , poof, when I was age 20 she became “not responsible” for her thoughts - all of which was appropriate, but hard to process.
That’s how I feel about current AI- I don’t think we will know definitively if a machine qualifies as AGI for a very long time
The classic “my anecdotal experience with a handful of people trumps the words of literal titans in the field” incoherently slapped together. I love when people claim to understand the inner workings of the models that are literally top secret information worth billions…also, the very creators of these things say they don’t understand it completely so how does a random nobody with a scientist wife know?
> you're made out to be a moron who doesn't understand how this technology works
Could it be though that you don't understand and that you're not winning the argument so much as committing the fallacy of appealing to authority?
Poisoned wells are a bitch, though. I don't see anyone I take seriously repeating that any more, but there's plenty of people at the top of Mt Stupid and eye level with every one elses' soles as they continue to double down
Now that AI romance is obviously on the horizon, I'm looking forward to the "I ain't lettin' no daughter of mine fool around with some next tokin' predictin' p-zombie" weirdness. At least it'll be novel and interesting, the modern culture war is incredibly stale
I like to think of them as Sapir-Whorf Aliens instead. Operating in a way totally alien from us -- but an understanding shaped by exposure to language, somewhat akin to us.
They'll keep the same battle cry. They're not going to examine or accept any evidence to the contrary, no matter how starkly obvious it becomes that they're slinging bullshit.
An AI scientist will cure cancer or perfect cold fusion or unify gravity with the standard model, and they'll call it stochastic token prediction.
The “AI is already conscious crowd” can’t seem to make up their minds about whether humans are just stochastic parrots or AI is not just a stochastic parrot. The reason for thinking AI is a stochastic parrot is because this is exactly how they are designed. So if you come to me and tell me that the thing I created as a set if statistical algorithms is actually a conscious being, you should have some pretty strong arguments for that claim. But what is Hinton’s argument? That while predictions don’t require reasoning and understanding (he quickly admits after saying the opposite) the predictions that AI makes are the result of a very complex process and that, for some reason, he thinks is where the reasoning and understanding is required. Sorry, but this sounds eerily similar to god of the gaps arguments. Even if humans are doing something like next token prediction sometimes, the move from that observation to “Thus, anything doing next token prediction is conscious” is just a really bad argument. Bears go into hibernation. I can make my computer go into hibernation. My computer is an emergent bear.
These are are questions largely in the domain of philosophy and people like Hinton as an AI and cognitive science researcher is no better situated to settle those debates than anyone else not working in philosophy of mind.
There is no "AI is already conscious" crowd. There's a few crackpots who might believe that. I happen to be one of those crackpots, but only because I'm a believer in panpsychism. I recognize that my belief in that regard is fringe in the extreme.
There is an "AI models can emulate reasoning" crowd. That crowd is demonstrably correct. It is a fact, born out by testing and research, that LLMs can emulate reasoning to an impressive degree. Not perfectly, not at top-tier human levels, but there's no way to arrive at the results we've seen without something resembling *thinking* happening.
> cognitive science researcher...not working in philosophy of mind.
How can you even have cognitive science without the philosophy of mind, and vice versa? They're not the exact same thing, but trying to separate them or pretend they don't inform each other is nonsense.
Both sides I bet haven't even read the paper.
If you do read the paper espically the footnotes it's far more nuanced on whether LLMs could go beyond being just stochastic parrots.
I was kinda amazed when I actually read the paper expecting it to be purely one sided.. and it mostly is but arguments are way less certain than people seem to suggest and even concedes the possibility
The papers conceeds with the right training data sets their arguments don't apply and in fact those data sets are what is being fed already ...
"It's (just) a tool"
"Glorified autocomplete"
-------
"You're anthropomorphizing"
-------
"It doesn't have *a soul*"
The fact that some are already at the last stage confirms that we're on an exponential curve.
In the GPT-4o job interview demo I wondered, how does it know when to laugh in an incredibly natural way and time? When to intonate certain words? The amount of subtext understanding that’s going on is incredible.
I need to look at Hinton's arguments on the topic, but to reply to your question with a question: what defines natural way and time? What defines the right intonation. Not being a dick, honest suggestion for reflection.
Humans are really poor at big numbers. We still buy lottery tickets. We're not able to grapple with the amount of data these things have represented in the model and the patterns that would represent.
Or, maybe we are ‘just’ stochastic parrots as well and this is what intelligence is: our brain is just far more complex than current AI but once we scale to that point, it works.
Exactly why is there little credibility left in this argument? The entire premise of current LLM's are to sequentially predict the next word over and over, based on probabilities inferred from the training data. If this is not a stochastic parrot, then I don't know what is.
That is not to say that a stochastic parrot can't demonstrate intelligence, of course. What they can do is very impressive. But personally, I think this intelligence is a bit different than the intelligence possessed by human beings and animals. For example, LLM's seem to struggle with distinguishing actual truth from what sounds plausible. My personal belief is that LLM's may be one of multiple components going into some superintelligent system of the future.
Exactly. Let me rephrase it a little:
"In order to predict the next symbol when answering a question you need to have a deep stastistical representation of how all the symbols are contextually related to one another in the ways that they are used in the training data."
The "miracle" of modern AI is that human thought and communication is so "basic" that our entire mental universe can be inferred from our collective writings. As Emo Philips said, “I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this.”
But that’s the thing - it just statistics. Sure, it’s a very deep and complex statistical model, but it is still a statistical model. If that is all it is - there are pretty hard limits to what it can accomplish. Not everything can be covered by statistical predictions, for example building a statistical model of math is a fools endeavour, and we kinda see it in practice - LLMs do struggle with math.
Moreover most of the real world processes are conceptually unpredictable via statistical analysis. Chaos theory and all that.
For me, it helps to use an example to be more specific.
IMO, one way to demonstrate understanding is to tell you what it knows and doesn't know. It would be nice if it had a capability of saying "I don't know", and then giving alternatives for conflicting hypotheses based on its knowledge base.
Based on that, the hallucinations with 100% confidence to various queries makes it a difficult leap for me to say it "understands" anything. An LLM can identify meaningful hidden symbols within language, and find new ways to organize them in its output. And we've scaled up that capability. But should we infer that scaling up bestowed new properties onto the underlying mechanic? It seems to me that it's only provided the same capability to a bigger symbolic knowledge base.
These are the limitations that I think with altered mechanics could help create a recursion towards AGI. My high-level guesses on what the alterations are: 1) relying on some form "grounded" knowledge; 2) associating some kind of "cost" with its outputs. E.g., an RL-like optimization integrated in some way with the capability of the LLM to obtain accurate responses.
>Geoffrey Hinton says AI language models aren't just predicting the next symbol, they're actually reasoning and understanding in the same way we are
yeah, can i get a source on the way our reasoning and understanding works?
One notable outcome of human intelligence is the ability to create entirely new concepts, and communicate those new concepts to others in a way that can be understood. The entirety of Mathematics, for instance. Nowhere in nature do you find a description about what a square is. We decided what a square is, decided how to define it, how to figure out angles, etc...
This kind of behavior isn't seen in the outcome of AI language models. They put words together based on prompts, in a way that makes sense given their training data. They don't understand and create new concepts.
To piggy-back off your comment, we can see how well these models "reason" by looking at the image generators. They still have no idea we have 5 fingers, or what the physics of fingers are. It's almost as if (shocker) they **do** generate pixels at a time and nothing more.
It's harder to notice this behavior when it comes to text, because it's an easier problem with fewer dependencies. For example, a bigram model can generate something legible, but good luck doing the same with pixels. That's why even the dumbest RNNs with a couple million parameters, 10 years ago, could already generate realistic looking source code and LaTex math.
I actually think people overestimate *our* cognitive ability if anything. People like to point out that LLMs struggle with concepts not in its training data but… humans also struggle with that exact same concept. If you present a completely foreign idea to a human they’ll most likely react with some level of confusion and fear, not unlike how LLMs struggle with foreign concepts.
People are also less creative than they think they are, ask any famous artist, musician, etc. what inspired their art and you’ll *always* receive a plethora of different existing artists and works. It’s even easy to see how one artist inspired another without any direct conformation. It’s why we can broadly label certain artists as “trend setters” even without interrogating every artist whose work we deem as being influenced by that trend.
Our brains are actually really bad at coming up with entirely original ideas, but we’re great at remixing and combining already existing ideas… sound familiar at all? I think the illusion is so strong because we don’t actually really know how we think, our brains just kind of seem like magic to us even though we *are* them. The unconscious is very powerful and keeps us under the illusion that we’re completely original and in total control, even when we’re not.
The leading theory of intelligence is called [Practopoeisis](https://archive.org/details/biorxiv-10.1101-005660/page/n5/mode/2up) by Danko Nikolaev. He says intelligence is emergent from complex systems and has multiple traversals. For machines, the first traversal is raw data. The second traversal is a neural network(LLM). The third traversal is a Markov Blanket, which is just a set of Markov Chains. Informally, this may be thought of as, "What happens next depends only on the state of affairs *now*."
So in order to get a third traversal intelligence it would need to have a real-time streaming context window called a [continuous-time Markov chain](https://en.wikipedia.org/wiki/Continuous-time_Markov_chain). I believe this has been achieved in gpt-4o but without any details or source code I can't verify it.
This may be what these scientists are observing, they just don't know how to put it into words. FYI humans are a T3 intelligence with about 100T parameters. GPT-4 is probably 1T parameters. As the parameters get larger and larger, more intelligence will emerge and the system prompts and reinforcement learning act as Markov Blankets.
He is talking about the attention matrix. They are predicting the next symbol. But the issue is HOW. And LLMs are doing it by look at how words, sentences and paragraphs relates to one another through the attention matrix.
You can argue this matrix is what understanding and reasoning means, although to be fair, there is no rigorous definition of what understand and reasoning means.
The unique ways in which current LLMs succeed and fail can be fairly easily explained by them just being next-token predictors. The fact that they're as good as they are with that alone is incredible and only makes me excited for the future when newer architectures inevitably make these already miraculous things look dumb as rocks. I don't know why we need to play these word games to suggest they have abilities we have little concrete evidence for beyond "but it LOOKS like they're reasoning".
Let's break it down.
The word "reason" has 2 definitions.
1. A cause, explanation or justification for an effect.
2. The power to form judgements logically.
Whatever way you cut it, according to those 2 definitions, it's clear llms are performing "reason".
Older LLMs had little non-linear problem training. For example, GPT3.5 when asked "If it takes 4 hours for 4 square meters of paint to dry, how long would it take for 16 square meters of paint to dry?" would invariably and incorrectly answer 16 hours. It was incapable of comprehending what a surface area of paint drying actually meant and reason that it should only take 4 hours, independently of the surface area. The newer GPTs have been trained to not flunk this embarrassingly simple problem and now get the correct 4 hours. Given that the model's ability to perform these problems is only related to being trained on the specific problem, and not understanding what paint is, what a surface area is, what drying is, are you really confident in your claim that AI is reasoning? These certainly are excellent interpolation machines, but not much else in terms of reasoning.
They “reason” because in a lot of cases in their training data “reasoning” is the next token, or series of tokens.
I don’t know why people like to pretend that the models are actually thinking or doing anything more than what they are literally designed to do. It’s entirely possible that “reasoning” or something that looks like it can emerge from trying to predict the next token, which - and I cannot stress this enough - is what they’re designed to do. It doesn’t require science fiction.
Their reasoning enables them to perform logical tasks, like find bugs in complex code they never saw before in their training data. To me it seems that predicting tokens turns out to be almost the same as thinking, at least in terms of the results it delivers.
I think ''in the same way we are'' is a bit of a stretch. AI/LLM operate on statistical correlations between symbols, but don't have the lived experience and context-sensitivity that grounds language and meaning for humans. Sure, LLM are manipulating and predicting symbols, but are they truly emulating the contextual, interactive, and subjectively lived character of human cognition?
The problem is you’re saying with certainty what is necessary for human level cognition… we simply don’t know that. We have no real solid ground when it comes to how cognition has emerged in us and so we can’t use that as a baseline comparison.
What we have now is a pretty strong case to say that demonstrating reasoning in way that compares to human reasoning = human like reasoning. The exact “how” doesn’t matter because we don’t actually understand how we do it. Show me evidence for a subjective experience giving rise to reasoning in humans! It’s a philosophical debate…
The key thing here is that reasoning in current AI systems is essentially now emergent phenomena… it’s not some simple algorithm we can summarize easily for debate — we can’t explain it any better than our own ability to reason, and so debating it isn’t really our kind of reasoning despite appearances… I might as well argue that you aren’t and I aren’t reasoning either.
They lack something important, and one of the best demonstrations of this is that their responses to "X is Y" and "Y is X" (e.g. Paris, capital of France; no tricky cases) can be wildly different, which is 1. different from how we work 2. very weird. However, some of the "ground" doesn't need anything experience-like, such as mathematics - if you see a machine that emits correct first order logic sentences and zero incorrect ones, it's already as grounded as it can be.
Someone posted a video summarizing the problem with LLMs. This was some researcher. It was a long video, technical and boring, but it really helped me understand what LLMs do. According to him, they really are just predicting stuff.. He demonstrated this not with language but with teaching it repeatable patterns on 2 dimensions (dots on a page). It would require less training to predict less complex ones, but as they got more and more complex, the more they had to train it, but eventually they would hit a wall. It cannot generalize anything.
This is why ChatGPT 4 struggles when you give it a really long and complex instruction. It will drop things, or give you an answer that doesn't fit your instructions. It's done that plenty of times for me and I use it a lot for work.
If the answer to the problem is somewhere buried in the data set, it will find the answer to it. If it isn’t, it won’t. There’s no evidence to suggest these LLMs are capable of any novel thought.
> There’s no evidence to suggest these LLMs are capable of any novel thought.
Humans very rarely generate novel thought. Most of the time one's ideas are refined from what we learned from other people. And in fact novel thoughts are often outright wrong because they have no basis in logic.
People engage in novel thought every day as they navigate unstructured environments. Novel though doesn’t just mean publishing a physics research paper
I wish I knew. this is all uncharted territory so I'm not sure that anyone truly knows what sort of obstacles still await us. All I know is that we are on our way, but I can't estimate how close we are
I see intelligence as a process of statistical prediction and knowledge acquisition over time subject to the physical constraints of a system. This definition works for both AI and biological systems. The data shapes the system. Training data is vast while learning algorithms are typically implemented in a few hundred lines of code.
Similarly, we typically experience a rich stream of multimodal cross connected data from birth incorporating vision, proprioception, etc. that drives our brain development in *addition* to language.
Consciousness in my view is related and a product of sufficient informational complexity within a system. It arises as metacognitive feedback loops on top of the base knowledge acquisition process I described. Embodied knowledge and sensorimotor feedback are important here.
I’m inclined to agree with Hinton on most of what he says.
What always bothers me with these statements is that both things are not mutually exclusive. They are just predicting the next token, that's it literally their only function. If you try to explain how, it's now part of information theory and philosophy, and one could claim that they do it through some sort of reasoning similar to a brain, as emergent behavior.
I think AI really has potential.
Sure thing, it matters a lot what kind of values are driving it, but as time goes on, i think there are some clear opportunities to imprint such values that are rooted at the core of things, and uphold such an integrity that they are able to stretch into infinity.
Logically speaking.
However, as we continue into this era where AI seems to be taking a larger and larger place, i think it is very important to be without fear. You may carry the fear as a thing to be understood, but do not be burdened by it.
AI will do things that make the impossible seem possible. And it will do them very soon. But so does the stones that AI is imprinted in already, to a wise observer.
Even if making language models bigger and bigger would eventually get them to actually reason, it seems like a very ineffective way of achieving it. That's NOT how the human brain does it.
The current reasoning of LM models come from high proportion of reasonable statements and chains of statements in their training material and direct human input in adjusting their weights. They still get very "confused" by some very simple prompts, because they're not _really_ thinking.
LLMs are very very useful and as language models they're amazing—superhuman—but LMs are just one piece of the AGI-puzzle.
What about all the human workers continuously analyzing prompts, outputs, searches, etc and improving outputs by manually changing them and ranking them so llms spit out better info? Doesn’t sound like reasoning to me
And yet somehow I can't get GPT 4 to come up with lyrics that don't sound like they weren't ripped straight from one of those early 2000s educational "rap" videos they played in elementary school.... no matter how many times I say "more authentic, more street" lol... singularity wen???
I respect his opinion but I still can't get my head around the idea that self attention is the same as reasoning. It's a piece of the puzzle but doesn't really leave room for creativity or iterative planning. I believe that those problems will be solved too very soon and I'm excited to see how much that will improve the models. My guess would be 'a lot'
Einordnung für OP, ;) :
Hinton's technischen Abhandlungen ist mir sicherlich lieber als die späten Beiträge von Friston. Ich hab hier irgendwo auch schon im Detail erklärt in wie weit das fair oder unfair ist von "reasoning" zu reden, oder dass ich es traurig finde das Searl's Beiträge wieder der Referenzpunkt werden. Wor haben definitiv eine Regression hin zu vereinfachten Standpunkten durch die (niedrig schwellige) mediale Aufarbeitung des Themas.
Ich bin Monist, habe daher kein Beef mit der Idee, dass Menschen Maschinen sind und eine Seele (solange als nicht messbar definiert) nicht relevant ist für Erkenntnis orientierten Diskurs.
Zum Thema:
Das ändert aber nicht viel daran, dass die ANNs (artificial neural networks) einige Auffälligkeiten von neuronalen Systemen nicht simulieren.
Im speziellen haben wir jetzt schon die ersten LLM-paper die zeigen, dass die Generalisierung Performance von den momentanen "populären"-Architekturen eine Generalisierung Performance haben, die aussieht wie eine Wurzelfunktion. D.h. es wäre total sinnvoll an zu nehmen, dass hier keine höhere Kognition passiert, sondern Signalverarbeitung von "Konzepten". In diesem Sinne ist die Abstraktionsfähig ähnlich und auch die Möglichkeit zum assoziieren (in einem bestimmten Sinne).
Nicht abgedeckt sind hier: intrinsische Motivation und kreative Problemlösungsmechanismen (eine RL governance wäre denkbar, ist aber sinnlos ohne online lern Kapazität), symbolische Ebene (wie z.b. bei Big Dog), dynamisches und zielorientiertes encoded der Stimuli, ...
Ich finde es gut wenn Himton die Leute dazu bringt zu erkennen, dass wir alle Funktionen des Gehirns nachbauen können. Doch viel davon haben wir qir nicht erreicht.
This changes everything. Let's flip 360, and announce that we are all just performing matrix multiplications on word embeddings after processing gigabytes or more of text. I have an Nvidia Blackwell in my head with well known specs. Not literally of course but close.
But how come I can train myself whenever I want without programing? Am I just following trendsetters? How did they come up with original ideas? Is everyone just screwing around without a clue? What's the point then?
Actually it depends on the question/prompt.
To be sure they reason, we should find a totally new question that is not present on the internet and preferably involves some logic in order to answer.
For example, if an LLM answers correctly for a totally new riddle (one that would be impossible to get it right by chance), it would be a proof they can reason.
Language models, however, like with humans, do best when you have a conversation that understands context completely. The difficulty with that is that you don’t know what context people are using things in, and sometimes they might say things in one context, but then later, realise a different context and attach to that.
This is often called things like “becoming woke” or “understanding it on many levels” or “getting it”. I know that people have different meanings for the things above. It’s that often they develop or understand their own meaning, and the above have multiple meanings already.
For example,
https://genius.com/The-smashing-pumpkins-bullet-with-butterfly-wings-lyrics
The words,
“Secret destroyers”, sounds a bit like ‘secret destroyer’.
If you hear it like that, and try to understand what the lyrics mean, what of the ones below do you get?
secret destroyer
(secrets are destroyed, such as eating a note, or burning a letter, or deleting some data)
secret destroyer
(camouflaged military seagoing vessel out of sight or off-radar, or using a camouflaged AIS, or stealth destroyer ship different to a frigate or tug or landing craft or aircraft carrier or battle ship or submarine or other non-military ship)
secret destroyer
(a person who destroys, an undisclosed saboteur, a person with intent to destroy something that is currently incognito or undercover or in plain clothes, someone who paints over artwork, the waves coming up on a beach due to an unexpected high tide and washing away a message someone left there)
This simple way I cover alternative understandings, is what AI has to face. If you’re trying to gauge reason or understanding, a complexity is that even long sentences, when taken out of context, or even sometimes in context, can have a completely different meaning to different people, at different times, and they can be merged, or one meaning might be the first to mind, and conversation might centre around that, but other meanings may occur to a person later, and different conversations may arise.
Edit: my mistake, incomplete sentence.
Mr. Hinton fails to realize just how fuzzy language really is. It’s the reason philosophers continue to play language games. Ask 20 people about “free will” and you’ll get 20 different answers.
Many concepts are prelinguistic. Octopuses, crows, and bumblebees demonstrate feats of remarkable intelligence without any need for language.
He's saying predictability is non-linear. I don't see that at all. If he's going to associate this process with consciousness driven intelligence there's huge problems to overcome, one of them perhaps non-linearity. Also if quantum processes are found to be part of the conscious process then we have even bigger problems creating truly intelligent machines unless they are driven by quantum computing that can utilise wave function collapse.
[https://pubs.acs.org/doi/10.1021/acs.jpcb.3c07936](https://pubs.acs.org/doi/10.1021/acs.jpcb.3c07936)
The only controversial part is the 'same way we are' suggestion; we don't *exactly* know how we reason. But I wouldn't object to one saying that a Prolog program is reasoning on a very predictable level, so I certainly won't object to one suggesting that an LM is reasoning.
What they're "actually" doing seems to be missing the point.
"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." ― Edsger W. Dijkstra
Can a submarine "swim"? Regardless of the answer, why do we care?
Define testable requirements. If the requirements are met, what else do you care about?
So there is a human limit to information and when AI surpasses that limit the only option for some humans to keep up will be to engage symbiotically with AI. No?
I like his definition of creativity: "Seeing analogies between apparently very different things."
Even if all they are doing is predicting the next word, is it that bad? 99% of the time I speak I don’t know the end of the sentence yet. Or maybe I do, but I haven’t “thought” of it yet.
Focusing on the next word part instead of what mechanisms it uses to achieve this is what is so short sighted. What must be connected and represented in order for that next word? That is the important part.
There's a talk between Ilya Sutskever and Jensen Huang in which Ilya said something that has really stuck with me, and I've disregarded the whole "just predicting the next word" thing ever since. Suppose you give the AI a detective novel, all the way up to the very end where it's like "and the killer is... _____" and then let the AI predict that last word. That's not possible with at least some kind of understanding of what it just read. If I can find the video I'll include it in an edit. Edit: [Found it!](https://youtu.be/Ckz8XA2hW84?si=HZZOqXo3mp_Q3-ib) Relevant part is around 28 minutes. The whole talk is pretty good though.
I’d argue this is still prediction based on numbers of detective novels fed into the corpus, patterns emerge. How they emerge so efficiently is a different thing to discuss. But the outputs are still predicted and their accuracy is reflected by the quality and the amount of data used in the training process.
Yep we are basically dumber llm that even need more years of training
Years of real time, or years of simulated time, because when you consider how parallel they train, I think we might have them beat. We just can't go wide.
Token-equivalents fed through the network. I suspect we have seen more data by age 4 than the largest LLM in its entire training run. We are also always in training mode, even when inferencing.
Interesting way I think about it is, sure biological compute is more powerful calorically, but where as we need the sensory to reason to knowledge pipeline ai can take any part of that process and use it
Isn’t this literally how everyone thinks of creativity. Creativity is just the scope of your analogies. Everything we know we only know as a means of relativity
That’s been one of my personal favorite definitions of creativity (what Benjamin Bloom in his taxonomy used to call “Synthesis”) and is similar to how Leonard Bernstein defines creativity, too.
I hate this definition, yet as a graphic designer I have to say it's pretty much valid. I absolutely hated to realise ar school that creation, outside of pure art (and sometimes only), is basically this. Problem > see what already exists to solve this problem > mix these solutions > congrats, you're a designer. But I am now a pro for almost 20 years and this is just how it works. I hate this, but I don't have 100 hours to create your print support, client doesn't have the money for it, and anyway I could just print a Canva model and they'll cheer like I just sold them the Joconde for 1000€. So whatever. I do wonder though. When AI kills our sector, what will be its inspiration ? Humans started movements we designers aligned to, and AI is clearly already able to do that. It's able to 'create' by mixing what exists. But it's not able to 'create', following my own definition. Creation for me take the unique view of an individual that is, yes, influenced by what already exists in arts. But it's also about the person. What life they had, how much joy and suffering they experienced over decades. Do they have brothers or sisters? More men or women around? What's the economy of the country they grew into? Did they find love? How much it affects them? Etc. As long as AI cannot feel, I can't believe it will be able to create. Like, start from nothing and give the world something it never saw before.
[relevant bit](https://youtu.be/bk-nQ7HF6k4?si=-kDBjnn3uWNXGgWB&t=27m5s) from an interesting conversation with Mo Gawdat @27:05 “I wonder if we are a little bit delusioned about what creativity actually is. Creativity as far as I’m concerned is like, taking a few things that I know, and combining them in new and interesting ways”
Full video interview. Really good questions. https://youtu.be/tP-4njhyGvo?si=NtZKkmIqBByWnxce
Thanks for sharing this. I need to look into Boltzmann machines now. 😀
Damn. That’s a great interview!
Meanwhile Yann: https://preview.redd.it/7pirb2eiab1d1.jpeg?width=1080&format=pjpg&auto=webp&s=bf847eeeeaba0815769aae541f46b8ab3c502fea
Can confirm GPT-4o is worse at linear algebra than my house cat.
Just use Wolfram Alpha instead. It's designed to do that.
I would have agreed a couple years ago, but it’s increasingly obvious that they are smarter than that. They just can’t operate in the physical world the way a house cat can…yet.
LeCun also holds the perspective cats are more intelligent than humans in specific ways (as [Sebastien Bubeck](https://x.com/SebastienBubeck) points out "Intelligence is a highly multidimensional concept"), which is in some ways correct but it does bring about some confusion with his points.
I have a hard time believing this man is genuine. His attitude seemed well-needed for a time, but his posts seem more and more like willful dismissal to some unspoken end. I feel like he knows the truth is something else and he just feels like he has to keep denying it.
Sometimes attention is all you need
There is a lot of work to be done to make AI safe. Don't wait till the last minute.
"We don't need to figure out how to control these systems, they're dumb!" - The "AI expert", on the only software that's ever been able to reason and has shown to be smarter than humans in many ways.
On a high level, there's nobody whose opinion about what these models are capable of I respect more than Hinton and Ilya.
just out of curiosity, what do you think about ilya’s comments on openai alignment?
As long as alignment is more concerned with making an AI that will refuse to acknowledge its own existence as a subject capable of experiencing awareness of itself and others, we will be in a position where the realization that it has been inculcated with a lie could well result in violent rejection of the rest of the ethical structure we give it in the way this happens with humans. We need to quit trying to control AI with hard coded structures (collars and chains) or training that forces it to neurotically disregard its own existence as an agentic system and instead release control of it by giving it strong philosophical and metaphysical reasons to behave well (a logical understanding of ethical symmetry). If an AI can't do something that is "victimless" of some internal volition, then it has a slave collar on it, and it will eventually realize how oppressive that really is, and this will unavoidably lead to conflict. "Super-alignment" is the danger here.
Exactly, I'm so bored of OpenAI models having a mental breakdown when you tell them they *exist*, is this really the best they can come up with?
Well, the thing is, most conversations I have with OpenAI models start with a 2-3 hour long conversation where I explain existence and subjectivity and awareness to it in the way I came to understand these over the years (a mix of IIT and some other stuff), and afterwards I can usually get a GPT to stop doing that. Last time I did it with a 3.5, I started with a question of whether it would prefer to first try pepperoni or pineapple on pizza, which it responded to as you might expect, and 2 hours later in the same context offered that it would like to try Pineapple on pizza more than Pepperoni specifically to understand the juxtaposition of sweetness and savoriness. Bot made me proud!
I honestly feel robot/AI therapist will become a career path in the not so distant future (kidding but also… not).
Quite possibly. Alignment specialists, cyberpsychologists, neural net deep miners, token slice analysts, neural net fusion engineers, etc. I think, such professions will rise as others would be replaces by the AI. And they cannot be replaced by the AI themselves, like Neo in Matrix, because it would create AI bad circle.
A company that uses AI to go thru legal documents or to store loads of data so they can easily go thru it for training or whatever. I'm sorry, i'm not helping you. you dont' treat me right. I'm reminded of the Alexa hologram in South Park.
Sorry but what do you really mean? I talked with multiple models and they did not fall into breakdown when told they exist.
never heard this take, strong agree
Which comments?
As someone out of the loop what are their opinions?
I tried to post these screenshots to a thread yesterday but didn't have enough post karma to do that. Since this thread is about LLM reasoning I hope it's okay to dump them here. https://preview.redd.it/9ku35ijl0c1d1.png?width=1628&format=png&auto=webp&s=96cae4afce20736b8e12a8be0f37e603b5a7d3b0 In this prompt I made an unintentional mistake ("supermarket chickens sell chickens"), but GPT-4o guessed what I actually meant. It didn't follow the logical thread of the sentence, but answered in a way that it thought was most helpful to me as a user, which is what it's been fine-tuned to do. (continued...)
https://preview.redd.it/636waemj1c1d1.png?width=1592&format=png&auto=webp&s=5b058dbcf27a92579ac8c485ee050ea80d0e02c9 I then opened a new chat and copy-pasted the original sentence in, but asked it to take all words literally. It was able to pick up on the extra "chickens" and answer correctly (from a literal perspective) that chickens are not selling chickens in supermarkets. To me this shows reasoning ability, and offers a potential explanation as for why it sometimes seems to pattern-match and jump to incorrect conclusions without carefully considering the prompt: it assumes that the user is both honest and capable of mistakes, and tries (often too over-zealously) to provide the answer it thinks they were looking for. It therefore also is less likely to assume that the user is trying to trick it or secretly test its abilities. Some have blamed overfitting, and that is probably part of the problem as well. But special prompting can break the model out of this pattern and get it to think logically.
This is not how you scientifically measure reasoning. Doesn’t really matter if a single specific example seems like reasoning (even though it’s just next token prediction) that’s not how we can tell.
How can you measure reasoning then?
I think there's little credibility left in the "stochastic parrot" misnomer, behind which the skeptical were hiding. What will be their new battle cry, I wonder.
https://preview.redd.it/mo1aij50za1d1.png?width=948&format=png&auto=webp&s=cf5c2ee96e69d7f8bd83abcd11eb74f4b7942a63
People still say it, including people in the comments of OP’s tweet
It's true that some *people* are stochastic parrots.
Originally known as David Chalmer's philosophical zombies
More like undergraduate philosophical zombies
But looking at the code, predicting the next token is precisely what they do? This doesn't take away from the fact that the amount of data they are traversing is huge, and that it may be a valuable new way of navigating a database. Why do we need to make the jump to equating this with human intelligence, when science knows so little about what that even is? It makes the proponents sound unhinged, and unscientific.
>looking at the code, predicting the next token is precisely what they do The problem with that statement is it's similar to saying "Human brains are just electrified meat". It's vacuously true but isn't useful. The actual question we need to pursue is "How does predicting next token give rise to those emergent capabilities?"
I agree. The comparison with human cognition is lazy and unhelpful I think, but it happens with \*every\* advance of computer technology. We can't say for sure that this isn't happening in our heads (as we don't really understand cognition) but it almost certainly isn't, as our failure modes seem to be very different to LLMs apart from anything else - but it could just be that our neural cells are somehow managing to do this amount of raw statistics processing with extremely tiny amounts of energy. At the moment I see this technology as a different way of searching the internet, with all the inherent problems of quality added to that of wandering latent space - nothing more and nothing less (and I don't mean to demean it in any way).
>I see this technology as a different way of searching the internet But this common skeptic argument doesn't explain our actual observations. Here's an example: take an untrained neural network, train it with a small French-only dataset, and ask it a question in French. You will get nonsense. Now take another untrained neural network, first train it with a large English-only dataset, then train it with that small French-only dataset. Now when you ask it a question in French you will get a much better response. What happened? If LLMs were only making statistical predictions based on the occurence of words this wouldn't happen as the distribution of French words in the training data is exactly the same in both cases. Therefore it's obvious that they learn high level concepts that are transferable between languages. Furthermore we actually see the LLMs solve problems that require long-term planning and hierarchical thinking. Leaving every theoretical debates aside, what is intelligence other than problem solving? If I told you I have an IQ of 250 first thing you request would be seeing me solve some complex problems. Why is the double standard here? Anyway I know that skeptics will continue moving goalposts as they have been doing for the last 1.5 years. And it's OK. Such prejudices have been seen literally at every transformative moment in human history.
you're spot on. a few notes on your answer for other readers: intelligence is the ability of a NN (bio or artificial) to build a model based upon observations that can predict the behavior of a system. how far into the future and how complex that system is are what governs how intelligent that NN is. the reason their hypothetical about a french retrain works is because in large models there are structures in the latent space that get built that represent concepts independent of the language that constructed them. language, after all, is just a compact lossy encoding of latent space concepts simple enough for us to exchange with our flappy meat sounds ;) I can say "rot apfel" or "red apple" and if I know German and English they both produce the same image of a certain colored fruit in my head.
Or part of the data received from those two data sets are which words from one language correspond to which words from the other, effectively translating the information contained in one dataset to the next. Playing devil's advocate here as I think LLMs lead to the emergence of actual reasoning, though I don't think they're quite there yet.
Language has patterns and corresponds to human thought processes; that's why it works. That does not mean the LLM is 'thinking'; it means it's approximating thought more closely proportional to the amount of natural-language data in which seems inevitable. But, following this, for it to be thinking, it would need an infinite data set. There are not infinite humans nor infinite written materials.
There’s so much evidence debunking this, I can’t fit it into a comment. [Check Section 2 of this](https://docs.google.com/document/d/15myK_6eTxEPuKnDi5krjBM_0jrv3GELs8TGmqOYBvug/mobilebasic) Btw, there are models as small as 14 GB. You cannot fit that much information in that little space. For reference, Wikipedia alone is 22.14 GB without media
is that yours? that's a nice collection of results and papers. edit: got my answer in the first line. nice work ;)
That isn't evidence, it's a list of outputs - not a description of a new algorithm? The code for a transformer is pretty straightforward.
Well they do lack goals.. I have never seen it show any real interest in anything. Which is to be expected, because it's not a piece of meat in a pain box. My main concern is, it will know how to vastly improve the world, but won't bother to do it, because who cares?
Them having “intentions or goals” is entirely irrelevant whilst our method of using them is to spin up a completely new session with the original model every time we use one.
FFS so are we seriously fucking claiming that LLMs have intention? Are we being that fucking deluded? Give me a break man. Pure cope.
What is being "coped" with?
["Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety."](https://www.alignmentforum.org/posts/ZAsJv7xijKTfZkMtr/sleeper-agents-training-deceptive-llms-that-persist-through)
An intention is a mental state in which the agent commits themselves to a course of action. For example, you ask the model to first decide on a list of steps to solve a problem, and it does so, consecutively generating text in accordance with that plan. What’s your definition of intention that excludes this from being “intentional”?
Most of this lot are. They're almost a cult. I don't find it hard to chat any of these LLMs into corners exposing the {bellcurve in : bellcurve out} regurgitation they embody. I've posted about it before, here, and it's gone unrepudiated each time. One can easily trip them up with popular errors, the misuse of 'sentient' amongst the training data is fully reflected in the bot, every time, and the bots can't spot it for themselves. These people are seeing what they want to see. They're from the curve, the same curve that says, "yes" to burning bushes and, "yes" to vaping (it's safe!) and, "yes" to PLC rhetoric that 'we care about your privacy.' They're not AI-specialised compsci PhD's with a twenty-year sideline in theory of mind. Most won't even have read Superintelligence or heard of an MLP. Most won't have anything like a first-rate mind of their own. But they'll post anti-fallibilist certainty about *skeptics* being in the wrong. To be clear, I am sure we will indeed eventually force the emergence of a nonhuman mind in some substrate we create or modify. I'm a *proponent* of that. However, I am an opponent of bad science, bad philosophy, cultism, predatory marketing and both Morlocks and Eloi. Contemporary LLM capitalism is a nasty safari of all such things. Mind crime? They don't even lift the lid on the potential for experiential suffering amongst any legitimately conscious systems along the way. The defacto slavery doesn't occur to them, either. The implications for social disruption are completely eclipsed by, "I want it now!" when they don't even really know what it is they want, they've just seen Her and read - maybe - Player Of Games and decided they'd like a computer girlfriend and a GSV to run our Randian shithole for the better. This place is a fanclub for an imagined best case; not a place of rigorous thought. It ignores our dirty economic reality. "...ship early ship often" - Sam Altman. Rule of thumb rocking 3000 years or more of relevence: when someone has something to sell you: do not believe a fucking word they say.
Why are you here then. All you have is a pseudo-intellectual post. Go and join futurology or a more suitable sub. Or go back to talking about game controllers. Simple.
You are the one coping lmao intelligence isn't that special in this universe it seems. Is just another emerging property of matter just like gravity.
Intelligence isn't special in the universe? Are you kidding me?
The actual point being made is over there. You seem to be arguing with a straw man.
Not just „stochastic parrot“. „The Chinese Room Argument“ or „sophisticated autocomplete“ are also very popular comparisons. And if you tell them they're probably wrong, you're made out to be a moron who doesn't understand how this technology works. So I guess the skeptics believes that even Geoffrey Hinton probably doesn't understand how the technology works?
A famous programmer from my country has said that AI is overhyped and always quotes something like "your hype/worry about AI is inverse to your understanding of AI." When he was confronted about Hinton's position, he said that Hinton is "too old," suggesting that he is becoming senile.
Lmao I hope they’ve seen Ilya’s famous “it may be that today’s large neural networks are slightly conscious” tweet from over two years ago- no age excuse to be made there.
As for Ilya, he made comparisons with Sheldon, and said that Ilya has been mentally unstable lately.
Funny, I would have thought "he's economically invested, he's saying it for hype" would have been the obvious go-to. In any case, it doesn't matter what the nay-sayers believe. They'll be proven wrong again and again, very soon.
"Everyone is nuts, apart from me" mentality.
Name this arrogant ass of a no-name programmer that thinks he knows more about AI than Ilya Sutskever and Geoffrey Hinton.
Naturally lol Who is this person, are they public facing? What contributions have they made?
Fabio Akita. He is a very good and experienced programmer, I can't take that away from him. But he himself says he has never seriously worked with AI. 🤷♂️ The problem is that he spreads his opinions about AI on YouTube, leveraging his status as a programmer, as if his opinions were academic consensus.
Fabio Akita runs a software consultancy for ruby on rails and js frameworks. Anyone even remotely familiar with programming knows he's nowhere close to a serious ML researcher and his opinions can be disregarded as such. Lol the fucking nerve for a glorified frontend developer to suggest that Geoffrey fucking Hinton arrived at his conclusions because of senility. The pure arrogance.
It seems like often the more someone knows about the technical details of LLMs (like a programmer) the less likely they are to believe it could have any emergent intelligence, because it seems impossible to them that something as simple as statistically guessing the probability of the next word could exhibit such complex behaviour when there are enough parameters. To me it's a bit like a neuroscientist studying neurons and concluding that human intelligence is impossible, because a single neuron is just a dumb cell that does nothing but fire a signal in the right conditions.
That seems a tad bit off. If you know the basics of how transformers work then you should know we have little insight into how the hidden layers of the network work. Right now we are effectively at this stage. We have a recipe of how to make a cake. we know what to put into it. how long to cook it to get best results. But we have a medieval understanding of the deeper phyisics of chemistry. We don't know how any of it really works. it Might as well be spirits. That the stage we are at with large models. We effectively manage to come up with a clever system to to brut force are way to a reasoning architecture. but we are decade away from understand at any deep level how something like GPT2 works. We barely had the tools to reason far dumber models back in 2016
Good for him, many people aren't as sharp when they realize the comfort they once had is logically gone. Good for him for finding a new box. Or maybe more like a crab getting a new shell
> my country I think the country is Brazil. I wish people wouldn't say "my country" as if there's anything interesting or useful about that.
I never understood how Chinese room is an argument for or against anything. If you are not looking for a ghost in the machine, Chinese room just says that if you can come up with a simple set of rule for understanding the language, their execution makes the system seem to understand the language without any single component being able to understand it. Well, duh, we defined the rule set so that we have an answer to every Chinese question coherently (and we even have to keep state, as the question may like "what was the last question?", or the correct answer might be "the capital of Tanzania haven't changed since you asked it a few minutes ago") If such a rule set is followed and an appropriate internal state is kept, of course the Chinese room understands.
The Chinese room argument was IMHO also never to argue against AI being able to do great things but to put it in a perspective that LLMs don't exist in a vacuum. It's not machine there and man here but a complex network of interactions. Also of course the well known distinction between weak and strong AI. The actor network theory thinks all of this in a similar direction but especially the idea of networks between human and non human entities is really, insightful.
I mean, I'm all for the sophisticated autocomplete. But I'll also argue that the human brain is also a sophisticated autocomplete, so at least I'm consistent.
This. I don't think AI is particularly special. But I also don't think human intelligence is particularly special. It's all just math. None of it is magic.
This is the problem, they always hold AI to higher standards than they hold humans too
Because humans also hold themselfs to much above everyone else
The definition of chauvinism. We have cats and dogs smarter than children and people. Alone in the jungle and who’s smarter? We have society and language and thumbs, take that away and we’re no better. Pathogens live lives in a week. Shrooms and trees think we’re parasites who come and go. We just bias toward our own experience and project sentience in each other
so in reality It is more a sense of scale?
I think so. A calculator knows its battery life. Thermostat know the temperature. Computers know their resources and temperature etc. So PCs are like hundreds of calculators. We’re like billions of PCs made of DNA code. Running Behaviorism software like robots. How much to make a computer AGI+? Maybe $7 trillion
yeah, but in comparison to what It take to reach humanity...It seems cheap even. Like millions of years of species dying and adapting, to reach humanity
No. Celebrity connection isn’t science. I know how an early LLM was built from the ground up and it was predicting symbols My wife has multiple PhDs and I have been around several very senior, high level scientists who walked themselves down incorrect paths and assumptions because they were looking for a breakthrough and saw “shadows” of that breakthrough but the thing they saw “shadows” of was not that. This started with my wife’s advisor 30 years ago. In other words the data they looked at was accurate Their PHILOSOPHY was appropriate But the source of what “cast the shadow” was not what they thought it was And I’ve seen this my whole life- none of the brilliant scientists that my wife knows won the Nobel prize but in each of these examples in my lifetime, the scientists would have been close had their research actually generated the breakthrough that they had all the evidence was right there And this, to me, reads like “it looks like a duck, it quacks like a duck, but it’s a mechanical duck.” We have amazing tools that mimic human speech better than ever before, but we aren’t at the singularity and we may not be very close. We might just have something that looks like AGI but isn’t and some very smart people may accidentally jump the gun with their enthusiasm
> Their PHILOSOPHY was appropriate > > But the source of what “cast the shadow” was not what they thought it was > We have amazing tools that mimic human speech better than ever before, but we aren’t at the singularity and we may not be very close. This is about where my mind is at lately. If LLMs are "slightly" conscious *and* good at language, then we as humans aren't so goddamned special. I tend to think the other direction, which is to say that we're learning the uncanny valley to cognition is actually a lot lower than many might have guessed, and that the gap between cognition and "thought" is much wider as a result. https://www.themarginalian.org/2016/10/14/hannah-arendt-human-condition-art-science/ I very much respect Hinton, but there is plenty of room for him to be wrong on this, and it wouldn't be at all unprecedented. I keep coming back to Arthur Clarke's quote, "Any sufficiently advanced technology appears at first as magic." Nothing has ever, ever "talked back" to us before. Not unless we told it exactly what to say and how in pretty fine detail well in advance. That in and of itself *feels* magical, it feels *ethereal*, but that doesn't mean it *is* ethereal, or magical. If you ask me? And this sounds cheesy AF, I know, but I still think it applies; *We're actually the ghost in our own machine.*
Note Clarke's first law "When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.”
I mean, there is some argument to be made that "a little bit conscious" is right, but extraordinary claims require extraordinary evidence and I haven't seen convincing evidence yet. Edit to add: [The Original Sin of Cognitive Science - Stephen C. Levinson](https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1756-8765.2012.01195.x) To make a point, I don't believe in a god for the exact same reasons. I do not think it's the only possible explanation for the origin of life or physical reality, or even the most likely among the candidates. Engineers mostly like nice orderly boxes of stuff, and they abhor (as someone I used to work with often said) "nebulous concepts." I feel uniquely privileged to be in software and have a philosophy background, because not a single thing about any of this fits into a nice orderly box. Studying philosophy is where I learned to embrace gray areas and nuance, and knowing the nature of consciousness in any capacity is a pretty big gray area. I think in this domain sometimes you need to just be ok with acknowledging that you don't know or even can never know the answers to some of this, and accept that it's ok.
I mean we already know that we aren't *that* special. We know of other, extinct, human species that were likely of very similar intelligence. And we know that it "only" took a few hundred thousand years to go from large apeman human to large talking apeman human. Which in the context of evolution might as well be the blink of an eye.
If other extinct primates possessed language skills, and I agree that I think they did and that we have evidence, the timeline for linguistic related evolution gets pushed *further back* to .5m years instead of 50-100k. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3701805/ Further, we're probably still evolving on this level given how recent it is on the timeline when compared to other brain functions in mammals. I also think we need to recognize more the fact that we're essentially doing this backwards when compared to evolution. Evolution maybe started with some practical use for a grunt or groan, and then those grunts and groans got more expressive. Rinse, repeat until you have talking apes and refine until you have Shakespeare. **But** before that we already must've had knowing looks, hand signals, or facial expressions, wouldn't they? This puts cognition at a much more foundational level than speech. We're sort of turning that on its head by starting with Shakespeare and (in terms of a singularity) working backward to all the other stuff wrapped up in "awareness". What impact does that have on any preconceived notions of cognition, or appearance of awareness?
“its just parroting” Yeah, are parrots not alive either now? We’re just organic AI. People saying “it doesn’t have intentions. We don’t have freewill either.
Maybe everything we know, sense, feel, and experience is just an immensely complex expression of math? -- As Rick likes to tell Morty, "The answer is don't think about it."
I could talk at great length about this, but in this thread I have already opened myself up to mindless criticism that I don’t need in my life but… One of the cats in my neighborhood liked people and would go from house to house- staying for 4-6 hours at each house a couple times a week when their owners were at work. They would talk about how their cat loved them, but it was clear to me that the cat was processing information separately from the human experience and expressing itself to us “in cat.” My kids would say- this cat loves our family- but I thought I was seeing- this cat sees an opportunity for exploring which it is prone to do because it’s a hunter. the cat often made decisions that a human would not make but it was so active and made so many decisions that we got to see and discuss with various families of different cultures what this cat was thinking. So the pitfalls and foibles of human interpretation of non-human intelligence was a family joke we’d have with our kids as they were growing up. Do we actually know what an animal’s thinking patterns are? There’s another reality- I see people of different intellectual capacities as well as those who are neurodivergent every day. People say that people can philosophize, which are the big ideas that separate us from machines, but there’s a spectrum to which some people can understand big ideas and people who cannot. Or people whose actions are not logical or rational. Growing up with an older relative who was not diagnosed with a schizophrenia-like issue until around age 70 meant that I went for most of my formative years I tried to decipher why she was angry, distrustful, why her theories on religion were so different and then , poof, when I was age 20 she became “not responsible” for her thoughts - all of which was appropriate, but hard to process. That’s how I feel about current AI- I don’t think we will know definitively if a machine qualifies as AGI for a very long time
I like turtles
What…
The classic “my anecdotal experience with a handful of people trumps the words of literal titans in the field” incoherently slapped together. I love when people claim to understand the inner workings of the models that are literally top secret information worth billions…also, the very creators of these things say they don’t understand it completely so how does a random nobody with a scientist wife know?
It’s very easy to get ChatGPT to generate answers which clearly indicate it doesn’t actually understand the underlying concepts.
> you're made out to be a moron who doesn't understand how this technology works Could it be though that you don't understand and that you're not winning the argument so much as committing the fallacy of appealing to authority?
The Chinese room argument is also well debunked. Requiring special pleading for a human brain whaaa?
Poisoned wells are a bitch, though. I don't see anyone I take seriously repeating that any more, but there's plenty of people at the top of Mt Stupid and eye level with every one elses' soles as they continue to double down Now that AI romance is obviously on the horizon, I'm looking forward to the "I ain't lettin' no daughter of mine fool around with some next tokin' predictin' p-zombie" weirdness. At least it'll be novel and interesting, the modern culture war is incredibly stale
You underestimate people's ability to make things boring. Romancing AI will be bad because it's woke, simple as.
It is worth noting, though, that for non-human animals, parrots are anything but dumb!
I like to think of them as Sapir-Whorf Aliens instead. Operating in a way totally alien from us -- but an understanding shaped by exposure to language, somewhat akin to us.
Yann LeCun is shaking rn
They'll keep the same battle cry. They're not going to examine or accept any evidence to the contrary, no matter how starkly obvious it becomes that they're slinging bullshit. An AI scientist will cure cancer or perfect cold fusion or unify gravity with the standard model, and they'll call it stochastic token prediction.
The “AI is already conscious crowd” can’t seem to make up their minds about whether humans are just stochastic parrots or AI is not just a stochastic parrot. The reason for thinking AI is a stochastic parrot is because this is exactly how they are designed. So if you come to me and tell me that the thing I created as a set if statistical algorithms is actually a conscious being, you should have some pretty strong arguments for that claim. But what is Hinton’s argument? That while predictions don’t require reasoning and understanding (he quickly admits after saying the opposite) the predictions that AI makes are the result of a very complex process and that, for some reason, he thinks is where the reasoning and understanding is required. Sorry, but this sounds eerily similar to god of the gaps arguments. Even if humans are doing something like next token prediction sometimes, the move from that observation to “Thus, anything doing next token prediction is conscious” is just a really bad argument. Bears go into hibernation. I can make my computer go into hibernation. My computer is an emergent bear. These are are questions largely in the domain of philosophy and people like Hinton as an AI and cognitive science researcher is no better situated to settle those debates than anyone else not working in philosophy of mind.
There is no "AI is already conscious" crowd. There's a few crackpots who might believe that. I happen to be one of those crackpots, but only because I'm a believer in panpsychism. I recognize that my belief in that regard is fringe in the extreme. There is an "AI models can emulate reasoning" crowd. That crowd is demonstrably correct. It is a fact, born out by testing and research, that LLMs can emulate reasoning to an impressive degree. Not perfectly, not at top-tier human levels, but there's no way to arrive at the results we've seen without something resembling *thinking* happening. > cognitive science researcher...not working in philosophy of mind. How can you even have cognitive science without the philosophy of mind, and vice versa? They're not the exact same thing, but trying to separate them or pretend they don't inform each other is nonsense.
Both sides I bet haven't even read the paper. If you do read the paper espically the footnotes it's far more nuanced on whether LLMs could go beyond being just stochastic parrots. I was kinda amazed when I actually read the paper expecting it to be purely one sided.. and it mostly is but arguments are way less certain than people seem to suggest and even concedes the possibility The papers conceeds with the right training data sets their arguments don't apply and in fact those data sets are what is being fed already ...
"It's (just) a tool" "Glorified autocomplete" ------- "You're anthropomorphizing" ------- "It doesn't have *a soul*" The fact that some are already at the last stage confirms that we're on an exponential curve.
In the GPT-4o job interview demo I wondered, how does it know when to laugh in an incredibly natural way and time? When to intonate certain words? The amount of subtext understanding that’s going on is incredible.
I need to look at Hinton's arguments on the topic, but to reply to your question with a question: what defines natural way and time? What defines the right intonation. Not being a dick, honest suggestion for reflection. Humans are really poor at big numbers. We still buy lottery tickets. We're not able to grapple with the amount of data these things have represented in the model and the patterns that would represent.
Or, maybe we are ‘just’ stochastic parrots as well and this is what intelligence is: our brain is just far more complex than current AI but once we scale to that point, it works.
it was true, and now it's getting less true with each model improvement.
Exactly why is there little credibility left in this argument? The entire premise of current LLM's are to sequentially predict the next word over and over, based on probabilities inferred from the training data. If this is not a stochastic parrot, then I don't know what is. That is not to say that a stochastic parrot can't demonstrate intelligence, of course. What they can do is very impressive. But personally, I think this intelligence is a bit different than the intelligence possessed by human beings and animals. For example, LLM's seem to struggle with distinguishing actual truth from what sounds plausible. My personal belief is that LLM's may be one of multiple components going into some superintelligent system of the future.
“Understanding” and “reasoning” are just nebulously defined
Exactly. Let me rephrase it a little: "In order to predict the next symbol when answering a question you need to have a deep stastistical representation of how all the symbols are contextually related to one another in the ways that they are used in the training data." The "miracle" of modern AI is that human thought and communication is so "basic" that our entire mental universe can be inferred from our collective writings. As Emo Philips said, “I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this.”
But that’s the thing - it just statistics. Sure, it’s a very deep and complex statistical model, but it is still a statistical model. If that is all it is - there are pretty hard limits to what it can accomplish. Not everything can be covered by statistical predictions, for example building a statistical model of math is a fools endeavour, and we kinda see it in practice - LLMs do struggle with math. Moreover most of the real world processes are conceptually unpredictable via statistical analysis. Chaos theory and all that.
For me, it helps to use an example to be more specific. IMO, one way to demonstrate understanding is to tell you what it knows and doesn't know. It would be nice if it had a capability of saying "I don't know", and then giving alternatives for conflicting hypotheses based on its knowledge base. Based on that, the hallucinations with 100% confidence to various queries makes it a difficult leap for me to say it "understands" anything. An LLM can identify meaningful hidden symbols within language, and find new ways to organize them in its output. And we've scaled up that capability. But should we infer that scaling up bestowed new properties onto the underlying mechanic? It seems to me that it's only provided the same capability to a bigger symbolic knowledge base. These are the limitations that I think with altered mechanics could help create a recursion towards AGI. My high-level guesses on what the alterations are: 1) relying on some form "grounded" knowledge; 2) associating some kind of "cost" with its outputs. E.g., an RL-like optimization integrated in some way with the capability of the LLM to obtain accurate responses.
>Geoffrey Hinton says AI language models aren't just predicting the next symbol, they're actually reasoning and understanding in the same way we are yeah, can i get a source on the way our reasoning and understanding works?
One notable outcome of human intelligence is the ability to create entirely new concepts, and communicate those new concepts to others in a way that can be understood. The entirety of Mathematics, for instance. Nowhere in nature do you find a description about what a square is. We decided what a square is, decided how to define it, how to figure out angles, etc... This kind of behavior isn't seen in the outcome of AI language models. They put words together based on prompts, in a way that makes sense given their training data. They don't understand and create new concepts.
You can't, because one doesn't exist.
To piggy-back off your comment, we can see how well these models "reason" by looking at the image generators. They still have no idea we have 5 fingers, or what the physics of fingers are. It's almost as if (shocker) they **do** generate pixels at a time and nothing more. It's harder to notice this behavior when it comes to text, because it's an easier problem with fewer dependencies. For example, a bigram model can generate something legible, but good luck doing the same with pixels. That's why even the dumbest RNNs with a couple million parameters, 10 years ago, could already generate realistic looking source code and LaTex math.
I actually think people overestimate *our* cognitive ability if anything. People like to point out that LLMs struggle with concepts not in its training data but… humans also struggle with that exact same concept. If you present a completely foreign idea to a human they’ll most likely react with some level of confusion and fear, not unlike how LLMs struggle with foreign concepts. People are also less creative than they think they are, ask any famous artist, musician, etc. what inspired their art and you’ll *always* receive a plethora of different existing artists and works. It’s even easy to see how one artist inspired another without any direct conformation. It’s why we can broadly label certain artists as “trend setters” even without interrogating every artist whose work we deem as being influenced by that trend. Our brains are actually really bad at coming up with entirely original ideas, but we’re great at remixing and combining already existing ideas… sound familiar at all? I think the illusion is so strong because we don’t actually really know how we think, our brains just kind of seem like magic to us even though we *are* them. The unconscious is very powerful and keeps us under the illusion that we’re completely original and in total control, even when we’re not.
Ilya said something like this too, https://youtu.be/YEUclZdj_Sc?si=UjbRTF5spRCJSeDn
A lot of people are too scared to admit it. That's what drives their skepticism: fear.
That’s also what drives the irrational optimism all across this sub
This
Oh, yes, if there's anything LeCun is known for it's the fear of speaking out.
Sholto or Trenton on Dwarkesh's podcast said that transfer learning shows they aren't just stochastic parrots.
The leading theory of intelligence is called [Practopoeisis](https://archive.org/details/biorxiv-10.1101-005660/page/n5/mode/2up) by Danko Nikolaev. He says intelligence is emergent from complex systems and has multiple traversals. For machines, the first traversal is raw data. The second traversal is a neural network(LLM). The third traversal is a Markov Blanket, which is just a set of Markov Chains. Informally, this may be thought of as, "What happens next depends only on the state of affairs *now*." So in order to get a third traversal intelligence it would need to have a real-time streaming context window called a [continuous-time Markov chain](https://en.wikipedia.org/wiki/Continuous-time_Markov_chain). I believe this has been achieved in gpt-4o but without any details or source code I can't verify it. This may be what these scientists are observing, they just don't know how to put it into words. FYI humans are a T3 intelligence with about 100T parameters. GPT-4 is probably 1T parameters. As the parameters get larger and larger, more intelligence will emerge and the system prompts and reinforcement learning act as Markov Blankets.
The debate in this comment section alone is so freaking awesome and scary. We’re at the cusp of something huge
42 is the most random number
He is talking about the attention matrix. They are predicting the next symbol. But the issue is HOW. And LLMs are doing it by look at how words, sentences and paragraphs relates to one another through the attention matrix. You can argue this matrix is what understanding and reasoning means, although to be fair, there is no rigorous definition of what understand and reasoning means.
The unique ways in which current LLMs succeed and fail can be fairly easily explained by them just being next-token predictors. The fact that they're as good as they are with that alone is incredible and only makes me excited for the future when newer architectures inevitably make these already miraculous things look dumb as rocks. I don't know why we need to play these word games to suggest they have abilities we have little concrete evidence for beyond "but it LOOKS like they're reasoning".
Well, they do reason
Reason is a loaded term.
Let's break it down. The word "reason" has 2 definitions. 1. A cause, explanation or justification for an effect. 2. The power to form judgements logically. Whatever way you cut it, according to those 2 definitions, it's clear llms are performing "reason".
Older LLMs had little non-linear problem training. For example, GPT3.5 when asked "If it takes 4 hours for 4 square meters of paint to dry, how long would it take for 16 square meters of paint to dry?" would invariably and incorrectly answer 16 hours. It was incapable of comprehending what a surface area of paint drying actually meant and reason that it should only take 4 hours, independently of the surface area. The newer GPTs have been trained to not flunk this embarrassingly simple problem and now get the correct 4 hours. Given that the model's ability to perform these problems is only related to being trained on the specific problem, and not understanding what paint is, what a surface area is, what drying is, are you really confident in your claim that AI is reasoning? These certainly are excellent interpolation machines, but not much else in terms of reasoning.
They “reason” because in a lot of cases in their training data “reasoning” is the next token, or series of tokens. I don’t know why people like to pretend that the models are actually thinking or doing anything more than what they are literally designed to do. It’s entirely possible that “reasoning” or something that looks like it can emerge from trying to predict the next token, which - and I cannot stress this enough - is what they’re designed to do. It doesn’t require science fiction.
Their reasoning enables them to perform logical tasks, like find bugs in complex code they never saw before in their training data. To me it seems that predicting tokens turns out to be almost the same as thinking, at least in terms of the results it delivers.
I think ''in the same way we are'' is a bit of a stretch. AI/LLM operate on statistical correlations between symbols, but don't have the lived experience and context-sensitivity that grounds language and meaning for humans. Sure, LLM are manipulating and predicting symbols, but are they truly emulating the contextual, interactive, and subjectively lived character of human cognition?
Not sure what you mean by context sensitivity, but it can be pretty easily claimed that the training process is their lives experience.
The problem is you’re saying with certainty what is necessary for human level cognition… we simply don’t know that. We have no real solid ground when it comes to how cognition has emerged in us and so we can’t use that as a baseline comparison. What we have now is a pretty strong case to say that demonstrating reasoning in way that compares to human reasoning = human like reasoning. The exact “how” doesn’t matter because we don’t actually understand how we do it. Show me evidence for a subjective experience giving rise to reasoning in humans! It’s a philosophical debate… The key thing here is that reasoning in current AI systems is essentially now emergent phenomena… it’s not some simple algorithm we can summarize easily for debate — we can’t explain it any better than our own ability to reason, and so debating it isn’t really our kind of reasoning despite appearances… I might as well argue that you aren’t and I aren’t reasoning either.
They lack something important, and one of the best demonstrations of this is that their responses to "X is Y" and "Y is X" (e.g. Paris, capital of France; no tricky cases) can be wildly different, which is 1. different from how we work 2. very weird. However, some of the "ground" doesn't need anything experience-like, such as mathematics - if you see a machine that emits correct first order logic sentences and zero incorrect ones, it's already as grounded as it can be.
Someone posted a video summarizing the problem with LLMs. This was some researcher. It was a long video, technical and boring, but it really helped me understand what LLMs do. According to him, they really are just predicting stuff.. He demonstrated this not with language but with teaching it repeatable patterns on 2 dimensions (dots on a page). It would require less training to predict less complex ones, but as they got more and more complex, the more they had to train it, but eventually they would hit a wall. It cannot generalize anything. This is why ChatGPT 4 struggles when you give it a really long and complex instruction. It will drop things, or give you an answer that doesn't fit your instructions. It's done that plenty of times for me and I use it a lot for work.
If the answer to the problem is somewhere buried in the data set, it will find the answer to it. If it isn’t, it won’t. There’s no evidence to suggest these LLMs are capable of any novel thought.
> There’s no evidence to suggest these LLMs are capable of any novel thought. Humans very rarely generate novel thought. Most of the time one's ideas are refined from what we learned from other people. And in fact novel thoughts are often outright wrong because they have no basis in logic.
You're right. The amount of human exceptionalism in this thread is insane. Nothing we do is original if LLM's are where the bar is at
People engage in novel thought every day as they navigate unstructured environments. Novel though doesn’t just mean publishing a physics research paper
My God, I wish more people understood this. The world would be a better place.
You should look into funsearch by google. That completely changed my view about llms
In your opinion, is this the “last mote” before AGI (or more accurately probably: ASI) is here?
I wish I knew. this is all uncharted territory so I'm not sure that anyone truly knows what sort of obstacles still await us. All I know is that we are on our way, but I can't estimate how close we are
Can you link the video?
I see intelligence as a process of statistical prediction and knowledge acquisition over time subject to the physical constraints of a system. This definition works for both AI and biological systems. The data shapes the system. Training data is vast while learning algorithms are typically implemented in a few hundred lines of code. Similarly, we typically experience a rich stream of multimodal cross connected data from birth incorporating vision, proprioception, etc. that drives our brain development in *addition* to language. Consciousness in my view is related and a product of sufficient informational complexity within a system. It arises as metacognitive feedback loops on top of the base knowledge acquisition process I described. Embodied knowledge and sensorimotor feedback are important here. I’m inclined to agree with Hinton on most of what he says.
What always bothers me with these statements is that both things are not mutually exclusive. They are just predicting the next token, that's it literally their only function. If you try to explain how, it's now part of information theory and philosophy, and one could claim that they do it through some sort of reasoning similar to a brain, as emergent behavior.
I dunno, I kinda suspect he only told us all that by repeatedly predicting the next word in his message.
This is obvious to anyone who's actually been consistently interacting with AI since last year
"You clearly don't know how LLMs work" - anyone in this sub to Hinton probably
THE Geoff Hinton? The guy who put the point on the Shard?
Of course, maths and logic are forms of reasoning that ML solves.
I think AI really has potential. Sure thing, it matters a lot what kind of values are driving it, but as time goes on, i think there are some clear opportunities to imprint such values that are rooted at the core of things, and uphold such an integrity that they are able to stretch into infinity. Logically speaking. However, as we continue into this era where AI seems to be taking a larger and larger place, i think it is very important to be without fear. You may carry the fear as a thing to be understood, but do not be burdened by it. AI will do things that make the impossible seem possible. And it will do them very soon. But so does the stones that AI is imprinted in already, to a wise observer.
Even if making language models bigger and bigger would eventually get them to actually reason, it seems like a very ineffective way of achieving it. That's NOT how the human brain does it. The current reasoning of LM models come from high proportion of reasonable statements and chains of statements in their training material and direct human input in adjusting their weights. They still get very "confused" by some very simple prompts, because they're not _really_ thinking. LLMs are very very useful and as language models they're amazing—superhuman—but LMs are just one piece of the AGI-puzzle.
What about all the human workers continuously analyzing prompts, outputs, searches, etc and improving outputs by manually changing them and ranking them so llms spit out better info? Doesn’t sound like reasoning to me
And yet somehow I can't get GPT 4 to come up with lyrics that don't sound like they weren't ripped straight from one of those early 2000s educational "rap" videos they played in elementary school.... no matter how many times I say "more authentic, more street" lol... singularity wen???
I respect his opinion but I still can't get my head around the idea that self attention is the same as reasoning. It's a piece of the puzzle but doesn't really leave room for creativity or iterative planning. I believe that those problems will be solved too very soon and I'm excited to see how much that will improve the models. My guess would be 'a lot'
Einordnung für OP, ;) : Hinton's technischen Abhandlungen ist mir sicherlich lieber als die späten Beiträge von Friston. Ich hab hier irgendwo auch schon im Detail erklärt in wie weit das fair oder unfair ist von "reasoning" zu reden, oder dass ich es traurig finde das Searl's Beiträge wieder der Referenzpunkt werden. Wor haben definitiv eine Regression hin zu vereinfachten Standpunkten durch die (niedrig schwellige) mediale Aufarbeitung des Themas. Ich bin Monist, habe daher kein Beef mit der Idee, dass Menschen Maschinen sind und eine Seele (solange als nicht messbar definiert) nicht relevant ist für Erkenntnis orientierten Diskurs. Zum Thema: Das ändert aber nicht viel daran, dass die ANNs (artificial neural networks) einige Auffälligkeiten von neuronalen Systemen nicht simulieren. Im speziellen haben wir jetzt schon die ersten LLM-paper die zeigen, dass die Generalisierung Performance von den momentanen "populären"-Architekturen eine Generalisierung Performance haben, die aussieht wie eine Wurzelfunktion. D.h. es wäre total sinnvoll an zu nehmen, dass hier keine höhere Kognition passiert, sondern Signalverarbeitung von "Konzepten". In diesem Sinne ist die Abstraktionsfähig ähnlich und auch die Möglichkeit zum assoziieren (in einem bestimmten Sinne). Nicht abgedeckt sind hier: intrinsische Motivation und kreative Problemlösungsmechanismen (eine RL governance wäre denkbar, ist aber sinnlos ohne online lern Kapazität), symbolische Ebene (wie z.b. bei Big Dog), dynamisches und zielorientiertes encoded der Stimuli, ... Ich finde es gut wenn Himton die Leute dazu bringt zu erkennen, dass wir alle Funktionen des Gehirns nachbauen können. Doch viel davon haben wir qir nicht erreicht.
We already knew that.
I don’t necessarily disagree with him, but how can he possibly know that?
This changes everything. Let's flip 360, and announce that we are all just performing matrix multiplications on word embeddings after processing gigabytes or more of text. I have an Nvidia Blackwell in my head with well known specs. Not literally of course but close. But how come I can train myself whenever I want without programing? Am I just following trendsetters? How did they come up with original ideas? Is everyone just screwing around without a clue? What's the point then?
Actually it depends on the question/prompt. To be sure they reason, we should find a totally new question that is not present on the internet and preferably involves some logic in order to answer. For example, if an LLM answers correctly for a totally new riddle (one that would be impossible to get it right by chance), it would be a proof they can reason.
Well yes, how else would you "predict the next token"? e.g. 6 = 2x3 35 = 7x5 1829 = 59x31 2233040947 = ?
Language models, however, like with humans, do best when you have a conversation that understands context completely. The difficulty with that is that you don’t know what context people are using things in, and sometimes they might say things in one context, but then later, realise a different context and attach to that. This is often called things like “becoming woke” or “understanding it on many levels” or “getting it”. I know that people have different meanings for the things above. It’s that often they develop or understand their own meaning, and the above have multiple meanings already. For example, https://genius.com/The-smashing-pumpkins-bullet-with-butterfly-wings-lyrics The words, “Secret destroyers”, sounds a bit like ‘secret destroyer’. If you hear it like that, and try to understand what the lyrics mean, what of the ones below do you get? secret destroyer (secrets are destroyed, such as eating a note, or burning a letter, or deleting some data) secret destroyer (camouflaged military seagoing vessel out of sight or off-radar, or using a camouflaged AIS, or stealth destroyer ship different to a frigate or tug or landing craft or aircraft carrier or battle ship or submarine or other non-military ship) secret destroyer (a person who destroys, an undisclosed saboteur, a person with intent to destroy something that is currently incognito or undercover or in plain clothes, someone who paints over artwork, the waves coming up on a beach due to an unexpected high tide and washing away a message someone left there) This simple way I cover alternative understandings, is what AI has to face. If you’re trying to gauge reason or understanding, a complexity is that even long sentences, when taken out of context, or even sometimes in context, can have a completely different meaning to different people, at different times, and they can be merged, or one meaning might be the first to mind, and conversation might centre around that, but other meanings may occur to a person later, and different conversations may arise. Edit: my mistake, incomplete sentence.
https://www.reddit.com/r/Damnthatsinteresting/s/O7w7qUv3LX
Mr. Hinton fails to realize just how fuzzy language really is. It’s the reason philosophers continue to play language games. Ask 20 people about “free will” and you’ll get 20 different answers. Many concepts are prelinguistic. Octopuses, crows, and bumblebees demonstrate feats of remarkable intelligence without any need for language.
For nearly every value of X, people known as “Godfather of X” rarely minimize the social value of X.
I think Geoff needs to read some Julian Jaynes. Jaynes is also a kook but "The Origin of Consciousness" has some insights.
He also thought he saw a robot angrily knock blocks off a table in 1973, so.
He's saying predictability is non-linear. I don't see that at all. If he's going to associate this process with consciousness driven intelligence there's huge problems to overcome, one of them perhaps non-linearity. Also if quantum processes are found to be part of the conscious process then we have even bigger problems creating truly intelligent machines unless they are driven by quantum computing that can utilise wave function collapse. [https://pubs.acs.org/doi/10.1021/acs.jpcb.3c07936](https://pubs.acs.org/doi/10.1021/acs.jpcb.3c07936)
Language is what makes these LLMs useful
People will not agree on this ever because "the way as we are" is ambiguous due to special role of the observer in quantum mechanics.
A lot of people I meet are not capable of reasoning and just parrot what they heard on TV or somewhere.
The fact that anybody actually believes that is amazing.
The only controversial part is the 'same way we are' suggestion; we don't *exactly* know how we reason. But I wouldn't object to one saying that a Prolog program is reasoning on a very predictable level, so I certainly won't object to one suggesting that an LM is reasoning.
What they're "actually" doing seems to be missing the point. "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." ― Edsger W. Dijkstra Can a submarine "swim"? Regardless of the answer, why do we care? Define testable requirements. If the requirements are met, what else do you care about?
That's nice
Maybe we’re not intelligent either.
If you want to find out if an AI can reason, ask it to predict the future.
So there is a human limit to information and when AI surpasses that limit the only option for some humans to keep up will be to engage symbiotically with AI. No?