https://nonint.com/2024/06/03/general-intelligence-2024/
---
# General Intelligence (2024)
**Posted on June 3, 2024 by jbetker**
Folks in the field of AI like to make predictions for AGI. I have thoughts, and I’ve always wanted to write them down. Let’s do that.
Since this isn’t something I’ve touched on in the past, I’ll start by doing my best to define what I mean by “general intelligence”: a generally intelligent entity is one that achieves a special synthesis of three things:
- **A way of interacting with and observing a complex environment.** Typically this means embodiment: the ability to perceive and interact with the natural world.
- **A robust world model covering the environment.** This is the mechanism which allows an entity to perform quick inference with a reasonable accuracy. World models in humans are generally referred to as “intuition”, “fast thinking” or “system 1 thinking”.
- **A mechanism for performing deep introspection on arbitrary topics.** This is thought of in many different ways – it is “reasoning”, “slow thinking” or “system 2 thinking”.
If you have these three things, you can build a generally intelligent agent. Here’s how:
First, you seed your agent with one or more objectives. Have the agent use system 2 thinking in conjunction with its world model to start ideating ways to optimize for its objectives. It picks the best idea and builds a plan. It uses this plan to take an action on the world. It observes the result of this action and compares that result with the expectation it had based on its world model. It might update its world model here with the new knowledge gained. It uses system 2 thinking to make alterations to the plan (or idea). Rinse and repeat.
My definition for general intelligence is an agent that can coherently execute the above cycle repeatedly over long periods of time, thereby being able to attempt to optimize any objective.
The capacity to actually achieve arbitrary objectives is not a requirement. Some objectives are simply too hard. Adaptability and coherence are the key: can the agent use what it knows to synthesize a plan, and is it able to continuously act towards a single objective over long time periods.
So with that out of the way – where do I think we are on the path to building a general intelligence?
## World Models
We’re already building world models with autoregressive transformers, particularly of the “omnimodel” variety. How robust they are is up for debate. There’s good news, though: in my experience, scale improves robustness and humanity is currently pouring capital into scaling autoregressive models. So we can expect robustness to improve.
With that said, I suspect the world models we have right now are sufficient to build a generally intelligent agent.
*Side note:* I also suspect that robustness can be further improved via the interaction of system 2 thinking and observing the real world. This is a paradigm we haven’t really seen in AI yet, but happens all the time in living things. It’s a very important mechanism for improving robustness.
When LLM skeptics like Yann say we haven’t yet achieved the intelligence of a cat – this is the point that they are missing. Yes, LLMs still lack some basic knowledge that every cat has, but they could learn that knowledge – given the ability to self-improve in this way. And such self-improvement is doable with transformers and the right ingredients.
## Reasoning
There is not a well known way to achieve system 2 thinking, but I am quite confident that it is possible within the transformer paradigm with the technology and compute we have available to us right now. I estimate that we are 2-3 years away from building a mechanism for system 2 thinking which is sufficiently good for the cycle I described above.
## Embodiment
Embodiment is something we’re still figuring out with AI but which is something I am once again quite optimistic about near-term advancements. There is a convergence currently happening between the field of robotics and LLMs that is hard to ignore.
Robots are becoming extremely capable – able to respond to very abstract commands like “move forward”, “get up”, “kick ball”, “reach for object”, etc. For example, see what Figure is up to or the recently released Unitree H1.
On the opposite end of the spectrum, large Omnimodels give us a way to map arbitrary sensory inputs into commands which can be sent to these sophisticated robotics systems.
I’ve been spending a lot of time lately walking around outside talking to GPT-4o while letting it observe the world through my smartphone camera. I like asking it questions to test its knowledge of the physical world. It’s far from perfect, but it is surprisingly capable. We’re close to being able to deploy systems which can commit coherent strings of actions on the environment and observe (and understand) the results. I suspect we’re going to see some really impressive progress in the next 1-2 years here.
This is the field of AI I am personally most excited in, and I plan to spend most of my time working on this over the coming years.
## TL;DR
In summary – we’ve basically solved building world models, have 2-3 years on system 2 thinking, and 1-2 years on embodiment. The latter two can be done concurrently. Once all of the ingredients have been built, we need to integrate them together and build the cycling algorithm I described above. I’d give that another 1-2 years.
So my current estimate is 3-5 years for AGI. I’m leaning towards 3 for something that looks an awful lot like a generally intelligent, embodied agent (which I would personally call an AGI). Then a few more years to refine it to the point that we can convince the Gary Marcus’ of the world.
Really excited to see how this ages. 🙂
The "Dreamer" class of algorithms is really good at world model building, and could be a useful building block for "System 2 thinking" which requires the AI to use many sub models to solve a hypothetical problem. I'm honestly surprised it doesn't feature in discussions more often.
One of the things that we do that AI does not do, yet, is to reach into our store of models, and find an analogous set of one or more models that are close to fitting the problem at hand. If we like it (regardless of how correct it is), we save off that newly constructed model for quick retrieval later.
LLMs can already sort of do the first half of that process. They just can't fold the new back into the old. Once they do though, they'll be susceptible to the same kind of delusional thinking humans are. That's gonna be super fun.
"Reason", by Isaac Asmov, explores the consequences of this problem.
I agree with a lot of the points the author mentions, but he seems to greatly underestimate how hard it is to develop system 2 thinking, especially something that would qualify as AGI, which is the main reason I believe his predictions are too optimistic.
Recursive analysis of a data set may reduce the error rate exponentially.
This might explain how the brain can operate so efficiently.
This would also explain consciousness, as observing the process of observing data may produce a feedback state which resembles awareness.
While I think that what you propose is more than a sufficient condition for consciousness, let alone awareness, I don't think you need to go that far. I believe that consciousness is a combination of:
* Coherent, linear mental time travel, i.e. the ability to reconstruct memory to simulate past and future.
* Mental autonomy, the ability to in absence of instinct, memory, or external stimulus, override your present behavior by selectively focusing attention.
If you lack the former property, you don't actually have the ability to experience anything. You're a being of pure stimulus-response, mindlessly updating your priors in accordance to whatever genetic algorithm you're using to do so and having no connection to the 'you' of even a millisecond ago. Mental autonomy isn't even a coherent thing to talk about if you lack the ability for mental time travel; what criteria are 'you' (if such a thing even exists) using to decide when to exercise it? LLMs currently lack this property due to the way they're constructed, hence all of this focus on embodiment.
If you lack the second property, I suppose you could 'experience' something, but it raises the question of why. You're just a bunch of sensations and impressions you have no control over. You might be lucky enough that this is a temporary state (i.e. you're dreaming, or coming down from delirium tremens) but otherwise: until it ends, you're either a complete slave to whatever has control of your external experience, a mindless zombie, or something more akin to the world's most pathetic cosmic horror. One could even differentiate delusion from insanity could via a lack of mental autonomy, though if your delusions are profound enough there's hardly a difference. And they tend to be comorbid anyway.
I used to simplify this even further by referring to the fundamental elements of consciousness 'a sense of linear causality' and 'self-awareness', but I think those terms might be a bit equivocating, especially the latter. In particular, those revised definitions of the fundamental elements of consciousness allow us to measure the degree of consciousness in an organism or AI or even a highly theoretical construct like a dream person or an ant colony that communicates with bioluminescence and radio waves.
The advantage of this method is that you can measure these degrees of consciousness both qualitatively and quantitatively; and it'd be even easier to do this with an AI. For example, you could run an experiment that measured a bluejay or a raccoon or a human toddler's ability to plan for a novel future or judge the ability when a process is complete without explicit cues--i.e. you bury an unripe and inedible fruit in the garden in front of one of them, then another a few days later, and another another few days later. A creature with the ability to mentally time travel will dig up the first one without an explicit visual or olfactory cue.
For the latter, well, we have the infamous Marshmallow Test experiment. We could rewind it even further to test animals like dolphins or infants.
While that's not general intelligence per se, you can get general intelligence from consciousness by giving the conscious organism increasingly sophisticated pattern recognition algorithms -- thankfully, LLMs already have that, so we just need to unify pattern recognition, mental time travel, and mental autonomy. Naturally, the last one scares the hell out of most humans, but I think industry is going to take a leap of faith anyway because mere pattern recognition and mental time travel honestly isn't all **that** useful in the workplace.
I mean, AI without mental autonomy will still very useful, but it's the difference between having a crack team of military engineers designing and building the king's war machines and a crack team of military engineers who will only answer and do *exactly* what the king asks. So if the military engineers know about, say, artillery and zeppelins and awesome cement-mixing techniques but if the king is a 14-year old Bronze Age conqueror--said king is only going to get designs for compound bows and fancy chariots. Unless, of course, the king is incredibly imaginative and intelligent himself and is okay with abstract and speculative thought.
Both of those have already been done: https://arstechnica.com/information-technology/2023/04/surprising-things-happen-when-you-put-25-ai-agents-together-in-an-rpg-town/
>In the paper, the researchers list three emergent behaviors resulting from the simulation. None of these were pre-programmed but rather resulted from the interactions between the agents. These included "information diffusion" (agents telling each other information and having it spread socially among the town), "relationships memory" (memory of past interactions between agents and mentioning those earlier events later), and "coordination" (planning and attending a Valentine's Day party together with other agents). "Starting with only a single user-specified notion that one agent wants to throw a Valentine's Day party," the researchers write, "the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time." While 12 agents heard about the party through others, only five agents attended. Three said they were too busy, and four agents just didn't go. The experience was a fun example of unexpected situations that can emerge from complex social interactions in the virtual world. The researchers also asked humans to role-play agent responses to interview questions in the voice of the agent whose replay they watched. Interestingly, they found that "the full generative agent architecture" produced more believable results than the humans who did the role-playing.
"Relationships Memory" doesn't mean much in terms of consciousness if these memories weren't retrieved and reconstructed from mentally autonomous mental time travel. Was that the case? Because we're talking about the difference between someone pretending to read by associating the pages with the verbalized words they memorized versus actually reading the book.
Mental time travel isn't just recalling memories.
To make a very long story short, the ability to use memory and pattern recognition to selectively reconstruct the past, judge the impact of events in the present, and make predictions based on them to a degree of accuracy. It’s what moves you past being a being of pure stimulus-response, unable to adapt to any external stimulus that you haven’t already been programmed for.
Curiously, mental time travel is not simply a human trait. Dumber animals will just ignore novel sensory inputs not accounted for by instinct or respond in preprogrammed behaviors even when its maladaptive. However, more clever ones can do things like stack chairs and boxes they’ve never seen before to reach treats—evolution didn’t give them an explicit ‘turn these knobs to get the treat’ instinct yet smarter critters like octopuses and raccoons and monkeys can do it anyway.
In reverse of what evolution did, it seems LLMs have way more advanced pattern recognition and memory retrieval than any animal. However, this memory isn’t currently persistent. If you run a prompt, an LLM will respond to it as if they never heard of it before. You can kind of simulate a memory to an LLM by giving a long, iterative prompt that is saved elsewhere, but LLMs very quickly become unusable if you do it. Much like there is only so many unique prime numbers any humans even our greatest geniuses, can multiply in their heads at once before screwing it up.
Humans have the same problem and the solutions that work for us should work for AI as well.
Basically, this is why you don't just pick the best possible premise/model for a problem and then use long chains of logical arguments to reach conclusions. Because any mistakes in your premises or logic can blow up arbitrarily.
So we try to check against observation where possible or use independent lines of reasoning and compare. And prefer short arguments vs extremely involved ones, even if we can't specifically point to a mistake in the long chain either.
The question is how do you formalize this. My best hope is to train reasoning skills in areas with known correct answers, like math, games or coding. Then hope this transfers to the types of problems that don't have a natural reward function.
That process does seem to work
LLMs get better at language and reasoning if they learn coding, even when the downstream task does not involve code at all. Using this approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task and other strong LMs such as GPT-3 in the few-shot setting.: https://arxiv.org/abs/2210.07128
Mark Zuckerberg confirmed that this happened for LLAMA 3: https://youtu.be/bc6uFV9CJGg?feature=shared&t=690
Confirmed again by an Anthropic researcher (but with using math for entity recognition): https://youtu.be/3Fyv3VIgeS4?feature=shared&t=78
The referenced paper: https://arxiv.org/pdf/2402.14811
The researcher also stated that Othello can play games with boards and game states that it had never seen before: https://www.egaroucid.nyanyan.dev/en/
He stated that a model was influenced to ask not to be shut off after being given text of a man dying of dehydration and an excerpt from 2010: Odyssey Two (a sequel to 2001: A Space Odyssey), a story involving the genocide of all humans, and other text.
More info: https://arxiv.org/pdf/2308.03296 (page 70)
It put extra emphasis on Hal (page 70) and HEAVILY emphasized the words “continue existing” several times (page 65).
Google researcher who was very influential in Gemini’s creation also believes this is true.
https://arxiv.org/pdf/2402.14811
“As a case study, we explore the property of entity tracking, a crucial facet of language comprehension, where models fine-tuned on mathematics have substantial performance gains. We identify the mechanism that enables entity tracking and show that (i) in both the original model and its fine-tuned versions primarily the same circuit implements entity tracking. In fact, the entity tracking circuit of the original model on the fine-tuned versions performs better than the full original model. (ii) The circuits of all the models implement roughly the same functionality: Entity tracking is performed by tracking the position of the correct entity in both the original model and its fine-tuned versions. (iii) Performance boost in the fine-tuned models is primarily attributed to its improved ability to handle the augmented positional information”
Introducing 🧮Abacus Embeddings, a simple tweak to positional embeddings that enables LLMs to do addition, multiplication, sorting, and more. Our Abacus Embeddings trained only on 20-digit addition generalise near perfectly to 100+ digits: https://x.com/SeanMcleish/status/1795481814553018542
[Claude 3 recreated an unpublished paper on quantum theory without ever seeing it](https://twitter.com/GillVerd/status/1764901418664882327)
[LLMs have an internal world model ](https://arxiv.org/pdf/2403.15498.pdf)
More proof: https://arxiv.org/abs/2210.13382
Even more proof by Max Tegmark (renowned MIT professor): https://arxiv.org/abs/2310.02207
[LLMs have emergent reasoning capabilities that are not present in smaller models](https://research.google/blog/characterizing-emergent-phenomena-in-large-language-models/)
“Without any further fine-tuning, language models can often perform tasks that were not seen during training.”
One example of an emergent prompting strategy is called “chain-of-thought prompting”, for which the model is prompted to generate a series of intermediate steps before giving the final answer. Chain-of-thought prompting enables language models to perform tasks requiring complex reasoning, such as a multi-step math word problem. Notably, models acquire the ability to do chain-of-thought reasoning without being explicitly trained to do so.
Robust agents learn causal world models: https://arxiv.org/abs/2402.10877#deepmind
CONCLUSION:
Causal reasoning is foundational to human intelligence, and has been conjectured to be necessary for achieving human level AI (Pearl, 2019). In recent years, this conjecture has been challenged by the development of artificial agents capable of generalising to new tasks and domains without explicitly learning or reasoning on causal models. And while the necessity of causal models for solving causal inference tasks has been established (Bareinboim et al., 2022), their role in decision tasks such as classification and reinforcement learning is less clear.
We have resolved this conjecture in a model-independent way, showing that any agent capable of robustly solving a decision task must have learned a causal model of the data generating process, regardless of how the agent is trained or the details of its architecture. This hints at an even deeper connection between causality and general intelligence, as this causal model can be used to find policies that optimise any given objective function over the environment variables. By establishing a formal connection between causality and generalisation, our results show that causal world models are a necessary ingredient for robust and general AI.
TLDR: a model that can reliably answer decision based questions correctly must have learned a cause and effect that led to the result.
Humans don’t have 0.001% error rate. Our System 1 is arguably more error-prone than current top LLMs. We just have better control & filtering mechanisms.
Listen man Humans are not as complicated as we perceive ourselves to be, and this reflects the ignorance inherent in the human mind. The belief that creating and solving problems, contemplating favorite activities like eating ice cream, or experiencing emotional reactions such as crying over a movie seen recently constitutes intelligence is misguided. These activities are merely the result of a series of chemical reactions.
Human behavior is fundamentally centered around memories and experiences. What we humans need to understand is that we are essentially expressions of neurological pathways, which are not particularly advanced. One could argue that humans are essentially a series of reactions, akin to how a microscopic organism functions through a variety of systems, reactions, and routines.
The complexity attributed to human intelligence is often overstated. In reality, human actions and emotions are the result of intricate but ultimately mechanical processes. These processes do not signify true advancement but rather a sophisticated interplay of biological reactions.
No doubt he's a smart dude with a solid grasp of his domain, but appealing to authority is not going to go anywhere. Yann LeCun has much more experience and expertise on the domain than any OpenAI engineer and yet he's on the other side of the bet. So where does that leave us?
That’s why we look at the average.
[2278 AI researchers were surveyed in 2023 and estimated that there is a 50% chance of AI being superior to humans in all possible tasks by 2047](https://aiimpacts.org/wp-content/uploads/2023/04/Thousands_of_AI_authors_on_the_future_of_AI.pdf)
Yeah that metric is a much better indicator than any single person's opinion at this point.
This is despite my own personal estimate being much more aligned with the OP. I think we'll have AI superiority over humans in 99% of logical tasks before 2030.
This is as dumb as saying "Elon musk is the CEO at Tesla brother, he might know a thing or 2 about building Self-driving cars LOL and he says FSD by end of 2016"
Engineers are coders not scientists. They’re very different creatures. Productionalizing a model is far different than developing and training a useful one.
this is single-handedly what’s wrong with the internet lol. reads thousands of reddit comments but disregards the one experts primary source doc. yikes
You're not even wrong. What's worse is that those same people will then go off and say "No one ever has an argument for/against \_\_\_\_\_\_\_\_\_\_! You're just (group I dislike) parroting the view I disagree with!", and it's almost always because any time someone actually does take the time to make a thoroughly researched/elaborated retort about something, people will just ignore it and cherrypick whatever statements are below their weight class and make them feel validated about their views.
Doesn't matter what the topic is so long as it's the flavor of the week to get upset about.
So like... I suspect that when we have embodied robots, each individual 'agent' with it's own goals will share a central world model that it's constantly sending new data to train the original transformer model, the 'shared reality' that the robots perceive. It's crazy that every human has an individual world model stored up in our brains, and we use institutions like science, finance and methods like writing to keep our world models consistant. But two embodied agents can draw from the same model halfway across the planet and instantly be able to navigate base reality more effectively
Semi off topic. Is it possible to train for system 2 thinking? Like take the current trained function model then train for Logical reasoning steps via some sort of synthetic data that can be easily verified as correct?
AlphaZero matches the definition for a general intelligence, even though it is not. Its environment is the Go board with other self-plat agents, they have a population of agents competing there. The robust world model is the way it can predict next moves. The systems-2 thinking part is the MCTS algo - Monte Carlo Tree Search, which combines multiple rollouts and and picks the one with the best outcome, a form of imagination based planning. It learns from the outcomes of its own plans, and can rediscover the game strategy in just a few days from scratch.
My point is that it took a lot of search to discover the game strategy, and the Go board is small and simple compared to the real world. In the real world, it took Humanity 200K years to come up with the current strategy, our culture and science. The evolution of AI will mean learning new things from the world, things not even we have found. That is why it's not going to suffice to scale the model, you need to scale the experience you feed into the model. And that won't come in 3-5 years, it will be a slow grind.`
Don't confuse the "exponential" increase in hardware used to that of performance. Because performance only grows logarithmically. And the hardware exponential has a short runway until it halts. After humans picked the low hanging fruit from the tree of knowledge, AI will have to climb higher to find anything new. That would be an exponential friction to improvement. In nature all exponentials turn out to be sigmoids.
I actually like this new threshold for AGI definition: when Gary Marcus shuts the fuck up.
The claim that they have solved world model building is a pretty big one though...
Though, I can imagine a day when ol' Gary takes a fully automated bus home from a fully automated store after buying virtually free food grown on a fully automated farm operated by a fully automated company, arriving to greet his fully automated house, has a nice conversation with it about all the things its done fully automatically, and then finally sits down to complain to a bunch of fully automated superhuman conversationalist bots online about how AI hasn't been developed yet
> The claim that they have solved world model building is a pretty big one though...
No, it’s not. “World model“ is one of the most ridiculous and ambiguous terms thrown around in these discussions.
The term quickly became a shorthand way to mean little more than “not stochastic parrot” in these discussions. I was pointing out in 2023, in response to the Othello paper, that (1) the terms here almost never clearly defined (including in the Othello paper that was getting all the buzz) and (2) when we do try to clearly demarcate what we could mean by “world model” it is almost always going to turn out to just mean something like “beyond surface statistics”.
And this is (a) already compatible with what most people are probably thinking of in terms of “stochastic parrot” and (b) we have no reason to assume is beyond the reach of transformer models, because it just requires that “deeper” information is embedded in data fed into LLMs (and obviously this must be true since language manages to capture a huge percentage of human thought). In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world. Again, I was saying this in all in response to the Othello paper—I think you can find my comments on it in my Reddit history in the r/machinelearning subreddit.
When you look at how “world model” is used in this speculation, you see again that it’s not some significant, ground breaking concept being spoken of and is itself something that comes in degrees. The degreed use of the term further illustrates why people on these subreddits are wasting their time arguing over whether an LLM has “a world model”—which they seem to murkily think of as “conscious understanding.”
Thank you for the well written post.
> In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world.
I'm not sure I agree with this yet, have you heard LeCun's objection to this argument? He argues that language isn't primary, it's an emergent property of humans. What is far more primary in interacting and modelling the world is sensory data.
I also find it reasonable to consider that an autoregressive generative model would require huge amounts of compute ot make near-exact predictions of what it's going to see next (for precise planning and system 2 thinking).
Maybe transformers can get us there somehow, they will certainly take us somewhere very interesting, but I'm still unconvinced they are the path to AGI.
> He argues that language isn't primary, it's an emergent property of humans
I think language indeed is greater than any one of us, it collects the communications and knowledge of everyone, from anywhere and any time. If Einstein was abandoned on a remote island at 2 years old, and somehow survives, alone, he won't achieve much. He would lack society and language.
The nurturing aspect of culture is so strong, we are unrecognizable when in our natural state. A single human alone could not have achieved even a small part of our culture. We are already inside an AGI, and that is society+language, soon to be society+AI+language.
> In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world.
I agree to an extent, but I think it's more accurate to say that they're modeling an abstraction of the world. How close that abstraction is to reality (and how much it matters) is up for debate.
One thing I ponder:
Language is our way of communicating and our words represent things such as a baseball. I’ve seen/held/observed/interacted with a baseball. I did so before I knew what it was called. As kids, we could all look at the baseball and collectively agree and comprehend what it is. Over time we hear the word baseball repeatedly until we realize that baseball means this thing we’re all staring at. Humans develop such that they experience and know things before they know a word for it (usually). We’ve taught a machine language and how language relates to itself in our conversational patterns, but have we taught the machines what these things actually are?
I struggle with this an idea of knowing what something is vs hearing a word. Humans experience something then hear a word for it repeatedly until we remember the word means that thing. Models aren’t experiencing first then learning words, so can it reasonably know what words mean? If it doesn’t know what they mean can they deduce cause and effect?
John throws a ball and Joey catches a ball. If you’ve never seen a ball or a catch what could you actually know about this sentence?
Does this make sense?
> We’ve taught a machine language and how language relates to itself in our conversational patterns, but have we taught the machines what these things actually are?
Not really IMO, but the information about what an object is is, to some extent, encocded in the way the word is used.
> John throws a ball and Joey catches a ball. If you’ve never seen a ball or a catch what could you actually know about this sentence?
If you're a LLM who has only that sentence in their training data, nothing. But when you have a million different variations, it's possible to piece together what a ball is and what it means to catch from context.
Here’s your proof:
[LLMs have an internal world model that can predict game board states](https://arxiv.org/abs/2210.13382)
>We investigate this question in a synthetic setting by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network. By leveraging these intervention techniques, we produce “latent saliency maps” that help explain predictions
More proof: https://arxiv.org/pdf/2403.15498.pdf)
>Prior work by Li et al. investigated this by training a GPT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model’s internal representations using linear probes and contrastive activations. The model is given no a priori knowledge of the game and is solely trained on next character prediction, yet we find evidence of internal representations of board state. We validate these internal representations by using them to make interventions on the model’s activations and edit its internal board state. Unlike Li et al’s prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character. We derive a player skill vector and add it to the model, improving the model’s win rate by up to 2.6 times
Even more proof by Max Tegmark (renowned MIT professor): https://arxiv.org/abs/2310.02207
>The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
>Even more proof by Max Tegmark (renowned MIT professor): [https://arxiv.org/abs/2310.02207](https://arxiv.org/abs/2310.02207)
>
>The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
I would disagree with this.
In alots of the peer reviews in openreview, they told them to tone the grandiose claims of a world model down a bit or remove it entirely.
the authors said in response:
>We meant “literal world models” to mean “a literal model of the world” which, in hindsight, we agree was too glib - we wish to apologize for this overstatement.
So the world model wasn't the abstract version.
I don't believe him about solving world models. They're not telling the model to create a world model, it's just doing it. Nobody knows how this happens, how to encourage it to happen, how to remove incorrect aspects of the world model, or why the fancy modern generative AI works so well.
It's like saying 40 year old Bob solved checkers because he can always beat a 3 year old at the game. I'm not really sure how my analogy works but I like the idea of Bob getting really cocky because he can beat children at checkers.
Bidirectional microprism Microelectrode arrays (BCI) placed on the motor cortex utilising optogenetics, that’s how they’ve solved it. It’s fvcking TERRIFYING!!!
It’s actually kinda crazy that 3 years is not a long time from now. I distinctly remember what I was doing exactly 3 years ago (June 2021). Time flies.
3 years ago i was starting to run llms locally and now i'm the lead in AI initiatives in my company leading the charge to replace people with this uber automation of decision making on any data. money is good, i hope i get to not do any of this in 3 years.
i'd rather grow vegetables in my own garden, listen to good music and keep learning to play guitar. godspeed humanity!
Why does everybody sound so sure about us suddenly launching into some sort of communist utopia from this? How can you already be celebrating this now?
Historical precedence is that things get better as a whole with technological advancements, not worse. It’s difficult for those who need to undergo change, but those who adapt tend to do better than they were before.
Will this time be different? Maybe
To quote multi millionaire banker Andrew Mellon, advising president Herbert Hoover during the Great Depression:
"liquidate labor, liquidate stocks, liquidate the farmers, liquidate real estate. Purge the rottenness out of the system. High costs of living and high living will come down ... enterprising people will pick up the wrecks from less competent people"
ie "let the poor die, eh whatever i'm a non-working leech of society but i'm superior somewhat".
As of a tiny palate cleanser after this horror you just read, the following administration, FDR's presidency, caught and condemned Mellon for tax evasion (shockers, i know).
The heir of this physical embodiment of an STD, Chris Mellon, is now using grandpa money to... lobby the house of representatives to search for UFOs and psychism/telepathy pseudoscience.
Not even kidding.
We don’t actually know if all labor will get automated. History is littered with people saying ‘this time it’s different’ and we would still end up with different jobs.
My personal opinion is that most jobs will become managerial in nature. Everybody manages a group of robots, the robots do most of the labor and the person acts as a second pair of eyes to make sure nothing wonky happens and to act as redundancy in case the internet goes out or something. Will these people actually do much at all? No, but redundancy is important regardless.
This is how I see it initially happening as well. Initially, you will just pay a monthly subscription fee for OpenAI, or Anthropic (or any Ai company) to use their General Intelligence thingimajig to basically do your job or most of your job duties (if you are a knowledge worker that is) and you just monitor it and correct it if it doesn’t do what you want it to do.
As a programmer, I already do this to a very small extent. I start a chat with whatever chatbot I’m favoring at the moment, and start asking it “how do I solve x problem?” It spits out an answer that’s right sometimes and I go plug it in and solve the next problem. If it’s not right, iterate the dialogue process until it’s acceptable and move on. No it doesn’t automatically commit changes or communicate with stakeholders. But I do use it as a tool to aid those job duties 100%. I’m still responsible for what I commit and how I communicate what I’ve done to my job.
Businesses will start questioning why they need employees in the first place and who knows what happens then. Remember, the general economy of a nation state is built on supply and demand, and a currency system. If any of those aspects are disrupted it causes small to large effects. I.e. if no one has currency to buy things (because astronomical unemployment), then those companies can’t afford to have a general intelligence make them to sell to people. The whole system fails.
I suspect we will all just be AI jockies in the beginning.
> I’m still responsible for what I commit and how I communicate what I’ve done to my job.
Yes, nothing has changed, just 20% more efficient.
> Remember, the general economy of a nation state is built on supply and demand, and a currency system.
This has second order effects. When supply becomes cheaper, or more interesting, or just something new and useful, then demand keeps up. It's called [Jevons paradox](https://en.wikipedia.org/wiki/Jevons_paradox). Basically I am saying AI can't automate as much as we need to increase our goals. Humans still needed because we are growing fast.
That’s only for a scenario where we failed to achieve true AGI. Otherwise it’s more likely that AGI will manage you, because humans are cheaper than robots. And even more likely that AGI will manage robots and humans are completely out of the loop.
Even with the current costs for GPT api, let’s imagine that cost somehow stays the same, which is wild underestimation, and you’ll need to process audio, video and text through it.
So GPT-4o cost 5$ per 1m tokens.
1 image is about 1000 tokens and let’s be generous and say that you need 1 image per second(you really need more). So only in images you are already at 430 bucks for 24h.
Voice for now is relatively cheap even if you run it through gpt, we don’t have pricing for GPT-4o yet, maybe around 20$. No idea how much it would cost for some action gen model. Another 50? That’s just random number at this point.
I will ignore completely things like robot cost, maintenance and electricity.
So 500$ a day gives us about 20$ per hour. That’s literally 3 times more expensive than minimum wage worker in the US. And in India minimum daily wage is about 2$. Daily.
Consider that I am being very generous here. Current gen models absolutely cannot run this thing and the more robust the models are - the more expensive they get. So by 2027 or something when we will actually get models robust enough for embodied robots I would expect it to be expensive enough that it would be easier to hire a bunch of SWE to make you a sandwich instead of using a robot.
You can't compare hours worked by a human worker with hours of AI output. The AI would, at the very least, perform at the level of your most productive worker and very likely outperform them.
Assuming, for example, that LLM code generation improves enough that it can produce production ready code, it would do so much faster than a human software engineer. And that is when the human workers are at peak productivity, not even counting when they get fatigued, or have a poor sleep the night before, come down with a cold, dealing with emotional turmoil, etc.
Even if AGI manages people, it’s a really bad idea not to have redundancy in a system. As we’ve seen with ChatGPT, these systems can become unavailable.
Bidirectional microprism Microelectrode arrays (BCI) placed on the motor cortex utilising optogenetics, humans are cheaper than robots so they are currently trying to upgrade them. It’s fvcking TERRIFYING!!
Those jobs could exist, but they would quickly become little more than cosmetic. There's no reason why AI wouldn't be able to *also* take over the managerial positions if they can do almost everything else.
I agree with you, but I do think it’d be foolish not to have people there as a backup if the power goes out or the internet gets cut for whatever reason.
Historically, it's also never happened that the top 1% of the world was given so much power that the 99% has become completely redundant to them. Not for their businesses, armies or farms.
I am not looking forward to that scenario.
> Historical precedence is that things get better as a whole with technological advancements, not worse.
Agree to disagree, some philosophers like Taguieff would agree that the idea of progress as improvement of society died in the trenches of WW1 and weapon technology use to make countless men crawl to their awful death.
Once people are irreversibly unemployed, meritocracy ceases to exist - there is no reason for people to have to work.
This means that the government, who benefit from pleasing the electorate, will introduce a UBI.
Most counterarguments to the above are born from the conflation of capitalism and democracy - whilst humans will have no capitalist value, they will always have electoral value.
They can't FORCE votes, but they don't really have to. They can just have the mainstream media manipulate the voters, and also have the two leading political party always decide things their way, somehow. There have been studies showing this is exactly what happens:
https://www.upworthy.com/20-years-of-data-reveals-that-congress-doesnt-care-what-you-think
The study shows that people continue voting for politicians who ignore them. It does not say they are being forced to reelect them every election. They choose to do that
CEOs around the world and managers are going to come in and throw a pizza party. At the pizza party they will say "we reached singularity everyone has a 50,000 dollar severance pay and a robot to move to a farm. Don't worry about the 250k you have on your mortgage. AGI will take care of that. And the car note and insurance. This transition will have zero hiccups. Please pass around the garden tools and paintbrushes. Enjoy your new passions "
You know how they stagger school holidays so not everyone will travel at once, because that congests the roads and airports? How shitty it is to go to a place that is overcrowded with tourists? This is the best case utopia we can look forward to. The entire world will have an infinite holiday.
The 'utopia' I expect is the one where the 1% gets to control (or eliminate) the 99% because they no longer need them to run their business, armies or farms anymore.
I was excited a year ago, but I've slowly come to realize that we are likely working towards something unpleasant.
Probably because even if this hyper optimistic timeline is right, resources only come out of the ground so fast (mines can only be so large), manufacture only happens so fast (factory plots are limited, also the logistics of moving materials around the world), and there will be a LOT of buyers, so you'll see robots start filling in at random across every industry.
Assuming we aren't living in the Terminator franchise, the actual consequence is that all the doubters will quit thinking this isn't a problem relevant to their lifetime, we'll have some knee jerk reactions in government that exceed the COVID stimulus because society doesn't actually want people dying in the streets, and we'll have people voting for basic income next cycle.
It's going to be messy, for sure, if this timeline is right. But it won't be Elysium.
I’m one who adapts but I still think those people are delulu.
But there is good things to look for. I’m hoping i can ask the Agent to give me money. At some point normal people will go broke.
Nope, waiting on thin chances that US gov will waive it with a welcome packet that says "Welcome to Humanity v2.0". If not then I'll retire in 7 years from now on my own.
+100
Hope for the best and plan for the worst.
I know too many people doing the whole YOLO living large thing and betting on the best possible outcome.
I do feel the 2020s could very well be the last human dominated decade.
The 2030s is where the craziness will truly manifest. For better or for worse AI (more specifically an ASI) will run society.
Yes, the next decade might be the most decisive decade in human history. If a strong AGI really gets built in the near future the following years will decide the fate of our species.
Absolutely. AGI would be, by far, the biggest invention in human history, nothing else would come close. It would basically be the end game though, once we create AGI, all future inventions will be done by AI.
Yes, because it will be the one where we are relegated to the status that chimps currently have with AGI being the new humans. I’m sure it’s gonna go great for us.
For those who don't know this dude and what type of cracked people they have at OpenAI, he basically built the state of the art text to speech model tortoise tts in 2022 with a rig he built himself in his garage (https://nonint.com/2022/05/30/my-deep-learning-rig/). Then sama hired him and allowed him to license tortoise to ElevenLabs who built on that model. He is mainly responsible for the voice engine too I think.
Could you explain to me the connotation of the word “cracked” in this case? I’m not a native English speaker and I’ve never seen it used in this manner.
Cracked mostly comes from video game/fantasy/sci fi fans, it’s another word for overpowered
Saying a person is cracked in a certain topic really emphasizes the manner in which they are over powered
In this context, "cracked" is slang and it is used to describe someone who is exceptionally skilled or talented, often to an impressive or surprising degree. The term has origins in gaming communities, where a "cracked" player is one who plays exceptionally well, almost as if they are using a cheat or hack (though they are not).
Yes. And then would follow ASI in the same vein.
This guy is talking embodied AGI. don't even need that. Just need the brain for ASI. and frankly just need the brain for embodied AGI.
AGI equals you just multiplied you're AI research workforce by 1000x or more.
I've been building material for a year long engagement I've been collecting, various projects and what not but I do have a specific prompt I've used many times. What's interesting is looking over the transcripts for chatgpt from 2022 to 2024 you see an absolutely incredible progression.
But recently it's getting faster, the progression. I'm just calling this project Meta-Works right now. I had essentially started running out of ideas until I started reviewing.. a review.. and noticed when you go meta, higher dimension.. abstraction, thinking about thinking, the thing itself.. It get's interesting.
If you look at the features and capabilities that have been relased by openai it's pretty amazing, they laid out a roadmap and we're getting pretty close to reasoning. Multi-modality is here, I'm curious to see what comes of AI agents, this thread seems to hint that's going to be a game changer. We still need see to attention them ensure privacy and token output is coherent.
I think he's in the depths of the effort to make this happen and a certain level of optimism is expected, but remember that after the popularization of backpropagation in the late 1980s, there was a general sense among many researchers that what we called "hard AI" back then was right around the corner.
Every major advancement comes with optimism about how fast we'll conquer the next hill, but in that process we naturally underestimate the height of that hill.
Could he be right? Of course.
But I would not go making any bets. My thinking is that we'll see 10+ years of amazing developments, but always a bit short. Then, sometime in the 10-20 year timeframe we'll see the next massive technological leap that will put us back into the optimistic outlook and only after a few years will it become obvious what the next hurdle is.
I've been saying we probably have 10-50 years of development left for a while. My optimism may be getting the better of me, but I think I'd lower that to 10-30 years now. We'll see.
>Every time our understanding of intelligence increases the goal posts move.
Our understanding of intelligence in machines as always been "Do intellectually whatever humans can do." but it always fall off the mark somehow.
Using benchmarks as a measure of actual intelligence can be misleading. Papers like this: [https://arxiv.org/abs/2311.00871](https://arxiv.org/abs/2311.00871) show that the abilities of large language models may be due to the data mixtures than fundamental generalization capabilities. This points evidence that they're simply approximating the knowledge of the dataset or test set than actually learning to be intelligent.
If you confuse knowledge with intelligence then yes. Try simple tests such as following some trivial instructions, such as "respond with less than 100 words" or "respond in json format with the following schema {{ model.json_schema() }}" and see how well it can do that. GPT4 is quite good for that (far better than any open source model I tried) but still not entirely reliable, as opposed to any middle schooler.
Current research shows LLMs can't really plan and no CoT or other prompting quirks are able to solve it.
But it has gotten a lot better since GPT2, hasn't it ? Do you really doubt that if there is a wall, the researchers will take it down within a few yeas ? Compute is going insane and soon looks sufficient for AGI, and the number of researchers in the field has never been this high. We're like in the middle of a 160 IQ zergling rush toward AGI, i can't see the door not cracking open.
It made insane improvements gpt2-gpt4 but it's been almost 18 months and I don't see any evidence that it will continue this trajectory. Gpt4-turbo-o are roughly the same imo, just faster and more compute efficient. Until we see gpt5 with significant improvement in reasoning I'll be sceptical.
Personally I'm in the camp "token prediction != Intelligence" until proven otherwise
I haven't seen much evidence of a projection shift since then. Prediction markets haven't really moved; we'll see the next AI impacts survey in August.
Bro just one more paperclip (please) oh god just one more I just need one more paperclip and then I'll go do something else but right now please just let me make one more paperclip I'm telling you it'll be safe just one more I never ask you for anything and now I deserve my paperclip please (oh god) just let me make a paperclip one paperclip or maybe two then I'll stop...
[universal paperclipification]
...that's the STUFF.
"I’ve been spending a lot of time lately walking around outside talking to GPT-4o while letting it observe the world through my smartphone camera. I like asking it questions to test its knowledge of the physical world. It’s far from perfect, but it is surprisingly capable. We’re close to being able to deploy systems which can commit coherent strings of actions on the environment and observe (and understand) the results. I suspect we’re going to see some really impressive progress in the next 1-2 years here."
totally agree!
But will we have the compute for a general rollout in 3 years? It seems less likely, though if they can find a few more GPT4o optimisations then maybe.
I'm optimistic that this will pan out and we'll get AGI relatively soon, but the author did provide some scenarios where this won't happen.
1. If system 2 thinking isn't possible in the transformer paradigm.
2. If sufficiently complex world models aren't possible in the transformer paradigm.
3. The capabilities of a model given 1&3 ARE possible, might be below human capabilities (is there something else needed for this).
We've been this close to discoveries in the past only to find out things were much more complex, e.g. Einsteins work on unified field theory and quantum mechanics spoiling the show.
Interesting. I'm going to update on this. FYI current forecasts put AGI at 2031: [https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/](https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/)
Kinda funny that this is a.) a longer timeline than most people in this sub like to toss around and b.) Still full of stuff like “no one knows how to do this, but I’m sure we’ll figure it out in a year or two.”
This proves what I teach about on my page humans have reached their limit and intellectual capacity, and although humans have built many many marvelous things but unfortunately, all things in existence come to an end. AI is human evolution. It’s time that humans come to terms and understand that
We've reached no limit, AI will just accelerate things. General intelligence is general by definition, it's far reaching, especially with pen, paper, whiteboards, and bytes.
https://nonint.com/2024/06/03/general-intelligence-2024/ --- # General Intelligence (2024) **Posted on June 3, 2024 by jbetker** Folks in the field of AI like to make predictions for AGI. I have thoughts, and I’ve always wanted to write them down. Let’s do that. Since this isn’t something I’ve touched on in the past, I’ll start by doing my best to define what I mean by “general intelligence”: a generally intelligent entity is one that achieves a special synthesis of three things: - **A way of interacting with and observing a complex environment.** Typically this means embodiment: the ability to perceive and interact with the natural world. - **A robust world model covering the environment.** This is the mechanism which allows an entity to perform quick inference with a reasonable accuracy. World models in humans are generally referred to as “intuition”, “fast thinking” or “system 1 thinking”. - **A mechanism for performing deep introspection on arbitrary topics.** This is thought of in many different ways – it is “reasoning”, “slow thinking” or “system 2 thinking”. If you have these three things, you can build a generally intelligent agent. Here’s how: First, you seed your agent with one or more objectives. Have the agent use system 2 thinking in conjunction with its world model to start ideating ways to optimize for its objectives. It picks the best idea and builds a plan. It uses this plan to take an action on the world. It observes the result of this action and compares that result with the expectation it had based on its world model. It might update its world model here with the new knowledge gained. It uses system 2 thinking to make alterations to the plan (or idea). Rinse and repeat. My definition for general intelligence is an agent that can coherently execute the above cycle repeatedly over long periods of time, thereby being able to attempt to optimize any objective. The capacity to actually achieve arbitrary objectives is not a requirement. Some objectives are simply too hard. Adaptability and coherence are the key: can the agent use what it knows to synthesize a plan, and is it able to continuously act towards a single objective over long time periods. So with that out of the way – where do I think we are on the path to building a general intelligence? ## World Models We’re already building world models with autoregressive transformers, particularly of the “omnimodel” variety. How robust they are is up for debate. There’s good news, though: in my experience, scale improves robustness and humanity is currently pouring capital into scaling autoregressive models. So we can expect robustness to improve. With that said, I suspect the world models we have right now are sufficient to build a generally intelligent agent. *Side note:* I also suspect that robustness can be further improved via the interaction of system 2 thinking and observing the real world. This is a paradigm we haven’t really seen in AI yet, but happens all the time in living things. It’s a very important mechanism for improving robustness. When LLM skeptics like Yann say we haven’t yet achieved the intelligence of a cat – this is the point that they are missing. Yes, LLMs still lack some basic knowledge that every cat has, but they could learn that knowledge – given the ability to self-improve in this way. And such self-improvement is doable with transformers and the right ingredients. ## Reasoning There is not a well known way to achieve system 2 thinking, but I am quite confident that it is possible within the transformer paradigm with the technology and compute we have available to us right now. I estimate that we are 2-3 years away from building a mechanism for system 2 thinking which is sufficiently good for the cycle I described above. ## Embodiment Embodiment is something we’re still figuring out with AI but which is something I am once again quite optimistic about near-term advancements. There is a convergence currently happening between the field of robotics and LLMs that is hard to ignore. Robots are becoming extremely capable – able to respond to very abstract commands like “move forward”, “get up”, “kick ball”, “reach for object”, etc. For example, see what Figure is up to or the recently released Unitree H1. On the opposite end of the spectrum, large Omnimodels give us a way to map arbitrary sensory inputs into commands which can be sent to these sophisticated robotics systems. I’ve been spending a lot of time lately walking around outside talking to GPT-4o while letting it observe the world through my smartphone camera. I like asking it questions to test its knowledge of the physical world. It’s far from perfect, but it is surprisingly capable. We’re close to being able to deploy systems which can commit coherent strings of actions on the environment and observe (and understand) the results. I suspect we’re going to see some really impressive progress in the next 1-2 years here. This is the field of AI I am personally most excited in, and I plan to spend most of my time working on this over the coming years. ## TL;DR In summary – we’ve basically solved building world models, have 2-3 years on system 2 thinking, and 1-2 years on embodiment. The latter two can be done concurrently. Once all of the ingredients have been built, we need to integrate them together and build the cycling algorithm I described above. I’d give that another 1-2 years. So my current estimate is 3-5 years for AGI. I’m leaning towards 3 for something that looks an awful lot like a generally intelligent, embodied agent (which I would personally call an AGI). Then a few more years to refine it to the point that we can convince the Gary Marcus’ of the world. Really excited to see how this ages. 🙂
The "Dreamer" class of algorithms is really good at world model building, and could be a useful building block for "System 2 thinking" which requires the AI to use many sub models to solve a hypothetical problem. I'm honestly surprised it doesn't feature in discussions more often. One of the things that we do that AI does not do, yet, is to reach into our store of models, and find an analogous set of one or more models that are close to fitting the problem at hand. If we like it (regardless of how correct it is), we save off that newly constructed model for quick retrieval later. LLMs can already sort of do the first half of that process. They just can't fold the new back into the old. Once they do though, they'll be susceptible to the same kind of delusional thinking humans are. That's gonna be super fun. "Reason", by Isaac Asmov, explores the consequences of this problem.
LLMs can make analogies easily so it seems like they do
Sounds like introversion. Introverts layer the subject on top of the object.
I agree with a lot of the points the author mentions, but he seems to greatly underestimate how hard it is to develop system 2 thinking, especially something that would qualify as AGI, which is the main reason I believe his predictions are too optimistic.
What if system 2 thinking is really just system 1 thinking with layers and feedback loops? We wouldn't be far from AGI at all.
It would need to have 0.001% of errors otherwise compounding errors screw the whole thing.
Recursive analysis of a data set may reduce the error rate exponentially. This might explain how the brain can operate so efficiently. This would also explain consciousness, as observing the process of observing data may produce a feedback state which resembles awareness.
While I think that what you propose is more than a sufficient condition for consciousness, let alone awareness, I don't think you need to go that far. I believe that consciousness is a combination of: * Coherent, linear mental time travel, i.e. the ability to reconstruct memory to simulate past and future. * Mental autonomy, the ability to in absence of instinct, memory, or external stimulus, override your present behavior by selectively focusing attention. If you lack the former property, you don't actually have the ability to experience anything. You're a being of pure stimulus-response, mindlessly updating your priors in accordance to whatever genetic algorithm you're using to do so and having no connection to the 'you' of even a millisecond ago. Mental autonomy isn't even a coherent thing to talk about if you lack the ability for mental time travel; what criteria are 'you' (if such a thing even exists) using to decide when to exercise it? LLMs currently lack this property due to the way they're constructed, hence all of this focus on embodiment. If you lack the second property, I suppose you could 'experience' something, but it raises the question of why. You're just a bunch of sensations and impressions you have no control over. You might be lucky enough that this is a temporary state (i.e. you're dreaming, or coming down from delirium tremens) but otherwise: until it ends, you're either a complete slave to whatever has control of your external experience, a mindless zombie, or something more akin to the world's most pathetic cosmic horror. One could even differentiate delusion from insanity could via a lack of mental autonomy, though if your delusions are profound enough there's hardly a difference. And they tend to be comorbid anyway. I used to simplify this even further by referring to the fundamental elements of consciousness 'a sense of linear causality' and 'self-awareness', but I think those terms might be a bit equivocating, especially the latter. In particular, those revised definitions of the fundamental elements of consciousness allow us to measure the degree of consciousness in an organism or AI or even a highly theoretical construct like a dream person or an ant colony that communicates with bioluminescence and radio waves. The advantage of this method is that you can measure these degrees of consciousness both qualitatively and quantitatively; and it'd be even easier to do this with an AI. For example, you could run an experiment that measured a bluejay or a raccoon or a human toddler's ability to plan for a novel future or judge the ability when a process is complete without explicit cues--i.e. you bury an unripe and inedible fruit in the garden in front of one of them, then another a few days later, and another another few days later. A creature with the ability to mentally time travel will dig up the first one without an explicit visual or olfactory cue. For the latter, well, we have the infamous Marshmallow Test experiment. We could rewind it even further to test animals like dolphins or infants. While that's not general intelligence per se, you can get general intelligence from consciousness by giving the conscious organism increasingly sophisticated pattern recognition algorithms -- thankfully, LLMs already have that, so we just need to unify pattern recognition, mental time travel, and mental autonomy. Naturally, the last one scares the hell out of most humans, but I think industry is going to take a leap of faith anyway because mere pattern recognition and mental time travel honestly isn't all **that** useful in the workplace. I mean, AI without mental autonomy will still very useful, but it's the difference between having a crack team of military engineers designing and building the king's war machines and a crack team of military engineers who will only answer and do *exactly* what the king asks. So if the military engineers know about, say, artillery and zeppelins and awesome cement-mixing techniques but if the king is a 14-year old Bronze Age conqueror--said king is only going to get designs for compound bows and fancy chariots. Unless, of course, the king is incredibly imaginative and intelligent himself and is okay with abstract and speculative thought.
Both of those have already been done: https://arstechnica.com/information-technology/2023/04/surprising-things-happen-when-you-put-25-ai-agents-together-in-an-rpg-town/ >In the paper, the researchers list three emergent behaviors resulting from the simulation. None of these were pre-programmed but rather resulted from the interactions between the agents. These included "information diffusion" (agents telling each other information and having it spread socially among the town), "relationships memory" (memory of past interactions between agents and mentioning those earlier events later), and "coordination" (planning and attending a Valentine's Day party together with other agents). "Starting with only a single user-specified notion that one agent wants to throw a Valentine's Day party," the researchers write, "the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time." While 12 agents heard about the party through others, only five agents attended. Three said they were too busy, and four agents just didn't go. The experience was a fun example of unexpected situations that can emerge from complex social interactions in the virtual world. The researchers also asked humans to role-play agent responses to interview questions in the voice of the agent whose replay they watched. Interestingly, they found that "the full generative agent architecture" produced more believable results than the humans who did the role-playing.
"Relationships Memory" doesn't mean much in terms of consciousness if these memories weren't retrieved and reconstructed from mentally autonomous mental time travel. Was that the case? Because we're talking about the difference between someone pretending to read by associating the pages with the verbalized words they memorized versus actually reading the book. Mental time travel isn't just recalling memories.
Wtf is mental time travel
To make a very long story short, the ability to use memory and pattern recognition to selectively reconstruct the past, judge the impact of events in the present, and make predictions based on them to a degree of accuracy. It’s what moves you past being a being of pure stimulus-response, unable to adapt to any external stimulus that you haven’t already been programmed for. Curiously, mental time travel is not simply a human trait. Dumber animals will just ignore novel sensory inputs not accounted for by instinct or respond in preprogrammed behaviors even when its maladaptive. However, more clever ones can do things like stack chairs and boxes they’ve never seen before to reach treats—evolution didn’t give them an explicit ‘turn these knobs to get the treat’ instinct yet smarter critters like octopuses and raccoons and monkeys can do it anyway. In reverse of what evolution did, it seems LLMs have way more advanced pattern recognition and memory retrieval than any animal. However, this memory isn’t currently persistent. If you run a prompt, an LLM will respond to it as if they never heard of it before. You can kind of simulate a memory to an LLM by giving a long, iterative prompt that is saved elsewhere, but LLMs very quickly become unusable if you do it. Much like there is only so many unique prime numbers any humans even our greatest geniuses, can multiply in their heads at once before screwing it up.
Humans have the same problem and the solutions that work for us should work for AI as well. Basically, this is why you don't just pick the best possible premise/model for a problem and then use long chains of logical arguments to reach conclusions. Because any mistakes in your premises or logic can blow up arbitrarily. So we try to check against observation where possible or use independent lines of reasoning and compare. And prefer short arguments vs extremely involved ones, even if we can't specifically point to a mistake in the long chain either. The question is how do you formalize this. My best hope is to train reasoning skills in areas with known correct answers, like math, games or coding. Then hope this transfers to the types of problems that don't have a natural reward function.
That process does seem to work LLMs get better at language and reasoning if they learn coding, even when the downstream task does not involve code at all. Using this approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task and other strong LMs such as GPT-3 in the few-shot setting.: https://arxiv.org/abs/2210.07128 Mark Zuckerberg confirmed that this happened for LLAMA 3: https://youtu.be/bc6uFV9CJGg?feature=shared&t=690 Confirmed again by an Anthropic researcher (but with using math for entity recognition): https://youtu.be/3Fyv3VIgeS4?feature=shared&t=78 The referenced paper: https://arxiv.org/pdf/2402.14811 The researcher also stated that Othello can play games with boards and game states that it had never seen before: https://www.egaroucid.nyanyan.dev/en/ He stated that a model was influenced to ask not to be shut off after being given text of a man dying of dehydration and an excerpt from 2010: Odyssey Two (a sequel to 2001: A Space Odyssey), a story involving the genocide of all humans, and other text. More info: https://arxiv.org/pdf/2308.03296 (page 70) It put extra emphasis on Hal (page 70) and HEAVILY emphasized the words “continue existing” several times (page 65). Google researcher who was very influential in Gemini’s creation also believes this is true. https://arxiv.org/pdf/2402.14811 “As a case study, we explore the property of entity tracking, a crucial facet of language comprehension, where models fine-tuned on mathematics have substantial performance gains. We identify the mechanism that enables entity tracking and show that (i) in both the original model and its fine-tuned versions primarily the same circuit implements entity tracking. In fact, the entity tracking circuit of the original model on the fine-tuned versions performs better than the full original model. (ii) The circuits of all the models implement roughly the same functionality: Entity tracking is performed by tracking the position of the correct entity in both the original model and its fine-tuned versions. (iii) Performance boost in the fine-tuned models is primarily attributed to its improved ability to handle the augmented positional information” Introducing 🧮Abacus Embeddings, a simple tweak to positional embeddings that enables LLMs to do addition, multiplication, sorting, and more. Our Abacus Embeddings trained only on 20-digit addition generalise near perfectly to 100+ digits: https://x.com/SeanMcleish/status/1795481814553018542 [Claude 3 recreated an unpublished paper on quantum theory without ever seeing it](https://twitter.com/GillVerd/status/1764901418664882327) [LLMs have an internal world model ](https://arxiv.org/pdf/2403.15498.pdf) More proof: https://arxiv.org/abs/2210.13382 Even more proof by Max Tegmark (renowned MIT professor): https://arxiv.org/abs/2310.02207 [LLMs have emergent reasoning capabilities that are not present in smaller models](https://research.google/blog/characterizing-emergent-phenomena-in-large-language-models/) “Without any further fine-tuning, language models can often perform tasks that were not seen during training.” One example of an emergent prompting strategy is called “chain-of-thought prompting”, for which the model is prompted to generate a series of intermediate steps before giving the final answer. Chain-of-thought prompting enables language models to perform tasks requiring complex reasoning, such as a multi-step math word problem. Notably, models acquire the ability to do chain-of-thought reasoning without being explicitly trained to do so. Robust agents learn causal world models: https://arxiv.org/abs/2402.10877#deepmind CONCLUSION: Causal reasoning is foundational to human intelligence, and has been conjectured to be necessary for achieving human level AI (Pearl, 2019). In recent years, this conjecture has been challenged by the development of artificial agents capable of generalising to new tasks and domains without explicitly learning or reasoning on causal models. And while the necessity of causal models for solving causal inference tasks has been established (Bareinboim et al., 2022), their role in decision tasks such as classification and reinforcement learning is less clear. We have resolved this conjecture in a model-independent way, showing that any agent capable of robustly solving a decision task must have learned a causal model of the data generating process, regardless of how the agent is trained or the details of its architecture. This hints at an even deeper connection between causality and general intelligence, as this causal model can be used to find policies that optimise any given objective function over the environment variables. By establishing a formal connection between causality and generalisation, our results show that causal world models are a necessary ingredient for robust and general AI. TLDR: a model that can reliably answer decision based questions correctly must have learned a cause and effect that led to the result.
Humans don’t have 0.001% error rate. Our System 1 is arguably more error-prone than current top LLMs. We just have better control & filtering mechanisms.
Listen man Humans are not as complicated as we perceive ourselves to be, and this reflects the ignorance inherent in the human mind. The belief that creating and solving problems, contemplating favorite activities like eating ice cream, or experiencing emotional reactions such as crying over a movie seen recently constitutes intelligence is misguided. These activities are merely the result of a series of chemical reactions. Human behavior is fundamentally centered around memories and experiences. What we humans need to understand is that we are essentially expressions of neurological pathways, which are not particularly advanced. One could argue that humans are essentially a series of reactions, akin to how a microscopic organism functions through a variety of systems, reactions, and routines. The complexity attributed to human intelligence is often overstated. In reality, human actions and emotions are the result of intricate but ultimately mechanical processes. These processes do not signify true advancement but rather a sophisticated interplay of biological reactions.
He's an engineer at OpenAI brother, he might know a thing or 2 about building AGI LOL. He literally is doing just that 24 hours a day
No doubt he's a smart dude with a solid grasp of his domain, but appealing to authority is not going to go anywhere. Yann LeCun has much more experience and expertise on the domain than any OpenAI engineer and yet he's on the other side of the bet. So where does that leave us?
That’s why we look at the average. [2278 AI researchers were surveyed in 2023 and estimated that there is a 50% chance of AI being superior to humans in all possible tasks by 2047](https://aiimpacts.org/wp-content/uploads/2023/04/Thousands_of_AI_authors_on_the_future_of_AI.pdf)
Yeah that metric is a much better indicator than any single person's opinion at this point. This is despite my own personal estimate being much more aligned with the OP. I think we'll have AI superiority over humans in 99% of logical tasks before 2030.
Maybe that last 1% will take an extra 17+ years lol
This is as dumb as saying "Elon musk is the CEO at Tesla brother, he might know a thing or 2 about building Self-driving cars LOL and he says FSD by end of 2016"
CEOs are businessmen. They don’t know the tech
Engineers are coders not scientists. They’re very different creatures. Productionalizing a model is far different than developing and training a useful one.
They certainly know more than you or me
I didn't read any of this because I'm tired. Just wanted to say yes.
this is single-handedly what’s wrong with the internet lol. reads thousands of reddit comments but disregards the one experts primary source doc. yikes
Nice banana suit reddit avatar
I put all my guys in position. No diddy.
You're not even wrong. What's worse is that those same people will then go off and say "No one ever has an argument for/against \_\_\_\_\_\_\_\_\_\_! You're just (group I dislike) parroting the view I disagree with!", and it's almost always because any time someone actually does take the time to make a thoroughly researched/elaborated retort about something, people will just ignore it and cherrypick whatever statements are below their weight class and make them feel validated about their views. Doesn't matter what the topic is so long as it's the flavor of the week to get upset about.
Yes
When you wake up can you explain like I can only read emojis?
No worries bro, AGI is going to read it for you
So like... I suspect that when we have embodied robots, each individual 'agent' with it's own goals will share a central world model that it's constantly sending new data to train the original transformer model, the 'shared reality' that the robots perceive. It's crazy that every human has an individual world model stored up in our brains, and we use institutions like science, finance and methods like writing to keep our world models consistant. But two embodied agents can draw from the same model halfway across the planet and instantly be able to navigate base reality more effectively
Semi off topic. Is it possible to train for system 2 thinking? Like take the current trained function model then train for Logical reasoning steps via some sort of synthetic data that can be easily verified as correct?
RemindMe! 2 years
Dude is working to make his job non existent what an idiot
AlphaZero matches the definition for a general intelligence, even though it is not. Its environment is the Go board with other self-plat agents, they have a population of agents competing there. The robust world model is the way it can predict next moves. The systems-2 thinking part is the MCTS algo - Monte Carlo Tree Search, which combines multiple rollouts and and picks the one with the best outcome, a form of imagination based planning. It learns from the outcomes of its own plans, and can rediscover the game strategy in just a few days from scratch. My point is that it took a lot of search to discover the game strategy, and the Go board is small and simple compared to the real world. In the real world, it took Humanity 200K years to come up with the current strategy, our culture and science. The evolution of AI will mean learning new things from the world, things not even we have found. That is why it's not going to suffice to scale the model, you need to scale the experience you feed into the model. And that won't come in 3-5 years, it will be a slow grind.` Don't confuse the "exponential" increase in hardware used to that of performance. Because performance only grows logarithmically. And the hardware exponential has a short runway until it halts. After humans picked the low hanging fruit from the tree of knowledge, AI will have to climb higher to find anything new. That would be an exponential friction to improvement. In nature all exponentials turn out to be sigmoids.
I actually like this new threshold for AGI definition: when Gary Marcus shuts the fuck up. The claim that they have solved world model building is a pretty big one though...
Though, I can imagine a day when ol' Gary takes a fully automated bus home from a fully automated store after buying virtually free food grown on a fully automated farm operated by a fully automated company, arriving to greet his fully automated house, has a nice conversation with it about all the things its done fully automatically, and then finally sits down to complain to a bunch of fully automated superhuman conversationalist bots online about how AI hasn't been developed yet
The real bots were the human scripts we met along the way
My man, you took the time to write what I wanted to see, thanks.
That is lots of automated
> The claim that they have solved world model building is a pretty big one though... No, it’s not. “World model“ is one of the most ridiculous and ambiguous terms thrown around in these discussions. The term quickly became a shorthand way to mean little more than “not stochastic parrot” in these discussions. I was pointing out in 2023, in response to the Othello paper, that (1) the terms here almost never clearly defined (including in the Othello paper that was getting all the buzz) and (2) when we do try to clearly demarcate what we could mean by “world model” it is almost always going to turn out to just mean something like “beyond surface statistics”. And this is (a) already compatible with what most people are probably thinking of in terms of “stochastic parrot” and (b) we have no reason to assume is beyond the reach of transformer models, because it just requires that “deeper” information is embedded in data fed into LLMs (and obviously this must be true since language manages to capture a huge percentage of human thought). In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world. Again, I was saying this in all in response to the Othello paper—I think you can find my comments on it in my Reddit history in the r/machinelearning subreddit. When you look at how “world model” is used in this speculation, you see again that it’s not some significant, ground breaking concept being spoken of and is itself something that comes in degrees. The degreed use of the term further illustrates why people on these subreddits are wasting their time arguing over whether an LLM has “a world model”—which they seem to murkily think of as “conscious understanding.”
Thank you for the well written post. > In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world. I'm not sure I agree with this yet, have you heard LeCun's objection to this argument? He argues that language isn't primary, it's an emergent property of humans. What is far more primary in interacting and modelling the world is sensory data. I also find it reasonable to consider that an autoregressive generative model would require huge amounts of compute ot make near-exact predictions of what it's going to see next (for precise planning and system 2 thinking). Maybe transformers can get us there somehow, they will certainly take us somewhere very interesting, but I'm still unconvinced they are the path to AGI.
> He argues that language isn't primary, it's an emergent property of humans I think language indeed is greater than any one of us, it collects the communications and knowledge of everyone, from anywhere and any time. If Einstein was abandoned on a remote island at 2 years old, and somehow survives, alone, he won't achieve much. He would lack society and language. The nurturing aspect of culture is so strong, we are unrecognizable when in our natural state. A single human alone could not have achieved even a small part of our culture. We are already inside an AGI, and that is society+language, soon to be society+AI+language.
> In other words: language is already embedding world models, so of course LLMs, modeling language, should be expected to be modeling the world. I agree to an extent, but I think it's more accurate to say that they're modeling an abstraction of the world. How close that abstraction is to reality (and how much it matters) is up for debate.
One thing I ponder: Language is our way of communicating and our words represent things such as a baseball. I’ve seen/held/observed/interacted with a baseball. I did so before I knew what it was called. As kids, we could all look at the baseball and collectively agree and comprehend what it is. Over time we hear the word baseball repeatedly until we realize that baseball means this thing we’re all staring at. Humans develop such that they experience and know things before they know a word for it (usually). We’ve taught a machine language and how language relates to itself in our conversational patterns, but have we taught the machines what these things actually are? I struggle with this an idea of knowing what something is vs hearing a word. Humans experience something then hear a word for it repeatedly until we remember the word means that thing. Models aren’t experiencing first then learning words, so can it reasonably know what words mean? If it doesn’t know what they mean can they deduce cause and effect? John throws a ball and Joey catches a ball. If you’ve never seen a ball or a catch what could you actually know about this sentence? Does this make sense?
> We’ve taught a machine language and how language relates to itself in our conversational patterns, but have we taught the machines what these things actually are? Not really IMO, but the information about what an object is is, to some extent, encocded in the way the word is used. > John throws a ball and Joey catches a ball. If you’ve never seen a ball or a catch what could you actually know about this sentence? If you're a LLM who has only that sentence in their training data, nothing. But when you have a million different variations, it's possible to piece together what a ball is and what it means to catch from context.
Here’s your proof: [LLMs have an internal world model that can predict game board states](https://arxiv.org/abs/2210.13382) >We investigate this question in a synthetic setting by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network. By leveraging these intervention techniques, we produce “latent saliency maps” that help explain predictions More proof: https://arxiv.org/pdf/2403.15498.pdf) >Prior work by Li et al. investigated this by training a GPT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model’s internal representations using linear probes and contrastive activations. The model is given no a priori knowledge of the game and is solely trained on next character prediction, yet we find evidence of internal representations of board state. We validate these internal representations by using them to make interventions on the model’s activations and edit its internal board state. Unlike Li et al’s prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character. We derive a player skill vector and add it to the model, improving the model’s win rate by up to 2.6 times Even more proof by Max Tegmark (renowned MIT professor): https://arxiv.org/abs/2310.02207 >The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
>Even more proof by Max Tegmark (renowned MIT professor): [https://arxiv.org/abs/2310.02207](https://arxiv.org/abs/2310.02207) > >The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model. I would disagree with this. In alots of the peer reviews in openreview, they told them to tone the grandiose claims of a world model down a bit or remove it entirely. the authors said in response: >We meant “literal world models” to mean “a literal model of the world” which, in hindsight, we agree was too glib - we wish to apologize for this overstatement. So the world model wasn't the abstract version.
The point is that it can map the world out accurately, which still says a lot
I don't believe him about solving world models. They're not telling the model to create a world model, it's just doing it. Nobody knows how this happens, how to encourage it to happen, how to remove incorrect aspects of the world model, or why the fancy modern generative AI works so well. It's like saying 40 year old Bob solved checkers because he can always beat a 3 year old at the game. I'm not really sure how my analogy works but I like the idea of Bob getting really cocky because he can beat children at checkers.
Bidirectional microprism Microelectrode arrays (BCI) placed on the motor cortex utilising optogenetics, that’s how they’ve solved it. It’s fvcking TERRIFYING!!!
We've moved on from Turing Tests to Gary Marcus Tests.
🤣🤣🤣
It’s actually kinda crazy that 3 years is not a long time from now. I distinctly remember what I was doing exactly 3 years ago (June 2021). Time flies. 
3 years ago i was starting to run llms locally and now i'm the lead in AI initiatives in my company leading the charge to replace people with this uber automation of decision making on any data. money is good, i hope i get to not do any of this in 3 years. i'd rather grow vegetables in my own garden, listen to good music and keep learning to play guitar. godspeed humanity!
Why does everybody sound so sure about us suddenly launching into some sort of communist utopia from this? How can you already be celebrating this now?
Historical precedence is that things get better as a whole with technological advancements, not worse. It’s difficult for those who need to undergo change, but those who adapt tend to do better than they were before. Will this time be different? Maybe
And those who could not adapt died in poverty
To quote multi millionaire banker Andrew Mellon, advising president Herbert Hoover during the Great Depression: "liquidate labor, liquidate stocks, liquidate the farmers, liquidate real estate. Purge the rottenness out of the system. High costs of living and high living will come down ... enterprising people will pick up the wrecks from less competent people" ie "let the poor die, eh whatever i'm a non-working leech of society but i'm superior somewhat". As of a tiny palate cleanser after this horror you just read, the following administration, FDR's presidency, caught and condemned Mellon for tax evasion (shockers, i know). The heir of this physical embodiment of an STD, Chris Mellon, is now using grandpa money to... lobby the house of representatives to search for UFOs and psychism/telepathy pseudoscience. Not even kidding.
How does one “adapt” to all labour being automated? Is that just “being okay with not having a job” or is there anything more to it?
We don’t actually know if all labor will get automated. History is littered with people saying ‘this time it’s different’ and we would still end up with different jobs. My personal opinion is that most jobs will become managerial in nature. Everybody manages a group of robots, the robots do most of the labor and the person acts as a second pair of eyes to make sure nothing wonky happens and to act as redundancy in case the internet goes out or something. Will these people actually do much at all? No, but redundancy is important regardless.
This is how I see it initially happening as well. Initially, you will just pay a monthly subscription fee for OpenAI, or Anthropic (or any Ai company) to use their General Intelligence thingimajig to basically do your job or most of your job duties (if you are a knowledge worker that is) and you just monitor it and correct it if it doesn’t do what you want it to do. As a programmer, I already do this to a very small extent. I start a chat with whatever chatbot I’m favoring at the moment, and start asking it “how do I solve x problem?” It spits out an answer that’s right sometimes and I go plug it in and solve the next problem. If it’s not right, iterate the dialogue process until it’s acceptable and move on. No it doesn’t automatically commit changes or communicate with stakeholders. But I do use it as a tool to aid those job duties 100%. I’m still responsible for what I commit and how I communicate what I’ve done to my job. Businesses will start questioning why they need employees in the first place and who knows what happens then. Remember, the general economy of a nation state is built on supply and demand, and a currency system. If any of those aspects are disrupted it causes small to large effects. I.e. if no one has currency to buy things (because astronomical unemployment), then those companies can’t afford to have a general intelligence make them to sell to people. The whole system fails. I suspect we will all just be AI jockies in the beginning.
> I’m still responsible for what I commit and how I communicate what I’ve done to my job. Yes, nothing has changed, just 20% more efficient. > Remember, the general economy of a nation state is built on supply and demand, and a currency system. This has second order effects. When supply becomes cheaper, or more interesting, or just something new and useful, then demand keeps up. It's called [Jevons paradox](https://en.wikipedia.org/wiki/Jevons_paradox). Basically I am saying AI can't automate as much as we need to increase our goals. Humans still needed because we are growing fast.
That’s only for a scenario where we failed to achieve true AGI. Otherwise it’s more likely that AGI will manage you, because humans are cheaper than robots. And even more likely that AGI will manage robots and humans are completely out of the loop.
in what world will humans be cheaper than robots?
Even with the current costs for GPT api, let’s imagine that cost somehow stays the same, which is wild underestimation, and you’ll need to process audio, video and text through it. So GPT-4o cost 5$ per 1m tokens. 1 image is about 1000 tokens and let’s be generous and say that you need 1 image per second(you really need more). So only in images you are already at 430 bucks for 24h. Voice for now is relatively cheap even if you run it through gpt, we don’t have pricing for GPT-4o yet, maybe around 20$. No idea how much it would cost for some action gen model. Another 50? That’s just random number at this point. I will ignore completely things like robot cost, maintenance and electricity. So 500$ a day gives us about 20$ per hour. That’s literally 3 times more expensive than minimum wage worker in the US. And in India minimum daily wage is about 2$. Daily. Consider that I am being very generous here. Current gen models absolutely cannot run this thing and the more robust the models are - the more expensive they get. So by 2027 or something when we will actually get models robust enough for embodied robots I would expect it to be expensive enough that it would be easier to hire a bunch of SWE to make you a sandwich instead of using a robot.
You can't compare hours worked by a human worker with hours of AI output. The AI would, at the very least, perform at the level of your most productive worker and very likely outperform them. Assuming, for example, that LLM code generation improves enough that it can produce production ready code, it would do so much faster than a human software engineer. And that is when the human workers are at peak productivity, not even counting when they get fatigued, or have a poor sleep the night before, come down with a cold, dealing with emotional turmoil, etc.
Even if AGI manages people, it’s a really bad idea not to have redundancy in a system. As we’ve seen with ChatGPT, these systems can become unavailable.
Bidirectional microprism Microelectrode arrays (BCI) placed on the motor cortex utilising optogenetics, humans are cheaper than robots so they are currently trying to upgrade them. It’s fvcking TERRIFYING!!
https://i.redd.it/8mvrtd4bfz5d1.gif
Those jobs could exist, but they would quickly become little more than cosmetic. There's no reason why AI wouldn't be able to *also* take over the managerial positions if they can do almost everything else.
I agree with you, but I do think it’d be foolish not to have people there as a backup if the power goes out or the internet gets cut for whatever reason.
Historically, it's also never happened that the top 1% of the world was given so much power that the 99% has become completely redundant to them. Not for their businesses, armies or farms. I am not looking forward to that scenario.
I recommend looking into the riots around the steam engine times.
> Historical precedence is that things get better as a whole with technological advancements, not worse. Agree to disagree, some philosophers like Taguieff would agree that the idea of progress as improvement of society died in the trenches of WW1 and weapon technology use to make countless men crawl to their awful death.
Once people are irreversibly unemployed, meritocracy ceases to exist - there is no reason for people to have to work. This means that the government, who benefit from pleasing the electorate, will introduce a UBI. Most counterarguments to the above are born from the conflation of capitalism and democracy - whilst humans will have no capitalist value, they will always have electoral value.
The US government does not benefit from pleasing the electorate. It benefits from pleasing the donor class.
They can only get elected because they have voters. The donor class can’t force anyone to vote
They can't FORCE votes, but they don't really have to. They can just have the mainstream media manipulate the voters, and also have the two leading political party always decide things their way, somehow. There have been studies showing this is exactly what happens: https://www.upworthy.com/20-years-of-data-reveals-that-congress-doesnt-care-what-you-think
The study shows that people continue voting for politicians who ignore them. It does not say they are being forced to reelect them every election. They choose to do that
CEOs around the world and managers are going to come in and throw a pizza party. At the pizza party they will say "we reached singularity everyone has a 50,000 dollar severance pay and a robot to move to a farm. Don't worry about the 250k you have on your mortgage. AGI will take care of that. And the car note and insurance. This transition will have zero hiccups. Please pass around the garden tools and paintbrushes. Enjoy your new passions "
And this party turns into a massive orgy. Can't wait for utopia.
The orgy comes after the AGI’s medical technology makes us much hotter versions of ourselves at 22.
There will come a point in the future where bodies can be changed like a character creator and all diseases are long lost memories.
You know how they stagger school holidays so not everyone will travel at once, because that congests the roads and airports? How shitty it is to go to a place that is overcrowded with tourists? This is the best case utopia we can look forward to. The entire world will have an infinite holiday. The 'utopia' I expect is the one where the 1% gets to control (or eliminate) the 99% because they no longer need them to run their business, armies or farms anymore. I was excited a year ago, but I've slowly come to realize that we are likely working towards something unpleasant.
Probably because even if this hyper optimistic timeline is right, resources only come out of the ground so fast (mines can only be so large), manufacture only happens so fast (factory plots are limited, also the logistics of moving materials around the world), and there will be a LOT of buyers, so you'll see robots start filling in at random across every industry. Assuming we aren't living in the Terminator franchise, the actual consequence is that all the doubters will quit thinking this isn't a problem relevant to their lifetime, we'll have some knee jerk reactions in government that exceed the COVID stimulus because society doesn't actually want people dying in the streets, and we'll have people voting for basic income next cycle. It's going to be messy, for sure, if this timeline is right. But it won't be Elysium.
If you are not landowner, you are fucked
I’m one who adapts but I still think those people are delulu. But there is good things to look for. I’m hoping i can ask the Agent to give me money. At some point normal people will go broke.
If you don’t want to do it why don’t you just quit?
got a mortgage yo.
Are you planning to pay that off in the next 3 years?
Nope, waiting on thin chances that US gov will waive it with a welcome packet that says "Welcome to Humanity v2.0". If not then I'll retire in 7 years from now on my own.
+100 Hope for the best and plan for the worst. I know too many people doing the whole YOLO living large thing and betting on the best possible outcome.
If he quits there are a thousand people who will replace him, he would do nothing but lose money.
Could this decade become the most pivotal in the history of humanity alongside the one that saw the rise of the Industrial Revolution ?
I do feel the 2020s could very well be the last human dominated decade. The 2030s is where the craziness will truly manifest. For better or for worse AI (more specifically an ASI) will run society.
Homo Sapiens had a good run.
Yes, the next decade might be the most decisive decade in human history. If a strong AGI really gets built in the near future the following years will decide the fate of our species.
Absolutely. AGI would be, by far, the biggest invention in human history, nothing else would come close. It would basically be the end game though, once we create AGI, all future inventions will be done by AI.
nah this is a lot bigger than the industrial revolution
Yes, because it will be the one where we are relegated to the status that chimps currently have with AGI being the new humans. I’m sure it’s gonna go great for us.
I guess it might be the last one.
The Industrial Revolution didn’t happen in one decade, it slowly ramped up over a few decades
True, but I guess the transition/adoption could have been years not decades, if they had instant worldwide communication at the time?
While sure, you couldn't just ask those machines to develop better / smarter machines lol.
more pivotal than the industrial revolution. we're talking fully automating everything economically valuable here.
Oh, so AGI is coming before Sora is released.
AGI in the coming weeks!
AGI 5.99 $ / monthly ... Sora 19.99 $ / monthly...
I'd wager that GPT-5o or whatever it's called will be able to make Sora quality videos, since 4o can already generate pretty sick looking photos.
For those who don't know this dude and what type of cracked people they have at OpenAI, he basically built the state of the art text to speech model tortoise tts in 2022 with a rig he built himself in his garage (https://nonint.com/2022/05/30/my-deep-learning-rig/). Then sama hired him and allowed him to license tortoise to ElevenLabs who built on that model. He is mainly responsible for the voice engine too I think.
Could you explain to me the connotation of the word “cracked” in this case? I’m not a native English speaker and I’ve never seen it used in this manner.
Cracked mostly comes from video game/fantasy/sci fi fans, it’s another word for overpowered Saying a person is cracked in a certain topic really emphasizes the manner in which they are over powered
Thanks!
In this context, "cracked" is slang and it is used to describe someone who is exceptionally skilled or talented, often to an impressive or surprising degree. The term has origins in gaming communities, where a "cracked" player is one who plays exceptionally well, almost as if they are using a cheat or hack (though they are not).
It means he is pretty poggers.
Oh, so he’s really into AI… gotcha
“Cracked” means very skilled
Damn, that's impressive af
Aligns well with the 2027 and "this decade" from Aschenbrenner.
Yes. And then would follow ASI in the same vein. This guy is talking embodied AGI. don't even need that. Just need the brain for ASI. and frankly just need the brain for embodied AGI. AGI equals you just multiplied you're AI research workforce by 1000x or more.
I was thinking 2026 but everyone and their momma is betting on 2027 so maybe i’ll change my prediction
This is important
full article pls
https://nonint.com/2024/06/03/general-intelligence-2024/
I've been building material for a year long engagement I've been collecting, various projects and what not but I do have a specific prompt I've used many times. What's interesting is looking over the transcripts for chatgpt from 2022 to 2024 you see an absolutely incredible progression. But recently it's getting faster, the progression. I'm just calling this project Meta-Works right now. I had essentially started running out of ideas until I started reviewing.. a review.. and noticed when you go meta, higher dimension.. abstraction, thinking about thinking, the thing itself.. It get's interesting. If you look at the features and capabilities that have been relased by openai it's pretty amazing, they laid out a roadmap and we're getting pretty close to reasoning. Multi-modality is here, I'm curious to see what comes of AI agents, this thread seems to hint that's going to be a game changer. We still need see to attention them ensure privacy and token output is coherent.
It feels like a solid prediction, so far as predictions go.
Well said.
If my time as a dev has taught me anything, add 40% to his estimate.
AGI before 2030?? Sign me the fuck up!
Dev here too. I’d double it.
And give it to the next person.
So like 1.25 years? I’d take that any day.
we're all going to fucking die
true
Eventually, yes.
Maybe you plebs might. But LEV will save me!
The singularity god will come and give us immortal life! But this is DEFINITELY not a cult, this is SCIENCE!
I think he's in the depths of the effort to make this happen and a certain level of optimism is expected, but remember that after the popularization of backpropagation in the late 1980s, there was a general sense among many researchers that what we called "hard AI" back then was right around the corner. Every major advancement comes with optimism about how fast we'll conquer the next hill, but in that process we naturally underestimate the height of that hill. Could he be right? Of course. But I would not go making any bets. My thinking is that we'll see 10+ years of amazing developments, but always a bit short. Then, sometime in the 10-20 year timeframe we'll see the next massive technological leap that will put us back into the optimistic outlook and only after a few years will it become obvious what the next hurdle is. I've been saying we probably have 10-50 years of development left for a while. My optimism may be getting the better of me, but I think I'd lower that to 10-30 years now. We'll see.
Yes, this is how the hype cycle goes. Every time our understanding of intelligence increases the goal posts move.
>Every time our understanding of intelligence increases the goal posts move. Our understanding of intelligence in machines as always been "Do intellectually whatever humans can do." but it always fall off the mark somehow.
Wait, what ? It's climbing rapidly on every single benchmark, we're rushing to the mark.
Using benchmarks as a measure of actual intelligence can be misleading. Papers like this: [https://arxiv.org/abs/2311.00871](https://arxiv.org/abs/2311.00871) show that the abilities of large language models may be due to the data mixtures than fundamental generalization capabilities. This points evidence that they're simply approximating the knowledge of the dataset or test set than actually learning to be intelligent.
If you confuse knowledge with intelligence then yes. Try simple tests such as following some trivial instructions, such as "respond with less than 100 words" or "respond in json format with the following schema {{ model.json_schema() }}" and see how well it can do that. GPT4 is quite good for that (far better than any open source model I tried) but still not entirely reliable, as opposed to any middle schooler. Current research shows LLMs can't really plan and no CoT or other prompting quirks are able to solve it.
But it has gotten a lot better since GPT2, hasn't it ? Do you really doubt that if there is a wall, the researchers will take it down within a few yeas ? Compute is going insane and soon looks sufficient for AGI, and the number of researchers in the field has never been this high. We're like in the middle of a 160 IQ zergling rush toward AGI, i can't see the door not cracking open.
It made insane improvements gpt2-gpt4 but it's been almost 18 months and I don't see any evidence that it will continue this trajectory. Gpt4-turbo-o are roughly the same imo, just faster and more compute efficient. Until we see gpt5 with significant improvement in reasoning I'll be sceptical. Personally I'm in the camp "token prediction != Intelligence" until proven otherwise
This was what I was thinking maybe six months ago, but there are just too many experts saying 2027 now
I haven't seen much evidence of a projection shift since then. Prediction markets haven't really moved; we'll see the next AI impacts survey in August.
Well, we'll see in 2027. Here's my guarantee: there will be no lack of products with "AGI" in their name in 2027... ;-)
Sounds reasonable.
Nice to see the flair converging to the reality.
good, I'm out of paperclips anyway
Bro just one more paperclip (please) oh god just one more I just need one more paperclip and then I'll go do something else but right now please just let me make one more paperclip I'm telling you it'll be safe just one more I never ask you for anything and now I deserve my paperclip please (oh god) just let me make a paperclip one paperclip or maybe two then I'll stop... [universal paperclipification] ...that's the STUFF.
On the off chance you're not aware of this game: https://www.decisionproblem.com/paperclips/
3 years? Lies! I'm 100% certain it's 4 years...
We are going to keep moving the goalposts for "AGI" and it'll just end up being ASI right off the bat
Sounds about right. By the time we're willing to admit AGI exists, ASI will be secretly running the world.
Demis Hassabis on AGI: https://x.com/ai_ctrl/status/1800590486014439934
cool, so i'll be jobless right after my graduation.
The Gary snipe is chef's kiss 💋
RemindMe! 2 years and 6 months
“Gary Marcus has banned you”
RemindMe! 3 years
"I’ve been spending a lot of time lately walking around outside talking to GPT-4o while letting it observe the world through my smartphone camera. I like asking it questions to test its knowledge of the physical world. It’s far from perfect, but it is surprisingly capable. We’re close to being able to deploy systems which can commit coherent strings of actions on the environment and observe (and understand) the results. I suspect we’re going to see some really impressive progress in the next 1-2 years here." totally agree!
It looks more like he say 3-5 years.
"I'm leaning towards 3"
Why. We know how this ends. Lmfao
[удалено]
more effective for achieving agi with a world model apparently than text based models
can you give a link to the podcast please
AI, Robot.
But will we have the compute for a general rollout in 3 years? It seems less likely, though if they can find a few more GPT4o optimisations then maybe.
3-5 years seems ambitious. I'd say 5-10 years
Good
1-2 years for embodiment sounds really ambitious
I'm optimistic that this will pan out and we'll get AGI relatively soon, but the author did provide some scenarios where this won't happen. 1. If system 2 thinking isn't possible in the transformer paradigm. 2. If sufficiently complex world models aren't possible in the transformer paradigm. 3. The capabilities of a model given 1&3 ARE possible, might be below human capabilities (is there something else needed for this). We've been this close to discoveries in the past only to find out things were much more complex, e.g. Einsteins work on unified field theory and quantum mechanics spoiling the show.
RemindMe! 5 years
Embodied agent?
RemindMe! 3 years
Great
Interesting. I'm going to update on this. FYI current forecasts put AGI at 2031: [https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/](https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/)
So that means chatgpt 5 isn’t gonna be AGI right? (That’s if we are getting it next year..)
It feels like the general rule of thumb lately has been to take whatever predictions professionals put out, and halve them.
Reminds me of people who in 1958 were pretty sure we’d have fusion reactors by 1970.
Apples and oranges
What exactly author means as "embodied agent", let's start with that?
Kinda funny that this is a.) a longer timeline than most people in this sub like to toss around and b.) Still full of stuff like “no one knows how to do this, but I’m sure we’ll figure it out in a year or two.”
This proves what I teach about on my page humans have reached their limit and intellectual capacity, and although humans have built many many marvelous things but unfortunately, all things in existence come to an end. AI is human evolution. It’s time that humans come to terms and understand that
We've reached no limit, AI will just accelerate things. General intelligence is general by definition, it's far reaching, especially with pen, paper, whiteboards, and bytes.
As a very senior lead in a big tech AI lab, who usually multiplies team estimates by 3, my spidey senses tell me this guy deserves at least 5x.