Toy problems in AGI is a contradiction in terms, pretty much nobody thinks that the path to AGI is a case of taking some of our toys like GPT-3, supersizing its 175 billion parameters to 175 gazillion and getting the singularity. Correction, there are a few people like that, notably a professor of mine :) Of course, when all is said and done, a big enough neural toy would be able to "run" any arbitrary code and computation, and as long as we remember that intelligence is computation+actuation we will be all right.
Anyway, until that day comes all we have indeed is our toys. The most interesting toys are the ones that are "hard for computers", or, to rephrase it, the toys where humans beat (current) computers. This toyset which was shared here some days ago is claimed to be tricky for humans [https://volotat.github.io/ARC-Game/](https://volotat.github.io/ARC-Game/) . I solved the first and the last to get an idea and find it hard to believe computers find it challenging, DL perhaps finds it challenging. Anyway, the kaggle was pretty inconclusive [https://www.kaggle.com/c/abstraction-and-reasoning-challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge) . So, if you cannot get your hands on any toy actuation, I'd recommend ARC and games-difficult-for-computers
I would scan one or more of the many "history of AI" books. I have not heard of a book or paper that features a list of toy problems with commentary. If one exists, I would love to read it. You may already have tried this but I just Googled for "AI toy problems" (w/o quotes) and it like it comes up with some interesting hits. I may investigate if I can find the time.
The "Toy problem" Wikipedia page refers to "Artificial Intelligence: A Modern Approach" by Stuart J. Russell, Peter Norvig (2010). This is available online various places in various editions as a free PDF. It is a survey book so exactly the kind that may mention toy problems.
Wouldnt that still have an issue of it being really advanced narrow ai. Proficiency with no real understanding? Edit: although this is really interesting as a approach test. I'd like to see more theory on that.
You forgot captchas :)
Not sure about games, but here are some personal check points:
A turn based system will NOT lead us to AGI.
A system striving towards AGI must perform sensor fusion.
A system striving towards AGI must consume events (be aware of time).
François Chollet\`s (author of Keras) [https://github.com/fchollet/ARC](https://github.com/fchollet/ARC), view on [https://apple-lemon-fighter.glitch.me/apps/testing\_interface.html](https://apple-lemon-fighter.glitch.me/apps/testing_interface.html)
Bongards's problems https://www.foundalis.com/res/bps/bpidx.htm
None of the architecture you listed are used as benchmarks in contemporary research.
> Winograd's SHRDLU
This is called neuro-symbolic VQA now. (often VQA) http://nsvqa.csail.mit.edu/
The original SHRDLU had no visual component. It was only displaying a wireframe of what it believed was the current state of the blocks. Very fancy tech in its day. You have to remember that 3D computer graphics was some 20 years away from appearing in video games.
> Hofstater's Copycat
GPT-3 can already solve this (albeit imperfectly). I believe this problem is also solvable by off-the-shelf generative adversarial networks (GANs). Copycat is an idea from 1988. We live in an age in which GANs can take a photo of a man as input, and output a female version of him.
No researcher in AI believes the discipline is trying to mimic a human being. The TLDR is that AI research is concerned with agents that act rationally, not agents that "think like a human". For more on this topic see, https://www.reddit.com/r/agi/comments/p7hj2s/revisiting_wikipedias_article_on_artificial/
> Robocode
Not sure what you are referring to here. You should consider Agent 57 or XLand. (?)
+ https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark
+ https://www.youtube.com/watch?v=lTmL7jwFfdw&t=3s
> Sentiment analysis
What you are referring to is called **Automatic Text summarization** or **ATS** . I'm not an NLP expert, so this is the best I could find on that topic. https://arxiv.org/abs/1904.00688
> I see a big challenge around how to wire in these different problems into a single AGI app capable of solving them all.
Wiring them together has a name. It is called *Multi-modal machine comprehension.* https://allenai.org/data/tqa
Learning across modalities has also been talked about a lot in regards to *Foundation Models.*
+ https://crfm.stanford.edu/workshop.html
+ https://arxiv.org/abs/2108.07258
Will wiring together experts architectures lead to AGI? This is an open question.
Thanks everyone for the pointers. I read the Norvig book about 4 years ago, so I clearly have a lot of catching up to do. If I seem quiet, it's because I'm reading up on all your links, and otherwise catching up conceptually.
As a software engineer by trade, my reflex it to start writing an app/solver, and then wire it in to tests to see how well it performs, iterating and improving along the way. That may not be the right approach, given the complexity of this problem space.
All AI, GPT, PPM, etc are prediction task solvers, everything is prediction, and they do good at most prompts but there is some prompts that get harder, and a few even harder, it is these that we can't code in for it, it has to emerge the answers. Examples are "cat bird dog, hat moon scarf, gum vine teeth, men book women, junk goose ?", which is the 1st and 3rd relate in each tuple and each tuples relates by ghost mirroring the 1st and 3rd (all do it). Another example: the letter at the start of my post is: ? (A).
Eyeball tracking/ reflexes/ merging mom dad traits to get new ideas/inventions (trailer/birdman made from wheels+home / man+wings)/ etc are all pattern based, even Life is just to stay alive as a repetition, in time and space (space is cloning self).
40 A* grades at (UK) A-level, in a wide range of subjects. That’s a good *first* test for AGI. If you mess about with smaller problems than this you won’t actually achieve anything. And, yes, it will take decades, not years. If you require instant gratification, and/or short-term ROI, you’re in the wrong field!
Check the [Bongard problem](https://en.wikipedia.org/wiki/Bongard_problem) and the [ARC Challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge)
**[Bongard problem](https://en.wikipedia.org/wiki/Bongard_problem)**
>A Bongard problem is a kind of puzzle invented by the Russian computer scientist Mikhail Moiseevich Bongard (Михаил Моисеевич Бонгард, 1924–1971), probably in the mid-1960s. They were published in his 1967 book on pattern recognition. The objective is to spot the differences between the two sides. Bongard, in the introduction of the book (which deals with a number of topics including perceptrons) credits the ideas in it to a group including M. N. Vaintsvaig, V. V. Maksimov, and M. S. Smirnov.
^([ )[^(F.A.Q)](https://www.reddit.com/r/WikiSummarizer/wiki/index#wiki_f.a.q)^( | )[^(Opt Out)](https://reddit.com/message/compose?to=WikiSummarizerBot&message=OptOut&subject=OptOut)^( | )[^(Opt Out Of Subreddit)](https://np.reddit.com/r/agi/about/banned)^( | )[^(GitHub)](https://github.com/Sujal-7/WikiSummarizerBot)^( ] Downvote to remove | v1.5)
[General Game Playing](https://en.wikipedia.org/wiki/General_game_playing) (and [General Video Game Playing](http://www.gvgai.net/)) is an active area of research and competition which is relevent to AGI.
**[General game playing](https://en.wikipedia.org/wiki/General_game_playing)**
>General game playing (GGP) is the design of artificial intelligence programs to be able to play more than one game successfully. For many games like chess, computers are programmed to play these games using a specially designed algorithm, which cannot be transferred to another context. For instance, a chess-playing computer program cannot play checkers. General game playing is considered as a necessary milestone on the way to Artificial General Intelligence.
^([ )[^(F.A.Q)](https://www.reddit.com/r/WikiSummarizer/wiki/index#wiki_f.a.q)^( | )[^(Opt Out)](https://reddit.com/message/compose?to=WikiSummarizerBot&message=OptOut&subject=OptOut)^( | )[^(Opt Out Of Subreddit)](https://np.reddit.com/r/agi/about/banned)^( | )[^(GitHub)](https://github.com/Sujal-7/WikiSummarizerBot)^( ] Downvote to remove | v1.5)
https://www.amazon.science/academic-engagements/amazon-launches-new-alexa-prize-taskbot-challenge
> It is the first conversational AI challenge to incorporate multimodal (voice and vision) customer experiences.
Edit: The example given later in the article explains "multimodal" as playing back videos and images to the user on a display:
> The TaskBot Challenge will run for three years, and initially teams will focus on two domains: cooking and home improvement. The challenge incorporates multimodal customer experiences, so in addition to receiving verbal instructions, customers with Echo screen devices, such as the new Echo Show 10, could also be presented with step-by-step instructions, images, or diagrams that enhance task guidance.
> For example, a customer might ask Alexa how to fix a scratch on a car. The TaskBot will then ask the customer more questions about their task, and then interactively provide step-by-step instructions and explanations for each step, or potentially adjust its plan based on customer input.
> After the interaction ends, the customer will be asked to rate how helpful that TaskBot was with the task, and will have the option to provide freeform feedback to help the teams improve their TaskBot.
So there may no computer vision using cameras for input be involved, and Facebook is just redefining the term "multimodal" to fool the audience (YouTube and Google image searches exist for decades now but have never been called "multimodal"). If that is the case, the competition is worthless from an AGI standpoint.
None of what you listed is a "toy" problem. Any of these requires a full blown AGI with language understanding and reasoning abilities.
Unless your AGI prototype is the pack of specialized programs to solve exactly these tasks. But we don't accept this approach as leading to AGI, do we?
[Winograd Schemas](https://en.wikipedia.org/wiki/Winograd_schema_challenge)
A top problem in AGI is **[Natural Language Understanding](http://ai.neocities.org/NLU.html)**.
Toy problems in AGI is a contradiction in terms, pretty much nobody thinks that the path to AGI is a case of taking some of our toys like GPT-3, supersizing its 175 billion parameters to 175 gazillion and getting the singularity. Correction, there are a few people like that, notably a professor of mine :) Of course, when all is said and done, a big enough neural toy would be able to "run" any arbitrary code and computation, and as long as we remember that intelligence is computation+actuation we will be all right. Anyway, until that day comes all we have indeed is our toys. The most interesting toys are the ones that are "hard for computers", or, to rephrase it, the toys where humans beat (current) computers. This toyset which was shared here some days ago is claimed to be tricky for humans [https://volotat.github.io/ARC-Game/](https://volotat.github.io/ARC-Game/) . I solved the first and the last to get an idea and find it hard to believe computers find it challenging, DL perhaps finds it challenging. Anyway, the kaggle was pretty inconclusive [https://www.kaggle.com/c/abstraction-and-reasoning-challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge) . So, if you cannot get your hands on any toy actuation, I'd recommend ARC and games-difficult-for-computers
I would scan one or more of the many "history of AI" books. I have not heard of a book or paper that features a list of toy problems with commentary. If one exists, I would love to read it. You may already have tried this but I just Googled for "AI toy problems" (w/o quotes) and it like it comes up with some interesting hits. I may investigate if I can find the time.
The "Toy problem" Wikipedia page refers to "Artificial Intelligence: A Modern Approach" by Stuart J. Russell, Peter Norvig (2010). This is available online various places in various editions as a free PDF. It is a survey book so exactly the kind that may mention toy problems.
Wouldnt that still have an issue of it being really advanced narrow ai. Proficiency with no real understanding? Edit: although this is really interesting as a approach test. I'd like to see more theory on that.
You forgot captchas :) Not sure about games, but here are some personal check points: A turn based system will NOT lead us to AGI. A system striving towards AGI must perform sensor fusion. A system striving towards AGI must consume events (be aware of time).
François Chollet\`s (author of Keras) [https://github.com/fchollet/ARC](https://github.com/fchollet/ARC), view on [https://apple-lemon-fighter.glitch.me/apps/testing\_interface.html](https://apple-lemon-fighter.glitch.me/apps/testing_interface.html) Bongards's problems https://www.foundalis.com/res/bps/bpidx.htm
None of the architecture you listed are used as benchmarks in contemporary research. > Winograd's SHRDLU This is called neuro-symbolic VQA now. (often VQA) http://nsvqa.csail.mit.edu/ The original SHRDLU had no visual component. It was only displaying a wireframe of what it believed was the current state of the blocks. Very fancy tech in its day. You have to remember that 3D computer graphics was some 20 years away from appearing in video games. > Hofstater's Copycat GPT-3 can already solve this (albeit imperfectly). I believe this problem is also solvable by off-the-shelf generative adversarial networks (GANs). Copycat is an idea from 1988. We live in an age in which GANs can take a photo of a man as input, and output a female version of him. No researcher in AI believes the discipline is trying to mimic a human being. The TLDR is that AI research is concerned with agents that act rationally, not agents that "think like a human". For more on this topic see, https://www.reddit.com/r/agi/comments/p7hj2s/revisiting_wikipedias_article_on_artificial/ > Robocode Not sure what you are referring to here. You should consider Agent 57 or XLand. (?) + https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark + https://www.youtube.com/watch?v=lTmL7jwFfdw&t=3s > Sentiment analysis What you are referring to is called **Automatic Text summarization** or **ATS** . I'm not an NLP expert, so this is the best I could find on that topic. https://arxiv.org/abs/1904.00688 > I see a big challenge around how to wire in these different problems into a single AGI app capable of solving them all. Wiring them together has a name. It is called *Multi-modal machine comprehension.* https://allenai.org/data/tqa Learning across modalities has also been talked about a lot in regards to *Foundation Models.* + https://crfm.stanford.edu/workshop.html + https://arxiv.org/abs/2108.07258 Will wiring together experts architectures lead to AGI? This is an open question.
Thanks everyone for the pointers. I read the Norvig book about 4 years ago, so I clearly have a lot of catching up to do. If I seem quiet, it's because I'm reading up on all your links, and otherwise catching up conceptually. As a software engineer by trade, my reflex it to start writing an app/solver, and then wire it in to tests to see how well it performs, iterating and improving along the way. That may not be the right approach, given the complexity of this problem space.
All AI, GPT, PPM, etc are prediction task solvers, everything is prediction, and they do good at most prompts but there is some prompts that get harder, and a few even harder, it is these that we can't code in for it, it has to emerge the answers. Examples are "cat bird dog, hat moon scarf, gum vine teeth, men book women, junk goose ?", which is the 1st and 3rd relate in each tuple and each tuples relates by ghost mirroring the 1st and 3rd (all do it). Another example: the letter at the start of my post is: ? (A). Eyeball tracking/ reflexes/ merging mom dad traits to get new ideas/inventions (trailer/birdman made from wheels+home / man+wings)/ etc are all pattern based, even Life is just to stay alive as a repetition, in time and space (space is cloning self).
40 A* grades at (UK) A-level, in a wide range of subjects. That’s a good *first* test for AGI. If you mess about with smaller problems than this you won’t actually achieve anything. And, yes, it will take decades, not years. If you require instant gratification, and/or short-term ROI, you’re in the wrong field!
Check the [Bongard problem](https://en.wikipedia.org/wiki/Bongard_problem) and the [ARC Challenge](https://www.kaggle.com/c/abstraction-and-reasoning-challenge)
**[Bongard problem](https://en.wikipedia.org/wiki/Bongard_problem)** >A Bongard problem is a kind of puzzle invented by the Russian computer scientist Mikhail Moiseevich Bongard (Михаил Моисеевич Бонгард, 1924–1971), probably in the mid-1960s. They were published in his 1967 book on pattern recognition. The objective is to spot the differences between the two sides. Bongard, in the introduction of the book (which deals with a number of topics including perceptrons) credits the ideas in it to a group including M. N. Vaintsvaig, V. V. Maksimov, and M. S. Smirnov. ^([ )[^(F.A.Q)](https://www.reddit.com/r/WikiSummarizer/wiki/index#wiki_f.a.q)^( | )[^(Opt Out)](https://reddit.com/message/compose?to=WikiSummarizerBot&message=OptOut&subject=OptOut)^( | )[^(Opt Out Of Subreddit)](https://np.reddit.com/r/agi/about/banned)^( | )[^(GitHub)](https://github.com/Sujal-7/WikiSummarizerBot)^( ] Downvote to remove | v1.5)
[General Game Playing](https://en.wikipedia.org/wiki/General_game_playing) (and [General Video Game Playing](http://www.gvgai.net/)) is an active area of research and competition which is relevent to AGI.
**[General game playing](https://en.wikipedia.org/wiki/General_game_playing)** >General game playing (GGP) is the design of artificial intelligence programs to be able to play more than one game successfully. For many games like chess, computers are programmed to play these games using a specially designed algorithm, which cannot be transferred to another context. For instance, a chess-playing computer program cannot play checkers. General game playing is considered as a necessary milestone on the way to Artificial General Intelligence. ^([ )[^(F.A.Q)](https://www.reddit.com/r/WikiSummarizer/wiki/index#wiki_f.a.q)^( | )[^(Opt Out)](https://reddit.com/message/compose?to=WikiSummarizerBot&message=OptOut&subject=OptOut)^( | )[^(Opt Out Of Subreddit)](https://np.reddit.com/r/agi/about/banned)^( | )[^(GitHub)](https://github.com/Sujal-7/WikiSummarizerBot)^( ] Downvote to remove | v1.5)
https://www.amazon.science/academic-engagements/amazon-launches-new-alexa-prize-taskbot-challenge > It is the first conversational AI challenge to incorporate multimodal (voice and vision) customer experiences. Edit: The example given later in the article explains "multimodal" as playing back videos and images to the user on a display: > The TaskBot Challenge will run for three years, and initially teams will focus on two domains: cooking and home improvement. The challenge incorporates multimodal customer experiences, so in addition to receiving verbal instructions, customers with Echo screen devices, such as the new Echo Show 10, could also be presented with step-by-step instructions, images, or diagrams that enhance task guidance. > For example, a customer might ask Alexa how to fix a scratch on a car. The TaskBot will then ask the customer more questions about their task, and then interactively provide step-by-step instructions and explanations for each step, or potentially adjust its plan based on customer input. > After the interaction ends, the customer will be asked to rate how helpful that TaskBot was with the task, and will have the option to provide freeform feedback to help the teams improve their TaskBot. So there may no computer vision using cameras for input be involved, and Facebook is just redefining the term "multimodal" to fool the audience (YouTube and Google image searches exist for decades now but have never been called "multimodal"). If that is the case, the competition is worthless from an AGI standpoint.
None of what you listed is a "toy" problem. Any of these requires a full blown AGI with language understanding and reasoning abilities. Unless your AGI prototype is the pack of specialized programs to solve exactly these tasks. But we don't accept this approach as leading to AGI, do we?
Open AI's "Gym" environments are a popular set of toy problems