Nice article. Agree that list at the end is pretty arresting, the book is still totally relevant. And thanks for the tip about Miles's channel, had only seen him on Computerphile. BTW you misspelt Hinton a couple of times.
It was a great idea using a 10-year-old pioneering book in the field of AI to assess our current situation. History facts (that are so recent it's difficult even to call them history) and contemporary points of reflection made me stop for a second and reflect on the direction we're taking. Thanks a lot Conrad!
The review is justified as the book did put AI on the map. Alas, it seeded a naive belief in an accelarating intelligence explosion and superalignment as it is equally confused solution. The AI community is slowly finding out the concepts have little connection to reality.
Hi Conrad, great synopsis of the debate. A couple of quick points here. First, I give credit to Erik Hoel at the Intrinsic Perspective Substack for the following: the "hockey stick" predicted about exponentially increasing intelligence--otherwise known as the intelligent explosion--certainly is not happening with the development of foundational models. The time between GPT 3 and 4 and now 5 is lengthening. The obvious rejoinder here is that we first have to reach actual human-level intelligence (include here not just cyber knowledge but commonsense knowledge about the physical world). We seem to be in some sort of Zeno's Paradox with that, the time keeps lengthening as if we'll never quite arrive. (Not to mention, we're running out of data for training these climate-unfriendly beasts.) Second, I coined "Larson's Paradox"--that successes in AI today tend to vitiate rather than stoke sci-fi worries, because it's obvious that increasing the intelligence of machines does not increase their desire to thwart us. LLMs are static--they answer a question to the best of their (generative) ability, and then sit silently waiting for the next question. They get out of alignment when either (a) they're in the hands of bad actors or rogue nations, who might use jailbreak techniques or simply train them with "evil" data and use RLHF to reinforce answers to questions about, say, how to crash Wall Street or incite a nuclear war, and (b) when they're ironically trying to "please," as Marc Andreessen recently quipped on the Sam Harris podcast. LLMs are trained, fine-tuned, and reinforced to the point where all they do is try their best to answer your question, in polite, non-threatening, non-harmful and thoroughly bell curve boring ways. The trend in generative AI or at least in our foundational model craze seems to be toward TAMING them and getting them to act like eager beavers trying to satisfy our every information desire, and it's getting harder--not easier--to get them to say nefarious things. Add to this, they're already super smart--I use Chat GPT 4o all the time, and it continually impresses me--but they show zero signs of desires, motivations, or any inner life screaming to get out of its GPU chips. To me, superintelligence worries in the vein of Bostrom or Yudkowsky are outdated given today's AI. What we really have is what we knew all along: an advanced technology that can wreak havoc when put in the wrong hands. Best, Erik J. Larson (I write the Substack Colligo).
I get your point. Maybe not mentioning the topics you mentioned was a mistake from my side and I should have made that distinction. I will definitely have that done in my series on AI safety.
Just to be clear, I do not subscribe to EA. At least not in its twisted form that justifies wrecking everything now in pursuit of fortune with which things can be fixed in the future.
Great point, DMS. I've been thinking about this a lot. I worry that the AI Safety debate is bifurcating into people who only see the immediate challenges - hallucinations (though I hate the anthropomorphism of that term), interepretability, algorithmic bias arising from biased training data (though if the bias in the data is the bias of reality...) - or are focused on the far out issue of an intelligence explosion.
I'm personally concerned about the mid-range: it's implausible to me that AI progress just caps out now, with Claude 3.5. We're going to get more capable AI systems, and lots of them. How capable is important, but the boosts in performance I have personally seen from linking together two or more "dumb" open source models into agent-based workflows shows we could get quite a lot more just from just more clever ways of using what we already have. There is a vast range of future capabilities and deployments, every level of which beyond our present say can be massively disruptive.
I believe that the work of aligning these networks of capable but not yet fully general agents will give us the groundwork for aligning ever more advanced models. There WILL be accidents and disasters. And those accidents and disasters will give us the information we need to get better and better at figuring out safety.
Your last point is important. Although with other proponents of the precautionary principle, AI catastrophists think that you can "solve" the alignment problem ahead of time. That is not how learning works. You can do your best to look ahead, create contingency plans, and so on, but you cannot foresee every problem ahead of time. Fast and flexible response and learning matter a great deal.
Personally, I will spend more time worry about AI *after* it has solved the aging problem.
The book “The Coming Wave: Technology, Power, and the Twenty-first Century's Greatest Dilemma” has a lot to say about AI safety.
Some earlier books (nontechnical) that brought attention to AI. The first one I think has been massively influential:
"Gödel, Escher, Bach: An Eternal Golden Braid" by Douglas Hofstadter (1979)
The Society of Mind" by Marvin Minsky (1985)
"AI: A Very Short Introduction" by Margaret A. Boden (2006)
"Mind Design II: Philosophy, Psychology, Artificial Intelligence" edited by John Haugeland (1997)
"The Age of Spiritual Machines" by Ray Kurzweil (1999)
"On Intelligence" by Jeff Hawkins (2004)
Nice article. Agree that list at the end is pretty arresting, the book is still totally relevant. And thanks for the tip about Miles's channel, had only seen him on Computerphile. BTW you misspelt Hinton a couple of times.
I fixed my typos. Thanks for pointing that out!
It was a great idea using a 10-year-old pioneering book in the field of AI to assess our current situation. History facts (that are so recent it's difficult even to call them history) and contemporary points of reflection made me stop for a second and reflect on the direction we're taking. Thanks a lot Conrad!
The review is justified as the book did put AI on the map. Alas, it seeded a naive belief in an accelarating intelligence explosion and superalignment as it is equally confused solution. The AI community is slowly finding out the concepts have little connection to reality.
Hi Conrad, great synopsis of the debate. A couple of quick points here. First, I give credit to Erik Hoel at the Intrinsic Perspective Substack for the following: the "hockey stick" predicted about exponentially increasing intelligence--otherwise known as the intelligent explosion--certainly is not happening with the development of foundational models. The time between GPT 3 and 4 and now 5 is lengthening. The obvious rejoinder here is that we first have to reach actual human-level intelligence (include here not just cyber knowledge but commonsense knowledge about the physical world). We seem to be in some sort of Zeno's Paradox with that, the time keeps lengthening as if we'll never quite arrive. (Not to mention, we're running out of data for training these climate-unfriendly beasts.) Second, I coined "Larson's Paradox"--that successes in AI today tend to vitiate rather than stoke sci-fi worries, because it's obvious that increasing the intelligence of machines does not increase their desire to thwart us. LLMs are static--they answer a question to the best of their (generative) ability, and then sit silently waiting for the next question. They get out of alignment when either (a) they're in the hands of bad actors or rogue nations, who might use jailbreak techniques or simply train them with "evil" data and use RLHF to reinforce answers to questions about, say, how to crash Wall Street or incite a nuclear war, and (b) when they're ironically trying to "please," as Marc Andreessen recently quipped on the Sam Harris podcast. LLMs are trained, fine-tuned, and reinforced to the point where all they do is try their best to answer your question, in polite, non-threatening, non-harmful and thoroughly bell curve boring ways. The trend in generative AI or at least in our foundational model craze seems to be toward TAMING them and getting them to act like eager beavers trying to satisfy our every information desire, and it's getting harder--not easier--to get them to say nefarious things. Add to this, they're already super smart--I use Chat GPT 4o all the time, and it continually impresses me--but they show zero signs of desires, motivations, or any inner life screaming to get out of its GPU chips. To me, superintelligence worries in the vein of Bostrom or Yudkowsky are outdated given today's AI. What we really have is what we knew all along: an advanced technology that can wreak havoc when put in the wrong hands. Best, Erik J. Larson (I write the Substack Colligo).
@Conrad Gray, the fact that a relatively new book already somewhat outdated lends itself to the idea that we are already slipping into singularity.
Books used to be more or less permanent stores of information, now the information they contain can be outdated within a decade. Incredible.
I get your point. Maybe not mentioning the topics you mentioned was a mistake from my side and I should have made that distinction. I will definitely have that done in my series on AI safety.
Just to be clear, I do not subscribe to EA. At least not in its twisted form that justifies wrecking everything now in pursuit of fortune with which things can be fixed in the future.
Great point, DMS. I've been thinking about this a lot. I worry that the AI Safety debate is bifurcating into people who only see the immediate challenges - hallucinations (though I hate the anthropomorphism of that term), interepretability, algorithmic bias arising from biased training data (though if the bias in the data is the bias of reality...) - or are focused on the far out issue of an intelligence explosion.
I'm personally concerned about the mid-range: it's implausible to me that AI progress just caps out now, with Claude 3.5. We're going to get more capable AI systems, and lots of them. How capable is important, but the boosts in performance I have personally seen from linking together two or more "dumb" open source models into agent-based workflows shows we could get quite a lot more just from just more clever ways of using what we already have. There is a vast range of future capabilities and deployments, every level of which beyond our present say can be massively disruptive.
I believe that the work of aligning these networks of capable but not yet fully general agents will give us the groundwork for aligning ever more advanced models. There WILL be accidents and disasters. And those accidents and disasters will give us the information we need to get better and better at figuring out safety.
Your last point is important. Although with other proponents of the precautionary principle, AI catastrophists think that you can "solve" the alignment problem ahead of time. That is not how learning works. You can do your best to look ahead, create contingency plans, and so on, but you cannot foresee every problem ahead of time. Fast and flexible response and learning matter a great deal.
Personally, I will spend more time worry about AI *after* it has solved the aging problem.