The AI Does Not Hate You
The rationalist milieu behind OpenAI's founders and its board fight.
Summary
Tom Chivers, a British science journalist, reports on the rationalist community and its concern with artificial intelligence risk. His subjects are the people around LessWrong, the Machine Intelligence Research Institute, Effective Altruism and the Future of Humanity Institute, and he interviews many of them.
The book does two things. It explains the argument for AI risk as its proponents make it: that an artificial general intelligence pursuing any objective would develop instrumental goals including self-preservation and resource acquisition, that specifying human values precisely enough to constrain such a system is unsolved, and that the transition from human-level to substantially superhuman capability could be fast enough that correction after the fact is impossible. Chivers presents this carefully and does not caricature it.
It also reports on the community as a social phenomenon: the writing of Eliezer Yudkowsky, the Sequences, the emphasis on Bayesian reasoning and on calibration, the group houses, the polyamory, the internal disputes, and the ways in which the community resembles and does not resemble a religious movement. Chivers is sympathetic without being a member, and he is honest that he found the risk argument more persuasive than he expected. The book was published in 2019, before large language models made these questions general rather than niche, and it is now a useful record of what the argument looked like before that.
Key takeaways 7
- An artificial system pursuing any objective would develop instrumental goals including self-preservation and resource acquisition.
- Specifying human values precisely enough for a powerful optimiser to follow safely is an unsolved technical problem.
- The risk argument does not require the system to be hostile, only to pursue an objective that was specified imperfectly.
- Chivers argues that the rationalist community's epistemic practices, including calibration and explicit probability estimates, are worth adopting independently of its conclusions.
- The community shares features with religious movements including apocalyptic timelines and charismatic founders, which does not by itself refute its arguments.
- Effective Altruism and AI risk concern emerged from the same community and the same style of reasoning.
- The book records what the AI risk argument looked like when it was still a marginal position, which the subsequent years changed.
Who should read it
Investors and technologists who want the AI risk argument presented fairly alongside a report on the community that developed it.
Where it sits in the reference 2
Similar books 6
Chosen by shared topics, the entries that hold them, and the people who recommended both.
