The AI Does Not Hate You

Tom Chivers, a British science journalist, reports on the rationalist community and its concern with artificial intelligence risk. His subjects are the people around LessWrong, the Machine Intelligence Research Institute, Effective Altruism and the Future of Humanity Institute, and he interviews many of them.

The book does two things. It explains the argument for AI risk as its proponents make it: that an artificial general intelligence pursuing any objective would develop instrumental goals including self-preservation and resource acquisition, that specifying human values precisely enough to constrain such a system is unsolved, and that the transition from human-level to substantially superhuman capability could be fast enough that correction after the fact is impossible. Chivers presents this carefully and does not caricature it.

It also reports on the community as a social phenomenon: the writing of Eliezer Yudkowsky, the Sequences, the emphasis on Bayesian reasoning and on calibration, the group houses, the polyamory, the internal disputes, and the ways in which the community resembles and does not resemble a religious movement. Chivers is sympathetic without being a member, and he is honest that he found the risk argument more persuasive than he expected. The book was published in 2019, before large language models made these questions general rather than niche, and it is now a useful record of what the argument looked like before that.