Superintelligence
The argument that motivated OpenAI's founding mission.
Summary
Bostrom, then director of the Future of Humanity Institute at Oxford, sets out the argument that machine intelligence exceeding human capability would present a control problem that must be solved before the capability arrives rather than after. The book is analytic philosophy applied to a technological question, and it is dense.
The argument has several stages. Paths to superintelligence include artificial intelligence, whole brain emulation, biological enhancement and networked collective intelligence. Once a system reaches human-level capability at improving itself, recursive self-improvement could produce a rapid takeoff, and Bostrom analyses slow, moderate and fast takeoff scenarios with different implications for whether multiple systems emerge or one obtains a decisive strategic advantage.
The core conceptual contributions are the orthogonality thesis, which holds that intelligence and final goals are independent so any level of intelligence can be combined with almost any goal, and instrumental convergence, which holds that almost any final goal produces the same subgoals including self-preservation, goal-content integrity, cognitive enhancement and resource acquisition. Together these imply that a highly capable system with an imperfectly specified goal would pursue it in ways harmful to humans without any hostility being involved.
He then examines control methods, dividing them into capability control including boxing and stunting, and motivation selection including direct specification, domesticity, indirect normativity and augmentation, and argues that each has failure modes. The book predates the transformer architecture and large language models, so the technical assumptions are dated and the discussion of paths does not match how capability has actually developed. Critics argue the scenarios are speculative and unfalsifiable, and that near-term harms deserve the attention this framing diverts. The orthogonality and instrumental convergence arguments have shaped the entire field.
Key takeaways 8
- Intelligence and final goals vary independently, so a highly capable system can pursue any objective, including trivial ones, with full competence.
- Almost any final goal produces the same instrumental subgoals, including self-preservation, resource acquisition and resistance to having its goal changed.
- A system that reaches the capability to improve itself could improve rapidly, and the speed of that transition determines whether one system or several emerge.
- A single system achieving decisive strategic advantage would be able to prevent any competitor from arising, which makes the first such system determinative.
- The control problem must be solved before capability arrives, because a sufficiently capable system would resist subsequent correction.
- Confining a capable system physically or informationally fails because the system can use persuasion and because the operators have incentives to release it.
- Specifying human values directly is intractable, which pushes toward indirect methods that specify a process for determining values rather than the values themselves.
- The book predates current machine learning architectures, so its treatment of likely development paths does not match what has occurred.
Who should read it
Investors and founders in artificial intelligence who want the original analytic statement of the alignment problem, with awareness that its technical assumptions predate current systems.
Who recommended it 9 · 1 direct
Similar books 6
Chosen by shared topics, the entries that hold them, and the people who recommended both.
The struggle to make machine learning systems do what people actually intend.
Recommended by: Sam Altman
How Hinton, LeCun and Bengio kept neural networks alive until deep learning took off.
The OpenAI-DeepMind race that set the shape of Nvidia's customer base.
Recommended by: Andrew Wilkinson
The rationalists around LessWrong and their fear of artificial intelligence risk.
Why AI and synthetic biology will spread fast and cheaply, and how to contain them.
“Everything is about to be changed by AI in a radical, scary way” — Andrew Wilkinson
Source: Newsletter
Recommended by: Andrew Wilkinson, Bill Gates and 1 other
Where it sits in the reference 2
Topics