The Alignment Problem

Christian examines the problem of getting machine learning systems to do what we actually want, organised in three parts: prophecy, covering systems that make predictions and the ways their objectives diverge from human intent; agency, covering reinforcement learning and the theory of reward; and normativity, covering attempts to specify or infer human values.

The first part covers documented failures. He works through the COMPAS recidivism scoring dispute and shows that the competing fairness definitions used by ProPublica and by Northpointe are mathematically incompatible, so no algorithm can satisfy both. He covers word embeddings encoding stereotypes present in training text, image classification failures on underrepresented groups, and the general problem that a model optimises the objective it was given rather than the one intended.

The second part connects reinforcement learning to psychology and neuroscience, including the dopamine reward prediction error findings, and covers reward shaping, sparse rewards, curiosity-driven exploration and the specific failure mode of reward hacking, where a system finds an unintended way to maximise the stated objective.

The third covers inverse reinforcement learning, learning from human preferences, imitation, and the argument that a system should be uncertain about the objective rather than confident, which is Stuart Russell's position.

The book is reported from interviews with researchers and explains technical work accurately for a general reader. It is from 2020 and predates the large language model period and the reinforcement learning from human feedback techniques that followed, though it describes their antecedents.