Noise: A Flaw in Human Judgment
Kahneman died 27 March 2024; this was his last book.
Summary
Kahneman, Sibony and Sunstein argue that organisations concentrate on bias, meaning systematic error in one direction, and ignore noise, meaning unwanted variability in judgements that should be identical. They show noise is large, expensive and almost entirely unmeasured.
The evidence comes from noise audits. In insurance underwriting, different underwriters priced the same risk with differences averaging around fifty-five percent, far larger than executives predicted. Judges gave different sentences for identical cases, and the same judge gave different sentences depending on time of day, weather and whether the local football team had won. Medical diagnoses, hiring decisions, forensic examinations and forecasts show similar variability.
They decompose noise into level noise, meaning some judges are systematically harsher; pattern noise, meaning individuals respond differently to particular case features; and occasion noise, meaning the same person judges differently on different occasions. Pattern noise is generally the largest component and the least recognised.
The remedies they propose are collectively called decision hygiene: use structured decision protocols that assess independent dimensions separately before aggregating, use relative rather than absolute judgements, aggregate independent judgements from several people before discussion, use simple algorithms or guidelines where available since they eliminate occasion noise entirely, and appoint a decision observer to watch the process.
The book is long and repetitive, and much of it could be stated in half the length. The core empirical finding is well supported and the practical implications for any organisation making repeated judgements are direct.
Key takeaways 7
- Unwanted variability in judgements that should be identical is large, expensive and almost never measured in organisations.
- Insurance underwriters pricing the same risk differed by around fifty-five percent on average, far more than their executives predicted.
- Pattern noise, meaning individuals responding differently to particular case features, is usually the largest component and the least recognised.
- Occasion noise means the same person judges the same case differently depending on mood, time of day and irrelevant recent events.
- Simple algorithms and guidelines outperform human judgement largely because they eliminate noise entirely rather than because they capture more information.
- Judgements should be broken into independent dimensions assessed separately and aggregated afterwards, since holistic judgement is where noise enters.
- Independent judgements aggregated before any discussion outperform group discussion, because discussion causes the group to converge on the first or loudest view.
Who should read it
Executives and investors running any process that makes repeated similar judgements, including hiring, underwriting, credit and investment decisions.
About Daniel Kahneman
Eugene Higgins Professor of Psychology Emeritus, Princeton University; Nobel laureate in economics 2002
Prospect theory and the heuristics-and-biases programme are the primary psychological foundation the whole of behavioral finance builds on.
Similar books 6
Chosen by shared topics, the entries that hold them, and the people who recommended both.
Papers on how people choose under risk, from prospect theory to framing.
A 20-year test of 28,000 expert forecasts: foxes with many models beat hedgehogs.
“All pundits and commentators should publish their prediction track records, yet don't. What to pay attention to and what to ignore.” — Marc Andreessen
Source: @pmarca, 'Thread: Books I've recently read and recommend'
Recommended by: Marc Andreessen, Michael Mauboussin and 2 others
Why people mistake luck for skill, above all in markets.
“Fooled by Randomness is a serious, intellectually sophisticated book, well worth reading carefully. At times, the book is condescending. Nevertheless, there are some great insights.” — Barton Biggs
Source: Biggs on a sophisticated book worth reading carefully
Recommended by: Barton Biggs, Michael Mauboussin and 11 others
Edited volume.
The Good Judgment Project; the practices that measurably improve forecasts.
Recommended by: Patrick Collison and 9 others
Where it sits in the reference 4
Topics