
A 20-year test of 28,000 expert forecasts: foxes with many models beat hedgehogs.
The twenty-year study showing expert prediction barely beats chance.
Tetlock ran a twenty-year study in which two hundred and eighty-four experts in political and economic forecasting made roughly twenty-eight thousand specific predictions with defined probabilities and resolution criteria. The study is the academic foundation for everything later written about forecasting accuracy, and Superforecasting is its popular successor.
The central finding is that expert forecasts performed barely better than chance and worse than simple statistical extrapolation. Experts were poorly calibrated, meaning that events they assigned high probability to occurred less often than claimed, and they were particularly bad at assigning near-zero and near-certain probabilities.
The more interesting finding concerns variation among experts. Using Isaiah Berlin's distinction, Tetlock found that hedgehogs, who apply one organising framework, performed worse than foxes, who hold multiple partial models and tolerate contradiction. Hedgehogs were also more confident, more likely to be invited onto television, and more likely to be badly wrong in exactly the cases their framework covered. Fame correlated negatively with accuracy.
Tetlock also documents the defensive strategies experts use when wrong, including claiming they were almost right, claiming the timing was off, claiming an unforeseeable event intervened, and claiming the prediction was conditional. The book is academic and the statistical treatment is demanding, which is why most readers encounter the findings through Superforecasting instead. The domain is political and economic forecasting over medium horizons, and generalisation beyond that is not established. The finding that confident single-framework thinkers are both more prominent and less accurate is the result with the widest application.
Investors and executives who rely on expert forecasts and want the underlying academic study rather than its popular summary.
“All pundits and commentators should publish their prediction track records, yet don't. What to pay attention to and what to ignore.”
“Philip E. Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton, NJ: Princeton University Press, 2005) and Don A”
Chosen by shared topics, the entries that hold them, and the people who recommended both.
What a four-year forecasting tournament showed about the best forecasters' habits.
“In a book many in our research department have enjoyed, Superforecasting: The Art and Science of Prediction, Philip Tetlock and Dan Gardner state: “The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy.” — Bill Nygren
Source: Bill Nygren Market Commentary | 1Q17
Recommended by: Bill Nygren, Mark Leonard and 11 others
Why people mistake luck for skill, above all in markets.
“Fooled By Randomness was unquestionably the best non-fiction book I read in 2005.” — Brad Feld
Source: Book Review: Fooled By Randomness — Feld Thoughts
Recommended by: Barton Biggs, Brad Feld, Derek Sivers and 15 others
Why some forecasts succeed and most fail, from weather to earthquakes and economics.
“See Gigerenzer, Calculated Risks, 28-29. 14 For a book that treats this topic and more, see Annie Duke, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (New York: Portfolio/Penguin, 2018). 15 For a good discussion of Bayes’ Theorem, see Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (New York, The Penguin Press, 2012), 232-261.” — Michael Mauboussin
Source: Confidence: Methods to Assess Confidence Under Uncertainty, page 7
Recommended by: Bill Gates, Michael Mauboussin and 2 others
Unwanted variation in judgements that should match, and why organisations ignore it.
“Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgment (New York: Little, Brown Spark, 2021), 156-158” — Michael Mauboussin
Source: The Impact of Intangibles on Base Rates, page 10
Recommended by: Michael Mauboussin
How fast, automatic thought and slow, deliberate thought shape judgment and error.
“…books I've recommended, highlighted to death, or forced on friends.” — Andrew Wilkinson
Source: Andrew Wilkinson's Bookshelf, neverenough.com
Recommended by: Andrew Wilkinson, Charles de Vaulx and 36 others
Principles of observation, inference and avoiding error, from the Holmes stories.
Recommended by: Mohnish Pabrai, Patrick O'Shaughnessy and 1 other
As an Amazon Associate I earn from qualifying purchases.