Superforecasting
The Good Judgment Project; the practices that measurably improve forecasts.
Summary
Tetlock reports the results of the Good Judgment Project, which competed in a four-year forecasting tournament run by IARPA against other university teams and against intelligence community analysts with access to classified material. Thousands of volunteers made specific probabilistic forecasts on geopolitical and economic questions with defined resolution criteria, scored by Brier score.
The finding is that a small group, roughly the top two percent, consistently outperformed, and that their performance persisted year to year, which rules out luck as the explanation. Tetlock calls them superforecasters and describes what they do: they break questions into components, start from base rates before considering case detail, gather evidence from multiple perspectives, update in small frequent increments as information arrives, use finely grained probability estimates rather than rounding to tens, and treat their own beliefs as hypotheses rather than as positions.
The book builds on Tetlock's earlier Expert Political Judgment, which found that credentialed experts forecast barely better than chance and that fame correlated negatively with accuracy. He uses Isaiah Berlin's hedgehog and fox distinction, with foxes who hold many partial models outperforming hedgehogs who apply one framework.
The practical content includes the finding that training in probabilistic reasoning improved accuracy by around ten percent, that teams outperformed individuals when structured to avoid conformity, and that most public forecasting is unfalsifiable by design because it lacks time frames and probability estimates. The limits are real. The tournament asked questions resolvable within months on geopolitical topics, and the transfer to longer horizons or to genuinely unprecedented events is unproven. Taleb has argued that the method works only where the underlying distribution is not fat-tailed.
Key takeaways 8
- A small minority of forecasters consistently outperform, and their advantage persists across years, so it reflects method rather than luck.
- Forecasting skill comes from technique and disposition rather than from subject matter expertise or access to secret information.
- Accurate forecasters start from the base rate of the reference class before examining the specifics of the case.
- Frequent small updates in response to new evidence outperform both stubbornness and dramatic revision.
- Probability estimates should be finely grained, and forecasters who distinguish between sixty and sixty-five percent score better than those who round.
- Most public forecasts cannot be scored because they lack a time frame and a probability, which protects the forecaster from any accountability.
- Brief training in probabilistic reasoning produced measurable accuracy improvements, so the skill is teachable.
- Belief in one's own judgment must be treated as a hypothesis to be tested, and the best forecasters actively looked for reasons they were wrong.
Who should read it
Investors and executives who make explicit predictions and want a scoring discipline and a set of practices that measurably improve accuracy.
About Dan Gardner
Canadian journalist who co-wrote Superforecasting with Philip Tetlock and How Big Things Get Done with Bent Flyvbjerg.
Superforecasting reports the Good Judgment Project tournament results and the habits that separated accurate forecasters: breaking questions into parts, updating in small increments, keeping score in numbers. How Big Things Get Done adds base-rate evidence from thousands of megaprojects on why budgets and schedules fail and what the successful ones did differently.
Where it sits in the reference 7
Who recommended it 9 · 1 direct
Similar books 6
Chosen by shared topics, the entries that hold them, and the people who recommended both.
