Superforecasting

Tetlock reports the results of the Good Judgment Project, which competed in a four-year forecasting tournament run by IARPA against other university teams and against intelligence community analysts with access to classified material. Thousands of volunteers made specific probabilistic forecasts on geopolitical and economic questions with defined resolution criteria, scored by Brier score.

The finding is that a small group, roughly the top two percent, consistently outperformed, and that their performance persisted year to year, which rules out luck as the explanation. Tetlock calls them superforecasters and describes what they do: they break questions into components, start from base rates before considering case detail, gather evidence from multiple perspectives, update in small frequent increments as information arrives, use finely grained probability estimates rather than rounding to tens, and treat their own beliefs as hypotheses rather than as positions.

The book builds on Tetlock's earlier Expert Political Judgment, which found that credentialed experts forecast barely better than chance and that fame correlated negatively with accuracy. He uses Isaiah Berlin's hedgehog and fox distinction, with foxes who hold many partial models outperforming hedgehogs who apply one framework.

The practical content includes the finding that training in probabilistic reasoning improved accuracy by around ten percent, that teams outperformed individuals when structured to avoid conformity, and that most public forecasting is unfalsifiable by design because it lacks time frames and probability estimates. The limits are real. The tournament asked questions resolvable within months on geopolitical topics, and the transfer to longer horizons or to genuinely unprecedented events is unproven. Taleb has argued that the method works only where the underlying distribution is not fat-tailed.

Canadian journalist who co-wrote Superforecasting with Philip Tetlock and How Big Things Get Done with Bent Flyvbjerg.

Superforecasting reports the Good Judgment Project tournament results and the habits that separated accurate forecasters: breaking questions into parts, updating in small increments, keeping score in numbers. How Big Things Get Done adds base-rate evidence from thousands of megaprojects on why budgets and schedules fail and what the successful ones did differently.