
What a four-year forecasting tournament showed about the best forecasters' habits.
The Good Judgment Project; the practices that measurably improve forecasts.
Tetlock reports the results of the Good Judgment Project, which competed in a four-year forecasting tournament run by IARPA against other university teams and against intelligence community analysts with access to classified material. Thousands of volunteers made specific probabilistic forecasts on geopolitical and economic questions with defined resolution criteria, scored by Brier score.
The finding is that a small group, roughly the top two percent, consistently outperformed, and that their performance persisted year to year, which rules out luck as the explanation. Tetlock calls them superforecasters and describes what they do: they break questions into components, start from base rates before considering case detail, gather evidence from multiple perspectives, update in small frequent increments as information arrives, use finely grained probability estimates rather than rounding to tens, and treat their own beliefs as hypotheses rather than as positions.
The book builds on Tetlock's earlier Expert Political Judgment, which found that credentialed experts forecast barely better than chance and that fame correlated negatively with accuracy. He uses Isaiah Berlin's hedgehog and fox distinction, with foxes who hold many partial models outperforming hedgehogs who apply one framework.
The practical content includes the finding that training in probabilistic reasoning improved accuracy by around ten percent, that teams outperformed individuals when structured to avoid conformity, and that most public forecasting is unfalsifiable by design because it lacks time frames and probability estimates. The limits are real. The tournament asked questions resolvable within months on geopolitical topics, and the transfer to longer horizons or to genuinely unprecedented events is unproven. Taleb has argued that the method works only where the underlying distribution is not fat-tailed.
Investors and executives who make explicit predictions and want a scoring discipline and a set of practices that measurably improve accuracy.
Canadian journalist who co-wrote Superforecasting with Philip Tetlock and How Big Things Get Done with Bent Flyvbjerg.
Superforecasting reports the Good Judgment Project tournament results and the habits that separated accurate forecasters: breaking questions into parts, updating in small increments, keeping score in numbers. How Big Things Get Done adds base-rate evidence from thousands of megaprojects on why budgets and schedules fail and what the successful ones did differently.
“In a book many in our research department have enjoyed, Superforecasting: The Art and Science of Prediction, Philip Tetlock and Dan Gardner state: “The humility required for good judgment is not self-doubt—the sense that you are untalented, unintelligent, or unworthy.”
“Long-term studies suggest that the underlying predictions or assumptions for visions are nearly always impractically vague or outright wrong (see Tetlock’s “Superforecasting”).”
“Tetlock and Dan Gardner, his co-author of the book Superforecasting, write, “Beliefs are hypotheses to be tested, not treasures to be protected.”
Chosen by shared topics, the entries that hold them, and the people who recommended both.
A 20-year test of 28,000 expert forecasts: foxes with many models beat hedgehogs.
“All pundits and commentators should publish their prediction track records, yet don't. What to pay attention to and what to ignore.” — Marc Andreessen
Source: @pmarca, 'Thread: Books I've recently read and recommend'
Recommended by: Marc Andreessen, Michael Mauboussin and 2 others
How fast, automatic thought and slow, deliberate thought shape judgment and error.
“…books I've recommended, highlighted to death, or forced on friends.” — Andrew Wilkinson
Source: Andrew Wilkinson's Bookshelf, neverenough.com
Recommended by: Andrew Wilkinson, Charles de Vaulx and 36 others
Why people mistake luck for skill, above all in markets.
“Fooled By Randomness was unquestionably the best non-fiction book I read in 2005.” — Brad Feld
Source: Book Review: Fooled By Randomness — Feld Thoughts
Recommended by: Barton Biggs, Brad Feld, Derek Sivers and 15 others
Rare, unforeseen, high-impact events drive history, explained only after the fact.
“The Black Swan U The best-known bird around today is The Black Swan, the second book from Nassim Nicolas Taleb.” — Howard Marks
Source: The Complete Collection, page 530
Recommended by: David Perell, Howard Marks, Marc Andreessen and 20 others
Why some forecasts succeed and most fail, from weather to earthquakes and economics.
“See Gigerenzer, Calculated Risks, 28-29. 14 For a book that treats this topic and more, see Annie Duke, Thinking in Bets: Making Smarter Decisions When You Don't Have All the Facts (New York: Portfolio/Penguin, 2018). 15 For a good discussion of Bayes’ Theorem, see Nate Silver, The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (New York, The Penguin Press, 2012), 232-261.” — Michael Mauboussin
Source: Confidence: Methods to Assess Confidence Under Uncertainty, page 7
Recommended by: Bill Gates, Michael Mauboussin and 2 others
Unwanted variation in judgements that should match, and why organisations ignore it.
“Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgment (New York: Little, Brown Spark, 2021), 156-158” — Michael Mauboussin
Source: The Impact of Intangibles on Base Rates, page 10
Recommended by: Michael Mauboussin
As an Amazon Associate I earn from qualifying purchases.