Expert Political Judgment

Tetlock ran a twenty-year study in which two hundred and eighty-four experts in political and economic forecasting made roughly twenty-eight thousand specific predictions with defined probabilities and resolution criteria. The study is the academic foundation for everything later written about forecasting accuracy, and Superforecasting is its popular successor.

The central finding is that expert forecasts performed barely better than chance and worse than simple statistical extrapolation. Experts were poorly calibrated, meaning that events they assigned high probability to occurred less often than claimed, and they were particularly bad at assigning near-zero and near-certain probabilities.

The more interesting finding concerns variation among experts. Using Isaiah Berlin's distinction, Tetlock found that hedgehogs, who apply one organising framework, performed worse than foxes, who hold multiple partial models and tolerate contradiction. Hedgehogs were also more confident, more likely to be invited onto television, and more likely to be badly wrong in exactly the cases their framework covered. Fame correlated negatively with accuracy.

Tetlock also documents the defensive strategies experts use when wrong, including claiming they were almost right, claiming the timing was off, claiming an unforeseeable event intervened, and claiming the prediction was conditional. The book is academic and the statistical treatment is demanding, which is why most readers encounter the findings through Superforecasting instead. The domain is political and economic forecasting over medium horizons, and generalisation beyond that is not established. The finding that confident single-framework thinkers are both more prominent and less accurate is the result with the widest application.