Statistical Consequences of Fat Tails
Technical companion to the Incerto; free preprint on arXiv
Summary
Taleb wrote this as a technical companion to his non-technical books, assembling the mathematical arguments about probability distributions with infinite or very large higher moments. It is written for readers with a working knowledge of probability and statistics.
The central technical point is that under fat-tailed distributions, the standard statistical apparatus produces answers that do not mean what they appear to mean. The law of large numbers still holds under conditions but converges far more slowly, so sample sizes adequate for a Gaussian distribution are inadequate here. The sample mean is not a reliable estimator when the variance is infinite or very large. Correlation and regression become unstable because they depend on second moments. Standard deviation is not a measure of dispersion for such distributions and its use is, in Taleb's view, the profession's most consequential error. Principal component analysis and other techniques resting on covariance matrices inherit the problem.
He develops the distinction between what he calls Mediocristan, where no single observation moves the aggregate, and Extremistan, where one can dominate the entire sample, and shows how to detect which regime a dataset comes from by examining how the maximum observation's share of the total behaves as the sample grows.
Applications covered include the estimation of tail exponents, the shadow mean for bounded observed data, the statistics of violent conflict, pandemics, income distribution and financial returns. He also attacks specific empirical literatures, including some of behavioural economics and some risk management practice, for applying thin-tailed methods to fat-tailed data.
The book is combative and disorganised, assembled from papers and lecture notes with substantial repetition. Statisticians accept much of the technical content, which is not original to Taleb, while disputing his claim that the profession is unaware of it. It is not an introduction and requires mathematical preparation.
Key takeaways 7
- Under fat-tailed distributions the sample mean converges so slowly that ordinary sample sizes give unreliable estimates.
- Standard deviation is not a meaningful measure of dispersion when higher moments are infinite or very large.
- Correlation and regression become unstable under fat tails because both depend on second moments.
- A dataset's regime can be diagnosed by examining how the largest observation's share of the total behaves as sample size grows.
- In fat-tailed domains a single observation can dominate the entire sample, so averages describe nothing.
- Empirical results in economics and finance that assume thin tails are frequently artefacts of the assumption rather than findings.
- Estimating tail behaviour requires fitting the tail directly rather than inferring it from the body of the distribution.
Who should read it
Quantitatively trained readers in finance and risk who need the technical statement of why standard statistical methods fail on financial and extreme-event data.
About Nassim Nicholas Taleb
Former options trader; Distinguished Professor of Risk Engineering, NYU Tandon; scientific adviser to Universa Investments
The Incerto is the primary source for how markets and institutions handle tail risk, and the accompanying technical work supplies the mathematics others cite.
Similar books 6
Chosen by shared topics, the entries that hold them, and the people who recommended both.
Rare, unforeseen, high-impact events drive history, explained only after the fact.
“The message of The Black Swan is how important it is to realize that the things everyone rules out can still come to pass.” — Howard Marks
Source: Oaktree memo 'The Limits to Negativism'
Recommended by: Howard Marks, Marc Andreessen, Nat Eliason and 18 others
Why standard finance models understate market risk, from fractal geometry's creator.
Recommended by: Mohnish Pabrai, Nassim Nicholas Taleb and 1 other
Incerto volume 4
“Nassim Taleb, author of the new book Antifragile, praises volatility and criticises those who would artificially damp it at great unseen cost. Taleb advises us to be 'anti-fragile'. This is exactly…” — Seth Klarman
Source: Klarman on Taleb's praise of volatility and what Baupost strives for
Recommended by: Mohnish Pabrai, Nat Eliason and 25 others
Why people mistake luck for skill, above all in markets.
“Fooled by Randomness is a serious, intellectually sophisticated book, well worth reading carefully. At times, the book is condescending. Nevertheless, there are some great insights.” — Barton Biggs
Source: Biggs on a sophisticated book worth reading carefully
Recommended by: Barton Biggs, Michael Mauboussin and 11 others
Power laws, scaling and critical points across physical and social systems
Recommended by: Nassim Nicholas Taleb
Where it sits in the reference 3
Topics