Summary

After conducting a study on forecasters, Philip Tetlock determined a set of traits that encapsulate super-forecasters. This includes: probabilistic, curious, humble, open-minded, intelligent, numerate, dragonfly-eyed / a fox, collaborative, reflective, and resilient. This book provides tips for being a better forecaster.

However, Tetlock applies interesting caveats, including in times where a leader’s authority is needed and in the long term (3-5 years) when black swan events tend to muddle any predictability and experts are as good as dart-throwing chimpanzees.

Key Takeaways

• Don’t bother predicting more than 3-5 years in advance
• Practice active open-mindedness: seek dissenting opinions
• Live in perpetual beta
• Share knowledge, join communities. Givers who openly contribute their insights tend to do better than takers

Notes

10 Commandments for Super-forecasters

1. Triage: Focus on questions not too easy or too difficult, enough to learn from
2. Decompose: Break seemingly intractable problems into sub-problems (eg. Fermi)
3. Predict: Strike the right balance between inside and outside views
4. Update: Strike balance between over- and under-reacting to new evidence (Bayesian)
5. Look for the clashing causal forces at work in each problem (seek counterarguments)
6. Granularity: Strive to determine correct amount of doubt in prediction
7. Strike the right balance between over- and under-confidence (calibration, resolution)
8. Learn: Look for the errors behind prediction mistakes, but beware hindsight bias
9. Collaborate: Bring out the best in others, let them bring out the best in you
10. Practice: Master the error-balancing bicycle
11. Don’t treat commandments as commandments (just guidelines)

Chapter 1: An Optimistic Skeptic

• Expert’s predictions basically random chance by 3-5 years out
• Butterfly affect: there are hard limits on predictability
• Super-forecasting: mostly way of thinking
  • Thinking that is open-minded, careful, curious, self-critical
• 200+ studies: algorithms beat subjective judgement

Chapter 2: Illusions of Knowledge

• We overestimate our expertise, validity / likelihood of our judgements
• We come up with narratives for our behavior, even if there is no true reason
• Apply doubt to your system 1 snap judgements, sanity check with system 2

Chapter 3: Keeping Score

• Many predictions are hard to evaluate due to vague, ambiguous language
  • Eg. “Likely” = 50% or 90%?
• Using language instead of numbers can be advantageous - explicit uncertainty
  • Using a number implies precise calc.
• Calibration: Model doesn’t tend to skew high / low certain segments of probability
• Resolution: Model is confident / decisive in predicting whether event will happen
• Briar score: Weighted distance between prediction and result (like betting odds)
• Hedgehog vs fox: (inventor of phrase) foxes outperform hedgehogs
  • “The fox knows many things, but the hedgehog knows one big thing”
• Wisdom of crowds: Average of many judgements often better than single

Chapter 4: Superforecasters

• We aren’t good at factoring in luck
  • Anomalous bets seem prophetic
  • No accountability: can say market will crash until it actually does
• Rule of thumb: regression to the mean

Chapter 5: Super-smart

• Inside view: overview / demographical view
• Outside view: adjusting to unknown particulars of case
• Anchoring: take the inside view first before using outside view to adjust
• Active open-mindedness: question your own judgement, look for dissenting viewpoints

Chapter 6: Super-quants

• Science is grounded in probability, nothing is certain
• “Scientific facts that look as solid as rock to one generation of scientists can be crushed to dust by the next. All scientific knowledge is tentative”
• Finding meaning (fate) in events is hallmark of a happy, healthy mind
• Belief in fate also correlated with less accurate forecasting
• Thus, fate seemingly puts forecasting / success at odds with happiness

Chapter 7: Super-news-junkies

• The best forecasters aren’t afraid to adjust predictions when new info presents itself
• The key is to not over- or under-account
• Bayes’ theorem: helps to properly weigh new data, adjust predictions
• Generally make small adjustments, but don’t be afraid to make large change in face of barbarous circumstances

Chapter 8: Perpetual Beta

• The best forecasters always keep learning
• Need to incorporate feedback from results of previous predictions
• Hard to learn from predictions w/o feedback (ambiguous lang., non-testable, qualitative)
• Time lag: (hindsight bias) we misremember, overestimate accuracy of our forecasts the further out they were (unless documented)
• Perpetual beta: degree committed to belief updating, self improvement (most powerful predictor of forecasting skills)

Chapter 9: Super-teams

• Groupthink vs wisdom of the crowd
• Fostering an environment where dissenting opinions are encouraged
• Givers vs takers: people who more openly contribute their insights tend to do better
• Diversity of background, opinions better
• Precision question: help others fully understand their own argument
• Constructive confrontation: learn to disagree without being disagreeable

Chapter 10: The leader’s dilemma

• Need to balance authority with uncertainty, openness to new ideas
• Leave room for autonomy
• Be humble yet confident

Chapter 11: Are they really so super?

• What you see is all there is (WYSIATI): Can’t see beyond our own egocentric viewpoint; most common cognitive bias (kind of like availability bias)
• Scope insensitivity: not correctly adjusting for scale in predictions (eg. 2k vs 200k)
• Black swan: An event that was previously unfathomable or incredibly unlikely
• ↑ Argument can be made that this is the only important thing in the long run