Summary

Among the two kinds of error - bias and noise - the latter is frequently overlooked as a source of high social and financial cost. For example a study conducted by an insurance firm showed a 55% median difference between underwriters, resulting in an estimated $100m+ yearly lost to noise-related inefficiency. Know the 3 types of system noise. Finally, Kahneman lays out 6 principles to overcome noise.

Key Takeaways

• Don’t overlook the costs of system noise: unwanted variability in judgements that should ideally be identical
• Know level noise (between-person) vs pattern noise (within-person, occasion noise)
• Know the 6 principles to overcome bias

Notes

6 Principles to Overcome Noise

1. Accept that decisions are about accuracy, not individual expression
2. Think statistically, and take an outside view of the problem
3. Structure judgement into independent tasks (avoid excessive coherence)
4. Resist premature intuitions
5. Leverage the wisdom of of crowds effect (independent judgements)
6. Favor relative judgements (tend to be less noisy)

Part 1: Finding Noise

• Two kinds of error:
  • Bias: Systematic deviation
  • Noise: Random scatter
• Can recognize and measure noise without knowledge of target or bias
• System noise: Unwanted variability in judgements that should ideally be identical
• Noise audit: Controlled test of system noise
• Median difference between insurance underwriters: 55% → est. $100M+ lost
• Other examples: judicial system, financial investors, patent applications
• Singular decision: (vs recurrent) can’t measure noise, but must recognize it’s still there (eg. Obama’s Zika response)

Part 2: Your Mind is a Measuring Instrument

• Within-person reliability: Single person, multiple trials
• Between-person reliability: Multiple people, same trial
• Many predictions are non-verifiable, eg. probabilistic, theoretical, long time horizon
• Due to unreliability, focus on process rather than result of prediction
• Evaluative judgements: (vs predictive) judgement by evaluating situation / choices
• Decisions involve predictive + evaluative “”
• Use MSE to determine overall error
  • Sum of squares of bias and noise
• Level noise: (between-person) disposition causing bias from average judge
• Pattern noise: Patterns of within-person noise / bias based on traits upon evaluation
  • Predisposition for trait, eg. racist
• System noise = level noise + pattern noise
• Occasion noise: Within-person noise related to context + intrinsic variability
  • Component of pattern noise
  • Internal probability distribution
  • Mood, stress, weather, sequence etc.
• Wisdom of crowds effect: averaging multiple predictions generally superior
• Gambler’s fallacy: Tend to underestimate the probably that streaks occur by chance
• Social influence creates significant noise across groups
• 1 upvote: 32% more likely than no upvote
• Internal discussion often creates greater confidence, unity, extremism → more noise

Part 3: Noise in Predictive Judgements

• Models outperform human judgement
• Simple models can be better, ignoring noise
• Broken leg exception: Decisive information for when to override model’s prediction
• AI/ML excels in unbiased pattern detection
• People are notoriously overconfident in forecasting abilities
• Objective ignorance: many phenomena are hard to predict at all, but we ignore
• Social sciences avg correlation coefficient r = 0.2, 3% r >= 0.5, highly unpredictable
• Causal thinking: when end is known, easy to see likely path - despite many forks in road leading to vastly overestimated probability
• Valley of the normal: Appear normal in hindsight but not expected / predictable
• Causal vs statistical thinking

Part 4: How Noise Happens

• Psychological biases can create system noise or bias depending on nature
• Bias with unknown result: factor that shouldn’t affect judgement does, vice versa
• Excessive coherence / halo affect: tend to jump to conclusions, stick to them (ordering matters)
• Matching: Matching value to another on some other scale (eg. mood → rating)
• Matching predictions: often fail to consider strength of correlation (regression to mean)
• Comparative / relative judgements are more accurate than categorical / absolute
• More sensitive to relative values than absolute values (eg. item cost)
• Scale ambiguity: Choice of scale can make large difference in noise of judgements
• Pattern noise: can be result of differences in knowledge, information
  • Big 5 model of personality: extroversion, agreeableness, conscientiousness, openness, neuroticism (OCEAN)
• Behavior = personality + situation (context)
• Pattern noise > level noise
• Outside view: Properly factoring in averages (similar cases), ignoring the particulars of the case
• Stable pattern noise: per-person bias for given case based on traits, largest component
• Easier to fix level noise, pattern noise often overlooked (eg. grading on a curve)
• Fundamental attribution error: Tend to assign blame / credit to agent that’s better explained by luck, objective circumstances

Part 5: Improving Judgements

• Criteria for a good judge: well trained, more intelligent, right cognitive style
• Respect expert: thought leader whose judgements cannot be objectively evaluated, only compared to other experts
  • Eg. politics, philosophy
  • Vs knowledge expert
• ^ how: years of experience, confidence + coherence, intelligence, grit
• Active open-minded thinking: actively search for contradicting personal opinions
• “When the facts change I change my mind”
• Decision hygiene: rules and steps that prevent bias and noise before they occur
• Base rate: Baseline probabilities of event before personal judgement introduced
• Diverse, independent group → better crowd wisdom, group judgement outcomes
• Guidelines / frameworks reduce noise
• Concreteness of rating scales v. important
• Evaluating companies: rate each part completely independently, individual ratings before discussion, view ratings at end and come to judgement organically

Part 6: Optimal Noise

• 7 objections to reduce or eliminate noise:
  1. Expensive
  2. Efforts might Introduce new errors
  3. Respect + dignity
  4. Essential for evolution of new values
  5. Gaming the system
  6. Noise → risk averse → deterrent
  7. Don’t want to be cogs in machine
• Watch out for noiseless but biased algos
• Rules vs standards: eg. no profanity (well-defined) vs no bullying (interpretable)
• Noise neglected by professionals who don’t see it as relevant to them