Summary

Key Quotes

An understood world is more precisely compressible

We are in danger of losing control of the world not to AI or to machines as such, but to models. To formal, often numerical specifications for what exists and what we want

In seeing a kind of mind at work as it digests and reacts to the world, we will learn something both about the world and also, perhaps, about minds.

The core challenge of AI isn't making systems smarter — it's ensuring they optimize for what we actually want. From reward hacking in reinforcement learning to biased training data in classification systems, AI consistently finds ways to satisfy its objective function while violating human intent. The problem runs deeper than technical fixes: we often can't articulate our own values clearly enough to encode them. Inverse reward design, cooperative inverse reinforcement learning, and debate-based approaches represent promising directions, but the alignment problem ultimately reflects our own difficulty in specifying what we mean by "good."

Key Takeaways

• Simple models are better

Notes

Introduction

• Word2Vec: math with words
• Systematic bias, alignment problem
• Three major fields of ML learning:
  • Unsupervised: find patterns, no rules
  • Supervised: labeled examples
  • Reinforcement: reward function
• The alignment problem: ↑

1: Prophecy

1: Representation

• Perceptron: many inputs to one output
  • Doesn’t work with more complex tasks
  • Need multiple layers
• ImageNet, Moore’s Law, GPUs, AlexNet
• Bias based on training data (eg. Kodak)
• What if the world itself is biased?
• Curse of dimensionality: N-gram → network
  • Embedding, vectors
• Modeling after the world we want
• Can use this data as a bellwether to assess progress made towards fixing societal bias

2: Fairness

• Compass parole systematic bias
• Differential privacy: fairness
• Definition of fairness is difficult
• Selection, confirmation bias

3: Transparency

• XAI: explainable AI
• Statistical always >> clinical judgment
• Why simple models are better:
  • Factors conditionally monotone
  • Measurement error
  • Imperfect objective function
• Human expertise: knowing what to look for
  • Not knowing how to weigh
• Multi-task learning: Inputs → outputs
  • Saliency, better results
• De-convolution: visualizing neural networks
  • Foundations of generative AI

2: Agency

4: Reinforcement

• Law of effect: reinforcement learning
• Scalar rewards and actions
• Credit assignment problem: how to determine lessons learned
• Reinforcement learning is universal among life (dopamine system)
• Dopamine: error in expectation of future rewards (promising)

5: Shaping

• Shaping: complex behavior through simple rewards, eg. successive approximations
• Epsilon greedy: exploit >> explore
• Problem of sparse rewards
• Exploitative loop: incentive gone wrong
• Gamification: reward states, not actions
  • Continuous feedback, quantitative

6: Curiosity

• Reinforcement learning: feature selection
• Intrinsic motivation: curiosity, no reward
  • Novelty, surprise, mastery
• Novelty meter based on model of state
  • Reward for novelty signal
• Desire to form a better model of the world
  • Information theory, data compression
  • “An understood world is more precisely compressible”
  • ↑ formal theory of creativity and fun
• Random network distillation (RND): surprise, prediction error as reward signal
• Curiosity → confidence
• Gambling: intrinsic > extrinsic reward
• Knowledge seeking agent

3: Normativity

7: Imitation

• Imitation: foundation of consciousness, social norms, values, ethics, empathy
• Over-imitation: determining relevancy
  • Humans assume rationality
• Advantages:
  • Efficiency: learn from others
  • Safety: can’t afford to fail
  • Do what is hard to describe: tacit
    • Indirect normativity
• RL and imitation: ↑
• Cascading errors: fundamental problem
  • Not shown examples of error recovery
• Solution: co-piloting: random switching off
• Effective altruism: balancing self-sacrifice
  • Slow and steady > unsustainable burst
• On-policy vs off-policy learning agents
• Downsides of imitation:
  • Model behind decision not understood
  • Hard to surpass teacher
• Self-imitation: antagonistic learning
• Amplification: joined fast and slow learning
  • MTS, transcendence
• Iterated distillation and amplification:
  • Human-in-the-loop learning

8: Inference

• Inverse RL: inferring objectives, beliefs
• Human-based reward shaping
• Legibility / transparency
• Conflict of interest (self vs master)
• Hawthorne effect: change in behavior due to being observed

9: Uncertainty

• Knowing when you don’t know
• Open category problem: label uncertainty
• Dropout: not using some portions of NN
• Stepwise relative reachability: take actions that minimize unreachable states
• Attainable utility preservation:
• Want systems that keep options open
• Err on the side of more complexity
• Inverse reward design: inferring actual goals from stated objective
• Effective altruism and evolving morals
• 1 second = 100 trillion lives
• AGI, embedded agency
• Our models should help us understand the world, not define it

“We are in danger of losing control of the world not to AI or to machines as such, but to models. To formal, often numerical specifications for what exists and what we want”
“In seeing a kind of mind at work as it digests and reacts to the world, we will learn something both about the world and also, perhaps, about minds.”