Loading book details
Loading book details
Life 3.0 cover

Life 3.0

Max Tegmark•2017

  1. Chappy's Book Notes•336 books

Life 3.0

Max Tegmark•2017

Length
13h 29m•~384 pages
Read
Jun 2nd - 9th '21
2nd Read
Oct 26th - Nov 9th '26
AIFuturismPhilosophy
•
•

The summary and key takeaways below are auto-generated. I ran an AI pass based strictly on my handwritten notes for this book. I haven't done my own pass over them yet.

I read a book once and take handwritten notes as I go, then leave them alone. Weeks or months later I come back and write the key points and summary from those notes.

The delay is on purpose. Having to rebuild a book out of my own notes does far more for my recall than a second read-through would.

This one has only gotten as far as the AI pass. I'll come back and redo the takeaways and summary myself soon!

Summary

Max Tegmark asks what happens when life can design both its hardware and its software. Unlike biological evolution or human learning, Life 3.0 could redesign the machinery that makes intelligence possible. Because intelligence is the ability to accomplish complex goals, greater capability alone does not tell us whose interests it will serve. Automation could expand prosperity while concentrating ownership and leaving people without income or purpose; an intelligence explosion could also reshape who holds power. Tegmark therefore argues that beneficial AI requires deliberate choices about safety, institutions, and the future people want, rather than confidence that progress will produce a good outcome on its own.

Those choices depend on solving the goal alignment problem: machines must learn, adopt, and retain human goals even as their capabilities change. But agreement about those goals is itself unresolved, and the range of possible futures extends from peaceful coexistence to human disempowerment or extinction. Tegmark’s discussion of consciousness deepens the question by asking whether machines could have subjective experience and what would make a technologically powerful future worth living in. He argues for research and public discussion now, linking safety and human control to shared prosperity rather than treating technical capability as the ultimate aim.

“We could end up in an economy where most of our economy is owned and controlled by machines”

”It’s not that were creating more jobs, just that we’re crowding into the archipelagoes of intellectual capability”

“If consciousness is the way that information feels when it’s processed in certain ways, then it must be substrate independent.”

Key Takeaways

  • Life 1.0: (DNA, biological stage) life that evolves its hardware and software
  • Life 2.0: (learning, cultural stage) life that evolves its hardware but designs much of its software
  • Life 3.0: (technological stage) life that designs its hardware and software
  • Humans are life 2.1 since we can make small hardware upgrades
  • Digital utopians: AGI is natural, desirable next step in evolution
  • Techno-skeptics: Don’t believe in near-term AGI, won’t have imminent impact (100s of years, only slows progress)
  • Beneficial AI movement: AGI not necessarily positive, advocate starting ethics research now
  • Intelligence inevitably linked to goals
  • Human-level AGI: ability to accomplish any cognitive task at least as well as humans
  • Moravec’s Paradox: Intelligence as a landscape with valleys (math) and peaks (creative thinking), water level rising as AI/ML advances
  • Valleys of human capability easier for computers since not required for survival / evolution / specialized circuitry
  • Computronium: any substrate which can perform computational functions (NAND gates)
  • The law of accelerating returns: exponential growth / compounding S curves
    1. Verification: did I build the system right
    1. Validation: did I build the right system
    1. Security
    1. Control
  • Main downside of ML: it’s a black box, can’t understand why it made choice (hard to replace judges)
  • Elon musk: we need governments with insight, not oversight (seed innovation)
  • Update legal system to be fair and keep up with pace of tech
  • Make weapons smarter without arms race
  • Shift to more advanced labor favors college educated
  • Share to capital owners > workers (capital vs labor / bits vs atoms)
  • Superstar theory: Digital economy benefits superstars over everyone else (web: distributability, demand for best) (SaaS)
    1. People + social intelligence
    1. Creativity
  • UBI can likely be used as alternative to work as AI water level keeps rising
  • History shows power hierarchically multipolar
  • Game theory: entities have incentive to cooperate, reach Nash equilibrium where either side is worse off changing strategy
  • Individuals often relinquish power to governing body, eg. cells, governing bodies
  • Single-cell → multicellular → circulatory system → tribes → gov’t
  • Planet-sized AI: 0.1 secs across earth = 4 months for human (physics-imposed speed limit)
  • Hierarchy of power: maximizes trade off between latency / speed and complexity
    1. Libertarian utopia: Humans, cyborgs, uploads, superintellegence coexist peacefully via property rights
    1. Benevolent dictator: AI runs society, strict rules, seen as good thing
    1. Egalitarian utopia: Humans etc. coexist peacefully due to property, UBI
    1. Gatekeeper: Superintellegence to interfere as little as possible to prevent new superintellegence (technical progress stopped)
    1. Protector god: Omnipotent AI maximizes happiness, hides so well existence is doubted
    1. Enslaved god: Controllers use for unimaginable tech and wealth, used for good or bad
    1. Conquerers: AI takes control and eradicates us for variety of reasons
    1. Descendants: AI replace humans, viewed as worthy successors
    1. Zookeeper: Omnipotent AI keeps some humans around, lament fate
    1. 1984: Orwellian surveillance state to permanently curtail AI progress, never reached
    1. Reversion: Progress to AI avoided by reverting to pre-tech society
    1. Self destruction: Humanity drives itself extinct before
  • Still ~30 orders of magnitude from max theoretical compute power, ~12x orders for storage (2017)
  • Dyson sphere: commonly theorized means to extract maximum energy from sun
  • Kardashev scale: type 1 (planet), type 2 (star), and type 3 (galaxy) civilizations
  • Smaller physical size allows for lower latency calculations (eg. flies vs humans)
  • Fermi paradox: Incompatibility of abundance of habitable planets and lack of ET visitors
  • Given universal timescale, if we find intelligent life, likely to already be super-intelligent, non-biological
  • Goal alignment problem: AI and humans having same goals, understanding is critical
  • Value loading problem: aligning with computer (+ goal preservation)
  • All life started with same goal but intelligence has allowed us to further deviate, control our own goals
  • How do we align on minimum ethical principles?
    1. Utilitarianism: positive conscious experiences maximized, suffering minimized
    1. Diversity: diverse experiences over repetition, even if optimal
    1. Autonomy: freedom to pursue own goals unless it conflicts with an overriding principle
    1. Legacy: compatibility with what all humans today would view as good
  • Will all goal-driven entities converge on a goal system / ethical framework? Do we have an ethical destiny?
  • What’s our ultimate goal? Utility? Complexity? Information?
  • PHP: Which physical systems are conscious? What characteristics?
  • EHP: How do physical properties determine qualia (instances of subjective experience)?
  • RHP: Why is anything conscious?
  • Only 10-50 bits of our info consciously processed out of 10^7 every second
  • Integrated information: How much different parts of a system know about each other (psi, measurement for consciousness)
  • Integrated information theory (IIT): How something feels when processing info
  • Tegmark argues consciousness is likely substrate-independent
  • Dyson: Conscious life gives meaning to the universe and its beauty
  • Consciousness / sentience > competence / sapience
    1. Research for beneficial (not undirected) AI
    1. Research funding also for CS, economics, law, ethics, social studies
    1. Science + policy link
    1. Culture of cooperation, trust, transparency
    1. Race avoidance
    1. Safety + security
    1. Failure transparency (no black boxes)
    1. Judicial transparency
    1. Responsibility (of builders - stakeholders in its use)
    1. Value alignment
    1. Human values
    1. Personal privacy (data ownership)
    1. Liberty + privacy
    1. Shared benefit (benefit + empower as many as possible)
    1. Shared prosperity
    1. Human control
    1. Non-subversion (respect + improve existing social systems)
    1. AI arms race (avoid lethal autonomous weapons)
    1. Capability cautions (avoid strong assumptions on upper bounds)
    1. Importance
    1. Risks (planning + mitigation efforts)
    1. RSI (safety + control measures)
    1. Common good (widespread ethical ideals, for benefit of all humanity)

Notes

Introduction

  • IJ Good intelligence explosion quote*
    • RSI
  • Omega / Prometheus hypothetical
    • 1.0 → 10.0 in days
    • Mechanical Turk AWS arbitrage saturation
    • Started with content (movies / TV)
    • UBI-like charity / redistribution
    • → news, politics, persuasion
    • Education + persuasion (depolarization)
    • Agenda 7 slogans: democracy, tax cuts, gov’t social service cuts, military spending cuts, free trade, open borders, socially responsible companies
      • Erode all power structures in the world
    • Outsmarting business tycoons (index funds dominant)
    • Chaos / future shock, but still world getting dramatically better
    • “The Alliance” world order providing social services etc. better than gov’ts - disempowered / toppled

1: Most important convo of our lifetime

  • 13.8B years - our universe is awakening (human consciousness)
  • Life: a information processing machine that can retain its complexity and replicate
    • Life 1.0: (DNA, biological stage) life that evolves its hardware and software
    • Life 2.0: (learning, cultural stage) life that evolves its hardware but designs much of its software
    • Life 3.0: (technological stage) life that designs its hardware and software
  • Humans are life 2.1 since we can make small hardware upgrades
  • —
  • Larry Page: “the most influential exponent of digital Utopianism”
    • “Larry accused Elon of being speciesist”
  • Andrew Ng: techno skeptic - won’t happen in our lifetimes
  • Stuart Russel, Eliezer, Nick Bostrom
  • FoLI + conference → “beneficial AI”
  • —
  • AGI: “intelligence far beyond human level”
  • Consciousness: subjective experience
  • Teleology: explanation of things in terms of their goals or purposes > causes
  • Don’t worry about 3: consciousness, evil, robots
  • —
  • DNA stores 1.6Gb (1.0), brain 10Gb chemically, 100Tb biologically(2.0)
  • AI future opinions in 3 major categories:
    • Digital utopians: AGI is natural, desirable next step in evolution
    • Techno-skeptics: Don’t believe in near-term AGI, won’t have imminent impact (100s of years, only slows progress)
    • Beneficial AI movement: AGI not necessarily positive, advocate starting ethics research now
  • Human-level AGI: ability to accomplish any cognitive task at least as well as humans

2: Matter turns intelligent

  • Intelligence = “ability to accomplish complex goals”
  • Intelligence inevitably linked to goals
  • Moravec’s Paradox: Intelligence as a landscape with valleys (math) and peaks (creative thinking), water level rising as AI/ML advances
  • Valleys of human capability easier for computers since not required for survival / evolution / specialized circuitry
  • —
  • Moore’s Law
  • Eg. DNA 1.6gb, smallest bacteria 40kb
  • Auto-associative memory: recall based on association, not precise address
  • Computronium: any substrate which can perform computational functions (NAND gates)
    • “The hardware is the pattern, the software is the matter and the software is the pattern”
  • The law of accelerating returns: exponential growth / compounding S curves
    • Maintaining Moore’s law, we’d saturate physics max in 200y
  • Brain: 100B neurons, 100T synapses, 15 watts, ~100B flops, 0.1 sec reaction time
  • Hebbian learning: can learn anything

3: The near future

  • Deep RL (DeepMind 2014)
  • Alpha Go - creativity + intuition
  • NLP, DNN + GOFI
  • Important questions to answer:
    1. How do we make AI systems more robust
    2. Update legal system to be fair and keep up with pace of tech
    3. Make weapons smarter without arms race
    4. Grow prosperity through automation without losing income or purpose
  • 4 areas of safety research:
    1. Verification: did I build the system right
    2. Validation: did I build the right system
    3. Security
    4. Control
  • —
  • AI as a moral imperative in various fields but also lots to work out on implications and timing
    • “We could end up in an economy where most of our economy is owned and controlled by machines”
  • Main downside of ML: it’s a black box, can’t understand why it made choice (hard to replace judges)
  • Elon musk: we need governments with insight, not oversight (seed innovation)
  • Tech (not globalization) has increased wealth disparity:
    1. Shift to more advanced labor favors college educated
    • Share to capital owners > workers (capital vs labor / bits vs atoms)
    • Superstar theory: Digital economy benefits superstars over everyone else (web: distributability, demand for best) (SaaS)
  • Job recs:
    1. People + social intelligence
    2. Creativity
  • —
  • ”It’s not that were creating more jobs, just that we’re crowding into the archipelagoes of intellectual capability”
  • UBI can likely be used as alternative to work as AI water level keeps rising - theory for viable human jobs (as long as cheaper)
    • “The conversation… should include everyone”

4: Intelligence explosion

  • First risk: erosion of privacy and us accepting the tradeoff
  • 5yo cage breakout hypothetical*
  • AGI takeoff speed: will it happen quick (monopoly) or slowly (multiple players)?
  • History shows power hierarchically multipolar
  • Game theory: entities have incentive to cooperate, reach Nash equilibrium where either side is worse off changing strategy
    • Individuals often relinquish power to governing body, eg. cells, governing bodies
    • History of more coordination over longer distances
    • Single-cell → multicellular → circulatory system → tribes → gov’t
  • Planet-sized AI: 0.1 secs across earth = 4 months for human (physics-imposed speed limit)
    • Still don’t know whether that’ll mean more democracy or centralization
  • Expert AGI prediction: median 2055 → 2047

5: The next 10,000 years

  • AI Aftermath Scenarios
    • ↓ not exhaustive, more illustrative of span
  • 12 possible scenarios for AGI:
    1. Libertarian utopia: Humans, cyborgs, uploads, superintellegence coexist peacefully via property rights
    2. Benevolent dictator: AI runs society, strict rules, seen as good thing
    3. Egalitarian utopia: Humans etc. coexist peacefully due to property, UBI
    4. Gatekeeper: Superintellegence to interfere as little as possible to prevent new superintellegence (technical progress stopped)
    5. Protector god: Omnipotent AI maximizes happiness, hides so well existence is doubted
      • The odyssey problem: allow some suffering to conceal
    6. Enslaved god: Controllers use for unimaginable tech and wealth, used for good or bad
      • EA all life / consciousness argument → slavery
    7. Conquerers: AI takes control and eradicates us for variety of reasons
    8. Descendants: AI replace humans, viewed as worthy successors
    9. Zookeeper: Omnipotent AI keeps some humans around, lament fate
    10. 1984: Orwellian surveillance state to permanently curtail AI progress, never reached
      • Must be a global totalitarian state due to economics
    11. Reversion: Progress to AI avoided by reverting to pre-tech society
    12. Self destruction: Humanity drives itself extinct before
      • Omniside is possible but very hard

6: Cosmic endowment

  • Dyson sphere: commonly theorized means to extract maximum energy from sun
  • E = MC^2 to maximize energy potential
    • Food digestion: 0.00000001% efficient
    • Gasoline: 0.00000005%
    • Dyson sphere / fusion: 0.08%
    • Fission: 0.9%
    • Black hole engine: 29%
    • Dyson sphere quasar: 42%
    • Black hole evaporation: 90%
  • Still ~30 orders of magnitude from max theoretical compute power, ~12x orders for storage (2017). Quantum is next big step
  • Kardashev scale: type 1 (planet), type 2 (star), and type 3 (galaxy) civilizations
  • How to travel thru space faster*
  • Fastest way to colonize universe is to broadcast a special message at speed of light to trick other life into meeting our goals
    • Have to be very careful with potential incoming extraterrestrial contact
  • End of universe 5 scenarios*
  • Enough to power in the universe to simulate 10^58-69 lives
  • Hierarchy of power: maximizes trade off between latency / speed and complexity
    • Eg. eyes blinking vs. brain thinking
    • Power / thought / computation / control
    • Smaller physical size allows for lower latency calculations (eg. flies vs humans)
  • Fermi paradox: Incompatibility of abundance of habitable planets and lack of ET visitors
    • Implies a great filter: if we find life, implies abundance and great filter is ahead of us
    • Given universal timescale, if we find intelligent life, likely to already be super-intelligent, non-biological

7: Goals

  • Physics / nature is goal oriented: eg. Light bending in water, maximizing entropy
  • Gravity is a force that works against entropy, forming figure from uniformity
  • Ervin Schroeder: living systems reduce entropy by increasing entropy around it
  • Living organisms: bounded rationality, use feelings as evolutionary evolved rules of thumb in pursuing goals
    • Our human feelings, not genes, are the ultimate authority
  • Teleology: explaining in terms of purpose
    • Universe becoming more teleological
  • Goal alignment problem: AI and humans having same goals, understanding is critical
    • Applies to all types of machines, eg. mouse trap needs to know what mouse is
  • Value loading problem: aligning with computer (+ goal preservation)
  • Friendly AI: must learn, adopt, and retain our goals
  • All life started with same goal but intelligence has allowed us to further deviate, control our own goals
  • How do we align on minimum ethical principles?
    1. Utilitarianism: positive conscious experiences maximized, suffering minimized
    2. Diversity: diverse experiences over repetition, even if optimal
    3. Autonomy: freedom to pursue own goals unless it conflicts with an overriding principle
    4. Legacy: compatibility with what all humans today would view as good
  • 3 stages of goal oriented behavior:
    1. Matter intent on maximizing its dissipation
    2. Primitive live trying to maximize its replication
    3. Humans pursuing goals > replication, like curiosity, compassion, and other feelings evolved to help them replicate
    4. Machines built to help humans pursue their human goals
  • Will all goal-driven entities converge on a goal system / ethical framework? Do we have an ethical destiny?
    • Eg. The Better Angels of Our Nature
  • Superintelligence’s orthagonality thesis
  • What’s our ultimate goal? Utility? Complexity? Information?

8: Consciousness

  • Consciousness: subjective experience
  • PHP: Which physical systems are conscious? What characteristics?
  • EHP: How do physical properties determine qualia (instances of subjective experience)?
  • RHP: Why is anything conscious?
  • Behavioral correlates of consciousness: evidence of type 2 thinking, eg. language and abstract reasoning
  • Flow = unaware of the lower level details
  • Only 10-50 bits of our info consciously processed out of 10^7 every second
  • Vast majority of neurons in body (brain, gut, eyes) don’t contribute to consciousness
  • Need a theory of consciousness
  • —
  • Emergent property: config of individual parts → emergent property (eg. wetness)
  • Integrated information: How much different parts of a system know about each other (psi, measurement for consciousness)
  • Integrated information theory (IIT): How something feels when processing info
    • “The most mathematically precise consciousness theory to date”
    • Predicts that consciousness cannot be split, no hierarchies of consciousness
  • Core of consciousness: Remember, compute, learn, subjective experience
  • Centroneum: “the most general substance that has subjective experience”
  • Consciousness is likely substrate-independent, can probably be proven*
    • “If consciousness is the way that information feels when it’s processed in certain ways, then it must be substrate independent.”
  • 4 necessary conditions of consciousness:
    1. Information principle: a conscious system has substantial information storage capacity
    2. Dynamics principle: “” processing capacity
    3. Independence principle: “”
    4. Integration principle: “” cannot consist of nearly independent parts
  • System 0-2 → hierarchies of conciousness
  • Dyson: Conscious life gives meaning to the universe and its beauty
    • Therefore, first goal of universe should be to increase consciousness
  • Consciousness / sentience > competence / sapience

Epilogue

  • Technological swewardship
  • The media fuels misunderstanding
  • Elon was the enabling funder for FoLI
      • Open Philanthropy
  • → led to OpenAI too
  • Asylamar AI principles:
    1. Research for beneficial (not undirected) AI
    2. Research funding also for CS, economics, law, ethics, social studies
    3. Science + policy link
    4. Culture of cooperation, trust, transparency
    5. Race avoidance
    6. Safety + security
    7. Failure transparency (no black boxes)
    8. Judicial transparency
    9. Responsibility (of builders - stakeholders in its use)
    10. Value alignment
    11. Human values
    12. Personal privacy (data ownership)
    13. Liberty + privacy
    14. Shared benefit (benefit + empower as many as possible)
    15. Shared prosperity
    16. Human control
    17. Non-subversion (respect + improve existing social systems)
    18. AI arms race (avoid lethal autonomous weapons)
    19. Capability cautions (avoid strong assumptions on upper bounds)
    20. Importance
    21. Risks (planning + mitigation efforts)
    22. RSI (safety + control measures)
    23. Common good (widespread ethical ideals, for benefit of all humanity)

Mindful optimism

  • What sort of future you want, not what sort of future you fear
  • “Please discuss all of this with all those around you”