Summary

AI and synthetic bio are general-purpose technologies (GPTs) – the 24th and 25th in human history – and they cannot be contained. The essential dilemma: unconstrained technology leads to catastrophe or dystopia, but population collapse means pausing isn't an option. Four features make this wave uniquely dangerous: asymmetric impact, hyper-evolution, omni-use application, and increasing autonomy. Containment must be possible even if history says it isn't – through chokepoints, audits, treaties, and a culture that demands practitioners be the critics. Trust in government has collapsed (70%+ → under 20%), making the challenge existential.

“The closer you are to a technology's beating heart, the more you can affect outcomes.”

Key Takeaways

Notes

1: Homo technologicus

1: Containment is not possible

• Proliferation of tech in waves
• AI + synthetic biology
• Core dilemma: powerful technology leads to catastrophic or dystopian outcomes
• Pessimism aversion trap: overwhelmed by fear of confronting potentially dark realities

2: Endless proliferation

• Engine → driving history
• GPTs: general purpose technologies
• 3 GPTs forming foundation of civilization:
  • Language, agriculture, writing
• ~24 GPTs in history: eg. internet
  • 7 in the last 100 years
• Impossible → inevitable

3: The containment problem

• Consequences of technology are unknown
• Containment: ability to curtail development and use as needed
• History tells us that progress is inevitable
• Nuclear break trend of proliferation
  • Tremendous cost, complexity
  • Still many near misses

2: The next wave

4: The technology of intelligence

• Bits and genes
• Technology is a set of evolving ideas
  • Collide, combine (natural selection)
  • Cumulative, compounding process
• AI is eating the world
• Intro to transformers
• Scaling hypothesis
• Singularity is a red herring
• Automation of todo lists → entire companies
• ACIs: long-term planning

5: The technology of life

• Genetic engineering: read, write, edit
• Alpha Fold
• BCIs

6: The wider wave

• Tech advancements will converge
• Quantum computing
• Fusion

7: 4 features of the coming wave

• hugely asymmetric impact
  • Systemic impacts
• hyper-evolve
  • Even including atoms (vs bits)
• general, omni-use
  • Wider societal effects, spillover
• increasingly autonomous
  • Emergent properties
  • Increasingly impossible to explain

8: Unstoppable incentives

• Sputnik moment: kicked off space race
• ↑: Alpha Go for China and AI
• China is beating the US in almost all education, tech, science measures
• Inevitable state arms race
• Openness = strategic advantage for getting top researchers
• Pursuit of profit → progress

3: States of failure

9: The Grand Bargain

• Tech companies: effective coordination at scale (vs government)
• Regulation is the only option for addressing the upcoming wave
• Decided tech + tools > policy
• Democracy relies on trust, which has collapsed
• Trust in US gov’t has collapsed:
  • Over 70% → under 20%
• More inequality

10: Fragility amplifiers

• Un-contained asymmetry: hackers gaining access to advanced offensive systems
• Tech democratizes access to power
• Democratizing access = democratizing risk
• ↑ ↔︎ centralization ↔︎ anonymity
• Deepfakes → misinformation, truth collapse
• Lab leaks, bioengineering
  • Gain of function research
• Technological unemployment

11: The future of nations

• Big tech companies are massive
• Samsung = 20% of GDP
• Centralization more possible
• Big data → totalitarianism
• Also possible unbundling of power
• ↑ centralizing and decentralizing riptides

12: The dilemma

• Essential dilemma: catastrophe ↔︎ dystopia
• Population collapse → pausing tech is not an option

4: Through the wave

13: Containment must be possible

• Tech is changing too quick to regulate
• Regulation must work well in diverse states
• Eg. vast regulation of automobiles
  • Vs tech, internet, AI
• AI containment: power to manage, curtail
• Ask the following questions:
  • General purpose or specific?
  • Atoms → bits? Hyper evolution
  • Price + complexity coming down?
  • Viable alternatives?
  • Asymmetric impact?
  • Autonomous characteristics?
  • Geopolitical advantages?
  • Favor offense or defense?
  • Resource + engineering constraints?
• No quantitative risk for technology
  • Vs climate change

14: 10 steps towards containment

• Eg. 20% of budget to AI safety
• 1: containment
  • Replication limits, hardware constraint
• 2: audits
  • External scrutiny, red teaming
  • KYC, crypto ledgers
• 3: chokepoints
  • Nvidia → TSMC → ASML
  • Relevant before wide diffusion
• 4: makers
  • Critics must be practitioners
  • “The closer you are to a technology’s beating heart, the more you can affect outcomes”
• 5: businesses: profit and social purpose
  • “Global interest company”
  • B-corp
• 6: governments: survive, reform, regulate
  • Gov’t R&D in-house > commission
  • Already 800+ bills
  • Licensing requirements
  • Labor → capital tax
  • Automation tax
  • UBI
• 7: alliances: time for treaties
  • Eg. nukes, CFCs, bioweapons, genetics
  • Germ line editing pause
  • Secure DNA project
  • AAA: AI audit authority
• 8: culture: respectfully embracing failure
  • Social, moral responsibility
• 9: movements: people power
  • Grassroots activist movements
• 10: narrow path
  • Coherence: each step working together virtuously
• Countries are always at risk