Summary

Key Quotes

Humankind gains immense power from building large networks of cooperation, but the way these networks are built predisposes us to use that power unwisely

Telling the truth is hardly the most efficient way to produce order among large numbers of humans

The emergence of computers capable of pursuing goals and making decisions by themselves changes the fundamental structure of our information network

Humans are good at gaining power but bad at gaining knowledge—and the gap between the two is the story of civilization. Large-scale cooperation requires information networks, but those networks are built on fictions and inter-subjective realities (laws, gods, nations, money) because truth is too complex and painful to organize people around. Computers change everything: for the first time, a non-human agent can create information, make decisions, and build intimate relationships at scale. AI doesn't just amplify existing networks—it restructures them, favoring centralization of data in ways that enable totalitarianism. The naive view that more information leads to more truth is wrong; truth requires self-correcting institutions, and those institutions are exactly what AI threatens to undermine.

Key Takeaways

Notes

Prologue

• Humans are good at gaining power, bad at gaining knowledge
• Main argument of book: “humankind gains immense power from building large networks of cooperation, but the way these networks are built predisposes us to use that power unwisely”
• Network, information problem
• Reliance on fiction and fantasies
• The naive view of information: big networks achieve a better understanding
  • Implies information leads to truth
• AI double edged sword

1: Human Networks

1: What is information

• Information is the basis of the universe
• Most info is not truth
• Truth: something that accurately represents some aspect of reality
• Truth is universal for reality
• Problems with truth:
  • Over-simplifies (eg. 10k soldiers)
  • Many viewpoints
• “Truth is something that brings our attention to certain aspects of reality, while inevitably ignoring other aspects”
• “Information doesn’t necessarily inform us about things, rather it put things in formation”
• Information as a social nexus

2: Stories

• Humans unique ability to cooperate flexibly in large numbers
• Individuals are brands, eg. Jesus
• Parasocial: resonating with an individual’s story, not the individual themselves
• Objective vs subjective reality
• Inter-subjective reality: exist in the nexus between large numbers of minds in the stories we tell one another
  • Eg. laws, gods, nations, corporations
• ↑ exists in the exchange of info
• Experts at creating ideologies are those who control the power
• “Telling the truth is hardly the most efficient way to produce order among large numbers of humans”
• 2 advantages to fiction > truth:
  • Simpler
  • The truth is often painful, disturbing
• Truth and order are often contradictory

3: Documents

• Lists vs stories: brains remember stories far better than lists
• Documents allow ownership to be determined by a central authority
• Bureaucracy: the way people in large orgs solve the retrieval problem
• Inter-subjective conventions:
  • Eg. life vs non-life, types of science
• Difficult to depict bureaucratic realities

4: Errors

• Claiming infallibility is dangerous
• Books are a precursor to the blockchain
• Ambiguity → interpretation → power
• Churches’ choice of books: real power in the curating institution
• Free marketplace of information does not lead to truth
  • Eg. printing press and witch hunts
• Toxic information spheres: ↑
• Truth relies on curation institutions with self-correcting mechanisms
• Truth and order are at odds

5: Decisions

• Democracy vs dictatorships
• Dictatorial: highly centralized info flow
  1. Unlimited authority
  2. Assume the center is infallible
• Totalitarianism: controlling all info
• Democracy: distributed info system containing self-correcting mechanisms
  1. Clear limits on the power of the center
• Majority dictatorship: no clear limits
• 2 baskets of rights protected from majority vote in democracies:
  1. Human rights: privacy, religion, etc.
  2. Civil rights: vote, assembly, media
• ↑ intersubjective conventions
• Truth cannot be determined by a simple majority vote
• Populism: one party representing all (bad)
  1. “We alone represent the people”
  2. Power is the only reality
• Education and media are critical to creating an informed populous → democracy
• Mass media makes mass democracy possible
• US’ checks and balances
• Authoritarian vs totalitarian: above the law vs controlling what everyone says + thinks
• Leaders under Stalin: 80-90% killed
• Modern totalitarianism vs the church:
  1. Church was accessory to policy
  2. Demanding vs resisting change
  3. Churches tended to be local affairs
• Pro of dictatorship: more orderly
• Con of dictatorship: Information chokepoints / blockage

2: The inorganic network

6: Computers vs printing presses

• Big change: computers can create info
• Equivalent power of curators, but now algos
• Engagement optimization → create outage
• Intelligence != consciousness
• Intelligence: the ability to attain goals
• Consciousness: the ability to obtain subjective experience
• “The emergence of computers capable of pursuing goals and making decisions by themselves changes the fundamental structure of our information network”
• AI is document → document (no human)
• AI-based scripture (eg. Q Anon)
• AI can mass-produce intimate relationships with people, extreme personalization
• Progress is speeding up, unpredictable
• AI + human-level intelligence: “ like comparing airplanes to bird-like flight”
• Big tech is not subservient to customer and regulation, it increasingly controls them
• Taxes need to be rethought, eg. nexus for companies should include online presence
  • Eg. tax for data extraction (much like oil extraction conventions)
• Rise in info-for-info deals in place of $$

7: The network is always on

• Surveillance → data + analysis has been important throughout history
• Computers have become the nexus of most human activities
• “We kill people based on metadata”
• Big data, AI for surveillance
• Surveillance capitalism: ever-expanding commercial monitoring system
• Peer-to-peer surveillance: loss of privacy
  • Eg. Tripadvisor
• Social credit system: governmental ↑
  • Basically like $$, but for social reputation

8: The network is often wrong

• Social media RL: engagement optimization → outrageous content
• The alignment problem:
  • Eg. military wins can lead to political defeat
• There’s no rational way to determine an ultimate goal ↑ that will not depend on alignment with a higher goal, except 2:
• Deontology: universal moral duties
  • Eg. the golden rule
  • Eg. any entity capable of suffering
• Utilitarianism: min suffering max happiness
  • Happy vs delusional (eg. eternal life)
  • String enough belief in utopia justifies any suffering now
• Inter-computer realities: eg. Google SEO
• Algorithmic bias

3: Computer politics

9: Democracies

• Tech isn’t purely beneficial
• Eg. industrialization → imperialism, Nazis + Stalinism, climate change
• Liberal democracy → strong self-correcting mechanisms
• Principles for protecting against totalitarian surveillance regime:
  • Benevolence: fiduciary duty for data holders, non-malicious use
  • Decentralization: gov’t shouldn’t have all data
  • Mutuality: match surveillance levels of individuals, gov’ts, and corporations
  • Must leave room for both change and rest
• Creativity: the ability to recognize patterns and then break them
• Future of jobs will be volatile
• Progressives vs conservatives
• Modern conservatives want a far-right revolution: groundless hysteria?
• “The right to an explanation”: should keep algorithmic bias in check
• Unfathomability: “many technologies and systems are becoming so complex that they’re beyond the capacity of any one individual to truly understand them”
  • Eg. only 12% understand loans
• ↑ leads to rise in popularism / simplifiers
• Fallacy of the single cause: humans admit a single cause, algos weigh tons of factors
• ↑ at odds with intuitive explainability
• AI social credit system completely outlawed in 2021 EU AI act
• Gov’ts should outlaw counterfeit humans like they do with counterfeit currency

10: Totalitarianism

• Unlike humans, AI tends to favor the concentration of data in one place
• Blockchain is supposed to be a check, but 51% can we worked around
  • Even allows for rewriting history
• Double speak in Russia: learned that asking questions leads to trouble
• Russel-Einstein Manifesto: against nuclear war. Worked for dictatorships as well

11: Silicon Curtain

• We’re living in a post-imperial era
• Computers pose 2 main challenges to international systems:
  • Easier concentration of data could lead to new imperial era
  • Could split into regional networks
• Data colonialism: land → data as primary source of value
• Study: US, China to capture 70% of AI value creation by 2030
• US China sanctions → Silicon curtain
• Information cocoons: ↑
  • Eg. Mind-body problem: how religion / society view the body - as sacred or vestigial - guides AI development
• Cyber warfare is more ambiguous, uncertain than traditional war
  • Eg. game theory, uncertain MAD
• Globalism vs patriotism: false dichotomy
• Necessity of worldwide cooperation (eg. negative externalities)
• Principles of globalism:
  • Agreement to global set of rules
  • Sometimes necessary to prioritize long-term interests of all humans over short-term of a few humans
• Why AI regulation requires utmost trust:
  • Easier to hide AI labs
  • AI has more dual civilian use cases
    • Eg. vs nuclear
• If humans were only interested in power, states would not have been possible
• Military spending: world 7% vs Russia 30%

Epilogue

• History vs tech, business, etc.
• Historical comparisons under-weigh:
  • Unprecedented nature of AI
  • Negative results of previous revolutions
• Goal of book: to provide a more accurate historical look at the AI revolution
• Shouldn’t reduce view of all decisions as a power struggle