Summary

Researchers nurtured AI for decades before it got snapped up by big tech. Deep nets remained discredited long after the XOR objection was debunked; GPUs, more data and compute brought practical breakthroughs in STT and ImageNET. The DNN Research auction, DeepMind acquisition and NFL quarterback level salaries showed the monetary potential of deep learning. Open source put value in network effects rather than the software itself.

The pursuit of AGI brought rivalry, ethics boards and an OpenAI that moved from non-profit → for profit. Deepfakes demand critical thinking; data selection matters to reduce bias. Gary Marcus argued deep learning needs mechanical components. Games offer RL environments, but the real world has no score / optimization function.

“LeCun is the only one who can actually get neural nets to work”
– Andrew Ng

“What AI cannot create it cannot understand”

“Self belief to the point of delusion”
– Sam Altman

“My mission is to create broadly beneficial AGI”
– Sam Altman

Key Takeaways

Notes

Prologue: The Man Who Didn’t Sit Down

• Hinton: formed DNN Research startup with him and 2 colleagues in 2012 for >> acquisition (vs hiring) price with 2 colleagues
• ↑ auction: Baidu, Google, Microsoft, DeepMind (Demis)
• In $1M increments via email
• $20M → $40M+, sold to Google after stopping bidding
• First hint at monetary potential of deep learning
• The story of a group of researchers who nurtured AI for decades before it got snapped up by big tech

1: A new kind of machine

1: Genesis

• Frank Rosenblatt + perceptron
• First demo in 1958
• “The first serious rival to the brain”
• Marvin Minsky building first neural net, Dartmouth AI Conference in 1956
• Minsky tore down idea of perceptron - couldn’t recognize XOR
• Set back non-symbolic AI 15y

2: Promise

• Hinton had multi-generational successful academic lineage (eg. Everest, Bool)
• Physics → psychology → AI
• Minsky’s paper fascinated him - false characterization
• Backprop
• Boltzmann Machine “crazy idea”
• Moved from CMU to Toronto due to political climate

3: Rejection

• Yann LeCun CNNs
• Andrew Ng: “LeCun is the only one who can actually get neural nets to work”
• Deep nets considered 3rd best option in late 2000s
• Discredited for no XOR (debunked 20y earlier)
• Didn’t consider 1Mx more compute
• Andrew Ng preached neural networks at Stanford but received flack
• Basically just NYU, ↑, Toronto/Montreal, Switzerland, Japan
• LSTM
• Hinton fun / quirkiness

4: Breakthrough

• STT
• Hinton GPUs for ↑
• Google’s cloud didn’t have GPUs (first real need)
• Android STT applications

5: Testament

• Hawkins A Thousand Brains: brain biological algorithm → neural networks
• Andrew Ng ↑
• Averse to updating Google search due to black box
• Jeff Dean: DEC → Google page rank
  • The legend of “”
• 2012 Coursera + Udacity
• Dean not driven by ego
• Ilya Sutskever “the only person with more good ideas than me” - Hinton
• Alex K ImageNET
• Used LeCunn’s CNN from 80s
• → DNN Research auction

6: Ambition

• DeepMind crazy goal of AGI
• Bright, competitive, good at social interactions (dangerous combo)
• Hassabis video games company folded, went to get neuro PhD
• Need to be seriously considering AGI
• Mustafa Suleyman joined and provided early capital
• Eliezer Yudecowsky + Theil Singularity Summit in 2000s
• DeepMind: Hinton + LeCun as advisors
• Acquired by Google for $650M

2: Who owns intelligence?

7: Rivalry

• Facebook FAIR in NYC with LeCun
• DeepMind insisted on AGI ethics board upon acquisition but Mark wouldn’t
• Zuck’s vision for agents (explicitly no robotics)
• Open source belief = the value is in the network effects not the software itself
• AI researchers fetching NFL quarterback level salaries

8: Hype

• Find people at the intersections of industries who can change the picture
• Finally got go-ahead for $160M purchase of GPUs despite Google corporatism
• JG
• AI buzz word
• Alex K viewed AI as data science with more data and compute
• Vector embeddings
• TPUs
• Model distillation

9: Anti-hype

• Musk voiced concern about Google / Paige AGI / terminator
• Bostrom Super-intelligence
  • Paper clips + RSI
• AI safety in Google
• Ethical considerations
• FoLI in 2014
• OpenAI
  • Free from big tech and profit motives
  • Rosewood with Brockman, Elon, Sam, Ilya
  • Top 10 for wine, 9 joined 3 weeks later

10: Explosion

• Alpha Go moment

11: Expansion

• Google Brain vs DeepMind tension
• Suleyman better fit for Brain > DeepMind (more practical)

12: Dreamland

• Theory: Microsoft burdened by outdated procedural thinking of leadership + managers
• Qi Lu → Yoshua Bengio recruited for Microsoft - refused
• AI / robots need time to dream

3: Turmoil

13: Deceit

• GANs the product of Goodfellow’s late night drunk experimentation
• “What AI cannot create it cannot understand”
• Deepfakes
• People will have to rely on critical thinking

14: Hubris

• Google return to China via Alpha Go 2
• TensorFlow Dean / Google open source as AI + cloud compute bootstrap
• Alpha Go = China’s Sputnik moment → leader in AI by 2030
• China will win cuz data > algos

15: Bigotry

• Data selection matters to reduce bias
• Groupthink, insularity

16: Weaponization

• Project Maven controversyWeaponized AI controversy

17: Impotence

• Content moderation
• Fake news

4: Humans are underrated

18: Debate

• Google Duplex too powerful
• Gary Marcus criticism - narrow AI
  • Studied under Steven Pinker
• Needed more than deep learning → geometric intelligence
• Rebooting AI
• Deep learning needs mechanical components
• BERT for translation / NLP

19: Automation

• Robots + simulation
• OAI conspicuous projects
• Karpathy Elon poach → Tesla
• OAI non-profit → for profit

20: Religion

• Elon stoking fear
• AGI “out of touch with reality” → “insanely visionary”
• Drives researchers in a religious way
• Sam Altman
  • 20yo Loopd $30M
  • YC → focus on wealth + spotting talent
  • “Self belief to the point of delusion”
  • “My mission is to create broadly beneficial AGI”
• Games as RL environments - but can’t define for life
  • Real world has no score / optimization function
• Chips for NNs
• Google + Microsoft to provide money + hardware
• Nothing preventing AGI

21: X Factor

• Nobel prizes