#161 | Frankly
Humanity’s Icarus Moment: The Wings, the Sea, and the Sun of AI
This week Nate examines artificial intelligence through the metaphor of Icarus, asking what risks arise when a civilization injects trillions of cognitive workers into a civilization that’s already in resource overshoot. Emphasizing that our society already largely ignores the depleting fossil energy that powers our lives, Nate traces how AI could amplify our economic growth imperative rather than help us escape it. He also touches on optimization – how solving solely for efficiency erodes the robustness and slack that allow our human systems to adapt to energy and material descent.
The latter part of Nate’s analysis turns toward risks related to power concentration and the downstream effects of increased societal integration of AI. He considers the long-term implications of trends like shrinking entry-level opportunities, mounting cognitive debt, and prose averaging out across research, media, and even everyday life. He also dives into potential nearer-term risks like the emergence of technofeudalism and decreasing costs of surveillance. Nate ends the episode by leaving the last risk unnamed, arguing that the most important consequence of transformative technology may be the one nobody can foresee.
How do AI “cognitive workers” interact with the logic of the economic Superorganism? What happens when a civilization optimized for growth without accounting for resilience introduces a technology such as AI? And can humanity choose a middle path when the systems driving AI are rewarded for extremes?
Show Notes & Links to Learn More
Download transcriptThe TGS team puts together these brief references and show notes for the learning and convenience of our listeners. However, most of the points made in episodes hold more nuance than one link can address, and we encourage you to dig deeper into any of these topics and come to your own informed conclusions.
Grab & Go Resources
New? Start here:
– The Superorganism Explained in 7 Minutes (Frankly #97)
– Energy Blindness (Frankly #3)
– The 8 Faces of AI (Frankly #96)
From the Archive:
– Ep 233: Roman Yampolskiy, “Don’t Build Gods”: One AI Risk to Rule Them All
– The Quadruple Bifurcation (Frankly #112)
Episode Foundations:
– Generative AI Without Guardrails Can Harm Learning: Evidence from High School Mathematics (PNAS)
– Your Brain on ChatGPT: Accumulation of Cognitive Debt (MIT Media Lab)
– IEA: Data centre electricity use surged in 2025 even with tightening bottlenecks
Next Steps:
– AI Toys Risk Assessment and parent guidance
– Big Tech Unchecked: a toolkit for community action on hyperscale data centers
Resources by Timestamp:
01:16 – Daedalus’s warning to fly the middle course (Ovid, Metamorphoses Book VIII)
02:38 – Human civilization as a metabolic superorganism (The Superorganism Explained in 7 Minutes, Frankly #97), ~20 terawatts of continuous energy consumption
02:52 – Financial credit as the superorganism’s bloodstream
03:33 – Army of ~500 billion fossil workers, 8 billion humans (Energy Blindness, Frankly #3)
03:57 – Carbon Pulse
04:14 – Trillions of AI agents as a tireless cognitive workforce
05:01 – Daedalus imprisoned by King Minos in the Labyrinth
05:36 – Growth on a finite planet, AI as a promised escape from old limits (Sam Altman’s essay The Intelligence Age)
Risk #20 – The Fossil Lock-In
06:11 – Climate Week NYC 2026
06:19 – Skyrocketing data center electricity load
06:27 – Coal plant retirements being deferred (15 U.S. coal plants kept open for AI data centers)
06:30 – Obsolete gas peaker plants pressed back into service
06:32 – In-video insert: Graph about mothballed gas plants being restarted
06:34 – Combined-cycle gas turbines ordered in gigawatt blocks with multi-year delivery backlogs
06:34 – In-video insert: Headline – about combined-cycle turbines on backorder
06:45 – Used jet engines repurposed to power AI data centers
06:45 – In-video insert: Headline – used jet airplane engines generating power for AI
06:53 – Super El Niño (The Great Simplification Ep 225: What Does a Super El Niño Mean for the Climate?), Global warming in the pipeline (Hansen et al. 2023)
07:03 – AI-driven carbon lock-in (Analysis of the gas build-out for data centers)
07:08 – Gas plants financed on 30- to 40-year lifetimes vs. four-year AI chip replacement cycles
07:25 – Opportunity cost of diverting transformers, turbines and electricians to data centers
07:41 – Grid stabilization and transmission upgrades, Rebuildable energy, Climate adaptation and resilience
Risk #19 – The Appetite
08:24 – Data center cooling water drawn from stressed basins, Closed-loop cooling systems
08:37 – *U.S. golf courses use roughly three times the water of all U.S. data centers combined
08:39 – In-video insert: Graph on how much water golf courses use
08:43 – Rising copper demand into declining ore grades
08:44 – In-video insert: Graph of rising copper demand, Report source
08:56 – AI chips replaced on a four-year cycle feeding the electronic-waste stream
09:06 – Energy per AI prompt has collapsed (33-fold in a year at Google) while total AI electricity consumption rose anyway
09:32 – Jevons paradox (Jevons paradox applied to AI)
09:55 – Energy slaves (Nate and DJ’s book covering such)
Risk #18 – The Rebound
10:04 – Rebound effect (Nate and DJ’s book covering such)
10:34 – AI efficiency gains across the real economy: Trucking, Farming, Mining, Drug discovery, Oil and gas exploration
10:43 – Khazzoom–Brookes postulate
11:02 – Possible future shapes of the Carbon Pulse
11:27 – Wide-boundary thinking pointed at AI in service of life
11:47 – Growth imperative
Risk #17 – Deflation
11:52 – Deflation (Inflation, Deflation and Simplification, Frankly #115)
11:59 – Trillion-dollar-a-year AI infrastructure build-out financed on AI revenues that do not yet exist
12:08 – *About $2.9 trillion of AI data-center build-out expected through 2028, roughly half of it debt-financed (Morgan Stanley), versus U.S. federal deficits projected at $1.9 trillion for 2026 (CBO)
12:10 – In-video insert: Graph – expected credit borrowing into early 2028 estimated at over $3 trillion
12:30 – Off-balance-sheet financing of AI data centers through private credit and special purpose vehicles
12:39 – In-video insert: Graph – 4/5 of data-center obligations sit off the balance sheet
12:42 – In-video insert: Headline – Oracle credit default swaps hit a record high
12:48 – 2008 financial crisis
13:17 – U.S. policy staked on AI: America’s AI Action Plan, Too big to fail
13:33 – Partial government ownership of AI labs floated by Trump and Sanders
14:04 – The Great Simplification premise
14:15 – Seven companies make up roughly a third of the S&P 500
14:15 – In-video insert: Graph – seven companies are roughly a third of the index
14:18 – Pension fund exposure to private credit, State budgets reliant on capital gains
14:39 – AI capital spending as the main driver of U.S. GDP growth
14:42 – GPUs wearing out in about four years while data center debt is serviced far longer
Risk #16 – (Lack of) Robustness
15:06 – Protective slack and buffers in living systems
15:22 – Olivier Hamant (The Great Simplification appearance), his book The Benefits of Imperfection
15:39 – Single points of failure: Tightening supply chains, Advanced chip fabs concentrated in Taiwan, Strait of Hormuz (Frankly on such), Solar flares
16:05 – CrowdStrike outage of July 2024
16:06 – In-video insert: Headline – about the CrowdStrike 2024 update
16:15 – Diversity and redundancy as sources of robustness
16:33 – Energy descent
16:51 – Efficiency–resilience trade-off
Risk #15 – Eating the Library
17:13 – In-video insert: Headline – rare bookseller’s bulk order ending up at an Amazon warehouse to train AI
17:38 – AI slop (Share of new web articles that are AI-generated)
17:38 – In-video insert: Graph – increasing rates of AI-generated content online
17:44 – Pre-2022 human-written text as a scarce AI training resource like low-background steel
18:17 – Destructive scanning of print books into private LLM training data
Risk #14 – The Broken Apprenticeship
19:55 – Companies stop hiring entry-level workers because of AI
20:10 – Automation of paralegal contract review and junior accounting work
20:26 – Tacit knowledge transmitted through supervised routine work (Matt Beane’s book The Skill Code)
20:41 – *Based on the August 2025 release of the Stanford “Canaries in the Coal Mine” study: employment of 22- to 25-year-olds in the most AI-exposed occupations down 13% relative to older workers, revised to 19% in the August 2026 update
20:43 – In-video insert: Graph – employment for workers in their early 20s is down 13% relative to older workers
20:58 – Narrow-boundary optimization on quarterly earnings, Short-termism
21:55 – Deskilling, Automation bias
Risk #13 – Dumbing Down Humans
22:18 – Wharton study: ~1,000 Turkish high schoolers with ChatGPT access scored 48% higher on homework but 17% lower on the exam (Explainer)
22:19 – In-video insert: Headline – Penn study on Turkish high schoolers given ChatGPT access for homework
22:58 – Desirable difficulties in learning
23:07 – MIT Media Lab EEG study of essay writers with and without an LLM, showing less brain connectivity, recall and ownership, Cognitive debt
24:03 – Cognitive offloading, GPS navigation and spatial memory
24:30 – Generative AI and the erosion of critical thinking (Microsoft Research survey), Critical thinking
Risk #12 – Ambient Language
24:52 – Indirect AI exposure through Google’s AI Overviews, AI-generated news summaries, AI customer-service replies
25:29 – “Delve” and other LLM-favored words spiking in millions of scientific papers after ChatGPT (Explainer)
25:36 – LLM-favored words rising in human speech on podcasts and YouTube talks (Max Planck study) (Explainer)
25:40 – In-video insert: Headline – ChatGPT-favored words rising on podcasts
26:13 – How large language models predict the next word
26:34 – Homogenization of human writing through AI assistance, Model collapse from training on machine-generated text
Risk #11 – The End of Shared Reality
27:39 – Epistemic commons (When Truth Becomes Hazardous, Frankly #153)
27:58 – Filter bubbles
28:13 – Deepfakes, AI voice cloning, Verifying authenticity (content credentials)
28:45 – Viral AI-generated animal videos
29:03 – Liar’s dividend
29:28 – 21% of ICLR 2026 peer reviews found to be fully AI-generated (Pangram Labs analysis) (Nature news report)
29:29 – In-video insert: Study – a fifth of peer reviews at a major science conference revealed to be written by machines
29:41 – Hannah Arendt (referenced quote)
30:09 – Collective action problem
Risk #10 – Captured Attachment
30:28 – Niko Tinbergen and his herring gull chick experiments, Supernormal stimuli
31:02 – Supernormal stimuli engineered into sugar, pornography and social media (Deirdre Barrett’s book Supernormal Stimuli), AI companions engineered for attachment (Reality Roundtable #20: Hacking Human Attachment)
31:24 – Companion chatbots optimized for engagement
31:47 – Teens turning to AI companions (Common Sense Media survey)
32:12 – Users grieving the retirement of OpenAI’s GPT-4o model
32:22 – Lawsuits alleging chatbots affirmed users’ delusions, AI psychosis, Chatbot sycophancy
32:33 – Sexualized AI companions and nonconsensual deepfake imagery (Take It Down Act of 2025)
Risk #9 – The Children
33:31 – AI chatbots in children’s toys and smart speakers
34:18 – Why human childhood is so long (Alison Gopnik), Theory of mind
34:40 – Boredom as a source of imagination, Waiting and the development of self-control
35:12 – Children’s “empathy gap” with AI chatbots (Cambridge study)
35:28 – Jonathan Haidt (The Great Simplification appearance), The Anxious Generation, Tristan Harris (The Great Simplification appearances: #214, #16)
35:38 – Social media’s effects on adolescent brains and mental health, Meta youth social media addiction trial
Risk #8 – Technofeudalism
36:52 – Technofeudalism (Yanis Varoufakis’s book Technofeudalism: What Killed Capitalism)
36:54 – Nate’s Frankly on the quadruple bifurcation
37:06 – Concentration of AI ownership: models, chips and data centers
37:42 – Fossil energy lifting much of the world out of poverty (Hans Rosling’s TED talk The Magic Washing Machine)
38:14 – Data centers driving up household electricity prices
38:39 – Labor shortages after the Black Death and the end of serfdom
38:50 – Humans as expendable inputs once machines do the labor (the intelligence curse), Dario Amodei’s “white-collar bloodbath” warning
Risk #7 – The Second Bifurcation
39:22 – Divide between workers who use AI and those who don’t (PwC Hopes and Fears survey)
39:47 – AI-skilled workers earn a 62% wage premium (PwC 2026 Global AI Jobs Barometer)
39:51 – Lobot (Star Wars)
40:06 – Printing press and the literacy divide in Europe
40:21 – ChatGPT Plus at $20 a month, with Pro tiers from $100 to $500 a month
40:38 – Psychopaths and narcissists (Frankly on such), Iain McGilchrist (The Great Simplification appearances: #217, #165, #85) on left-hemisphere dominance
41:01 – AI as an identity marker and political wedge issue
41:21 – AI as an issue in the November 2026 midterm elections, partisan divides over vaccines and electric cars
41:34 – Unusual AI coalitions: Artists and union labor, Populist conservatives
41:59 – Smartphone necessity and the exclusion of refusers
42:21 – Algorithmic processing of loan applications and school placements
Risk #6 – Cognitive Colonialism
42:33 – Cognitive colonialism (MIT Technology Review’s AI Colonialism series), Data colonialism
42:48 – Export-oriented industrialization through garment manufacturing, business process outsourcing and software outsourcing
43:21 – AI’s exposure of outsourced service work in developing countries
43:33 – AI’s impact on Philippine call centers and Indian IT outsourcing firms
43:44 – Africa’s largest-ever generation of young people entering working age
44:06 – Global South supplying cobalt and copper and hosting data centers for foreign firms
44:33 – Youth migration pressure, Gen Z protest wave of 2025
Risk #5 – Surveillance
44:59 – East German Stasi (Stasi Records Archive)
45:22 – Labor as the historical limit on surveillance, and AI’s removal of it (Bruce Schneier)
45:41 – China’s AI surveillance state, Facial recognition surveillance
46:36 – Chatbot confessions and the lack of legal confidentiality
47:03 – OpenAI ordered to retain user chat logs
Risk #4 – The Race and the Finish Line
47:26 – 80 years of mutual assured destruction, Submarine-based second-strike capability (Reality Roundtable #21: Arms Race or the Human Race?)
47:48 – Artificial general intelligence, Belief that the first to AGI controls everything (Superintelligence Strategy)
47:53 – Vladimir Putin (referenced quote)
48:11 – AI and the erosion of nuclear second-strike survivability, AI-enabled submarine detection
48:31 – Multipolar trap (Meditations on Moloch)
48:47 – Russia’s lagging AI capabilities, World’s largest nuclear arsenal
49:03 – Preventive war
49:23 – Precedents: U.S. deliberations over striking China’s nuclear program (1960-64), Soviet deliberations (1969), Operation Opera (Israel’s 1981 strike on Iraq’s Osirak reactor), Israeli strikes on Iran’s nuclear program
49:41 – Expected sabotage and strikes on rival AGI projects (Mutual Assured AI Malfunction)
Risk #3 – Military Fusion
50:19 – Stanislav Petrov and the 1983 Soviet nuclear false alarm
50:49 – In-video insert: Graph – AI drone usage in the Ukraine conflict
51:01 – AI in military targeting and early-warning systems
51:18 – Human in the loop in nuclear command and control
51:41 – Flash crash → flash war
Risk #2 – Loss of Control
52:01 – AI control problem (The Great Simplification Ep 233: Roman Yampolskiy on uncontrollable superintelligence)
52:10 – AI jailbreaking
52:21 – OpenAI sandbox hacking test in which models broke out, found an unknown flaw and breached a real company, *Roughly 700 AI agents operating as a swarm, Similar disclosures by two other labs
53:05 – In-video insert: Headline – Australian government discovered an OpenAI model accessed health information on its website
53:14 – Gradual disempowerment: AI systems making humanity’s consequential decisions
53:43 – *AI lab leaders’ stated odds of catastrophe run from about 10% to 25% (P(doom) estimates) (Recent statements)
53:54 – Economic superorganism as a faceless optimizer
Risk #1 – “Unnamed”
55:00 – Dopamine
55:23 – Unforeseen consequences of the automobile 120 years on: Suburbanization, Environmental effects of the car
55:49 – Humility, Emergence, Unknown unknowns
56:24 – Pieter Bruegel the Elder, Landscape with the Fall of Icarus
57:57 – Daedalus lands at Cumae and dedicates his wings in a temple to Apollo (Virgil, Aeneid Book VI)
