Definition
An Existential Risk is one that threatens the premature extinction of Earth-originating intelligent life or the permanent and drastic destruction of its potential for desirable future development. Unlike “terminal” risks (like a single death), existential risks are global and irreversible.
Why It Matters
Existential risks are unique because we cannot learn from them through trial and error—we won’t survive the first mistake. Identifying these “global and irreversible” threats is the “priority zero” for humanity, forcing us to move beyond teenage-level impulsivity toward a level of wisdom that exceeds our growing technological power.
Core Concepts
- The Trial-and-Error Fallacy: We cannot learn from existential errors because we won’t survive the first one. Reactive approaches (see what happens, limit damages) are unworkable.
- Ecophagy (Gray Goo): The theoretical risk that self-replicating nanobots or an ASI will consume the biosphere to repurpose its atoms for Computronium or other instrumental goals.
- Dual-Use Paradox: Technologies that offer the greatest benefits (AI, Nanotechnology, Biotech) also pose the greatest risks.
- The Intelligence Explosion: The unique risk that the “Final Invention” (ASI) will move so fast that human defense becomes impossible.
- Intelligence Race: Competition between nations (the “Cat Nation vs. Mouse Nation”) drives a “race to the bottom” in safety standards.
- The Wisdom Race: Tegmark frames the challenge of superintelligence as a race between the growing power of technology and the wisdom with which we manage it.
- The Imprudent Teenager: MacAskill’s metaphor for civilization: we have developed adult-level power (nuclear, AI) but still possess teenage-level impulsivity and lack of foresight.
- Unrecovered Collapse: The risk that a global catastrophe (short of extinction) destroys industrial technology and we fail to re-industrialize due to Fossil Fuel Depletion or demographic loss.
- Accidental Omnicide: The risk of human extinction caused by “collective stupidity” or game-theory traps (e.g., accidental nuclear war).
- Nuclear Winter: A full-scale nuclear war could cause 20-35°C cooling, destroying the global food chain.
- EMP Vulnerability: High-altitude detonations could paralyze power and water infrastructure, causing mass death in days.