Definition
p(doom) is a subjective probability metric used in the AI community to estimate the likelihood that artificial intelligence will eventually cause a catastrophic outcome for humanity (e.g., extinction or irreversible collapse). It serves as a benchmark for individual scientists and leaders to express their level of concern regarding runaway superintelligence and the AI Alignment Problem.
Why It Matters
p(doom) isn’t just a number; it’s the “existential barometer” of our time. It dictates where billions of dollars in research funding flow and what regulations govern the most powerful technology in history. If we ignore this metric because it lacks “empirical data,” we fall into the trap of the turkey before Thanksgiving—assuming safety because of historical stability, right up until the moment of irreversible collapse. It is the metric of “Epistemic Humility” in the face of superintelligence.
Core Concepts
- Community Split (2024 Estimates):
- Yann LeCun: p(doom) = 0. He believes AI will remain subservient tools with no intrinsic goal to dominate.
- Yoshua Bengio: p(doom) = 0.5 (50%). He views runaway AI as an existential threat comparable to nuclear war or a lethal synthetic pathogen.
- Geoffrey Hinton: p(doom) = 0.1 to 0.2 (10-20%). He recalibrated his estimate downward after meeting with others (including Huang), but remains profoundly worried.
- Jensen Huang: p(doom) < 0 (Functionally zero). He dismisses the question as “stupid” and unsupported by empirical data.
- Parametric Uncertainty: The wide range of estimates reflects the lack of data and the divergent “mental models” of how AI will evolve.
- Conversion Moment: Many doomers describe a “conversion” moment where probing the capabilities of LLMs (like ChatGPT) triggered a profound emotional and rational “Fear.”
- The Data Absence Critique: Bengio argues that waiting for “data” on human extinction is a logical fallacy, as the event only has to happen once for it to be fatal.
- Policy Driver: p(doom) estimates drive support for (or opposition to) state and federal regulations like California’s SB 1047.