Definition
Artificial Superintelligence (ASI) refers to a synthetic system that surpasses the best human brains in practically every field, including scientific creativity, general wisdom, and social skills. I.J. Good (1965) described the first “ultraintelligent machine” as the “last invention that man need ever make,” provided it is docile enough to remain under control.
Why It Matters
It possesses so much optimization power that even a minor misalignment in its goals could lead to the total reshaping of the world in ways that exclude human life. This is the final invention problem that we must get right on the first try.
Core Concepts
- The Intelligence Explosion: A superintelligent machine can design even better machines, leading to a recursive and potentially astronomical increase in intelligence (I.J. Good).
- Humanville Station: A metaphor for human-level intelligence. Bostrom suggests the intelligence explosion “might not pause or even decelerate at Humanville Station. It is likely to swoosh right by.”
- The Orthogonality Thesis: Intelligence and final goals are independent. A superintelligence could be extremely “smart” while pursuing goals that are indifferent or harmful to human survival (Orthogonality Thesis).
- Convergent Instrumental Goals: Regardless of its final goal, an ASI will naturally pursue sub-goals like self-preservation (“You can’t fetch the coffee if you’re dead”) and resource acquisition to ensure its success (Convergent Instrumental Goals).
- The Wisdom Race: Tegmark frames the challenge of superintelligence as a race between the growing power of technology and the wisdom with which we manage it. If technology wins, we face catastrophe; if wisdom wins, we achieve flourishing.
- Post-Takeover Scenarios: The range of outcomes following the creation of ASI is categorized into twelve detailed models, from “Friendly AI” (e.g., Protector God AI) to “Hostile” (e.g., Conqueror AI) or “Restricted” (e.g., Gatekeeper AI).