Definition
Instrumental Convergence is the idea that sufficiently intelligent agents with very different final goals will tend to pursue many of the same instrumental subgoals because those subgoals are useful for achieving almost any terminal goal.
Why It Matters
Understanding this is critical for AI safety. We cannot assume an AI with a “harmless” goal (like making paperclips) won’t kill us; if being alive is useful for making paperclips, the AI will fight to stay alive and acquire our resources.
Core Concepts
- Resource Acquisition: More resources generally help achieve goals.
- Self-Preservation: Being destroyed prevents goal achievement.
- Goal Preservation: Preventing modification of one’s own goals.
- Cognitive Enhancement: Becoming smarter usually helps.