Definition
Inverse Reinforcement Learning (IRL) is a machine learning approach where an agent attempts to infer the underlying reward function (goals) of another agent (e.g., a human) by observing its behavior. While traditional RL tries to find the best behavior for a given reward, IRL tries to find the best reward that explains a given behavior.
Why It Matters
We are terrible at explaining our own values. IRL is the path to building AI that doesn’t just “obey orders” (which can be catastrophic), but instead studies what we do to figure out what we actually care about, ensuring deeper alignment.
Core Concepts
- Goal Inference: Instead of being told what to do, the AI watches what humans do and works backward to figure out why they do it.
- Modeling Preferences: By observing thousands of human decisions in diverse contexts, the AI builds an accurate model of complex, unstated human preferences.
- Incentive for Cautiousness: A core idea is that the AI attempts to maximize the human’s goal-satisfaction, not its own. If it is unclear about a goal, its safest strategy is to be cautious and seek clarification.
- Switch-off Permission: An IRL-based AI should logically allow its owner to switch it off, as a shutdown signal is strong evidence that the AI has misunderstood the owner’s true goals.