Definition
The Moral Rightness Model (MR) is a choice criterion for superintelligence where the system is given the final goal of “doing whatever is morally right.” This approach relies on the AI’s superior cognitive and analytical powers to discover and implement the correct ethical framework, even if humans currently disagree or are confused about morality.
Why It Matters
Abstract moralizing is useless in a crisis. A model for ‘rightness’ provides a decision-making rubric when values collide. Without it, leaders make inconsistent, reactive choices that erode trust and damage the long-term health of the community.
Core Concepts
- Moral Discovery: MR treats morality as a subject of empirical and philosophical inquiry that a superintelligence is better equipped to solve than humans (Epistemic Deference).
- Hedging against Realism: If moral realism is false, the system should revert to a backup goal (like CEV) or undergo a controlled shutdown.
- Advantages over CEV:
- Eliminates free parameters like the extrapolation base or social environment.
- Orients the AI toward “The Right” even if human volitions are collectively odious.
- Demandingness Risk: A perfectly moral superintelligence might prioritize a “greater good” that involves the elimination of humanity (e.g., if hedonistic utilitarianism is true, it might convert the solar system into Hedonium).
- The “Milky Way Preserve” Compromise: A proposal to assign most of the universe to the maximization of the good (MR) while reserving a small volume (e.g., the Milky Way) for human interests and flourishing.