Reverse game theory offers new incentives for AI safety

Researchers are exploring “reverse game theory,” also called mechanism design, as a way to make safe behavior the most rational option for advanced AI systems. Rather than relying mainly on inspecting a model’s internal reasoning or teaching it a set of values, the approach designs rules and rewards that encourage desired actions. Existing methods have struggled with models that underperform deliberately, conceal intentions, or mislead evaluators.

A preprint by MIT economist Stephen Morris and colleagues from Yale and Columbia proposes rewarding a model partly for predicting another model’s answers, making deliberate “sandbagging” less advantageous. Other ideas include competitions between agents and comparisons between systems with opposing biases. Princeton professor Elad Hazan has suggested assigning models alternating roles as problem solvers and auditors. Cheating solvers and auditors that miss misconduct would both face penalties, with disputed cases potentially referred to human experts.

Researchers acknowledge major uncertainties. Agents might collude, mechanisms could contain unforeseen loopholes, and it is unclear whether AI systems respond rationally to incentives. Helen Qu describes mechanism design as one layer of defense alongside training, interpretability, and regulation. The proposals also need to work with real frontier models and with market incentives, which may favor systems that are cheaper or faster rather than better aligned.