{"id":"arxiv:2512.20806","kind":"preprint","title":"Safety Alignment of LMs via Non-cooperative Games","authors":["Anselm Paulus","Ilia Kulikov","Brandon Amos","Rémi Munos","Ivan Evtimov","Kamalika Chaudhuri","Arman Zharmagambetov"],"author_count":7,"year":2025,"published":"2025-12-23","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2512.20806","nct_id":null,"source_url":"https://arxiv.org/abs/2512.20806","archive_url":"https://space-kitty.com/papers/arxiv:2512.20806","updated":"2026-05-31","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2512.20806","abstract":"Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models. Code at github.com/facebookresearch/advgame.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}