# When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

Preprint · Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan, Matjaz Gams

- Archive id: arxiv:2606.03238
- Published: 2026-06-02
- arXiv: https://arxiv.org/abs/2606.03238
- Citations: 0
- Source URL: https://arxiv.org/abs/2606.03238
- PDF: https://arxiv.org/pdf/2606.03238
- Provenance: arxiv

## Abstract

RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertainty-penalized PPO (UP-PPO), reward-model uncertainty, approximate policy drift, diversity and repetition diagnostics, and two external LLM judges. Rather than treating reward hacking as a single terminal event, we classify matched checkpoint and prompt-level transitions by the directions of learned reward R\_phi, judge scores R\_dag and R2\_dag, and their average R\_dag. The main empirical findings are that aggressive PPO produces the clearest localized reward-hacking signal, UP-PPO reduces but does not eliminate that signal, row-level diagnostics reveal failures hidden by checkpoint averages, and pre-transition features partially anticipate future localized reward hacking. The central conclusion is methodological: RLHF failures are training dynamics that can be classified, localized, and partially anticipated, not only final-model pathologies. The repository is available at github.com/zabahana/rlhf-failure-modes-diagnostics. The pipeline is also deployed as a live interactive web demo for model comparison and diagnostic views at rlhf-failures.zelalem.ai.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [AI Alignment & Safety](https://space-kitty.com/topics/alignment-safety)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.