# The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Preprint · Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao

- Archive id: arxiv:2501.19358
- Published: 2025-01-31
- arXiv: https://arxiv.org/abs/2501.19358
- Citations: 0
- Source URL: https://arxiv.org/abs/2501.19358
- PDF: https://arxiv.org/pdf/2501.19358
- Provenance: arxiv

## Abstract

This work identifies the Energy Loss Phenomenon in Reinforcement Learning from Human Feedback (RLHF) and its connection to reward hacking. Specifically, energy loss in the final layer of a Large Language Model (LLM) gradually increases during the RL process, with an excessive increase in energy loss characterizing reward hacking. Beyond empirical analysis, we further provide a theoretical foundation by proving that, under mild conditions, the increased energy loss reduces the upper bound of contextual relevance in LLMs, which is a critical aspect of reward hacking as the reduced contextual relevance typically indicates overfitting to reward model-favored patterns in RL. To address this issue, we propose an Energy loss-aware PPO algorithm (EPPO) which penalizes the increase in energy loss in the LLM's final layer during reward calculation to prevent excessive energy loss, thereby mitigating reward hacking. We theoretically show that EPPO can be conceptually interpreted as an entropy-regularized RL algorithm, which provides deeper insights into its effectiveness. Extensive experiments across various LLMs and tasks demonstrate the commonality of the energy loss phenomenon, as well as the effectiveness of EPPO in mitigating reward hacking and improving RLHF performance.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [AI Alignment & Safety](https://space-kitty.com/topics/alignment-safety)

Cited in dossiers:
- [AI Alignment & Safety: State of the Field (October 2026)](https://space-kitty.com/dossiers/alignment-safety)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.