# A Unifying Lens on Reward Uncertainty in RLHF

Preprint · Ely Hahami, Yoel Zimmermann, Ray Zhou, Jack Benarroch Jedlicki

- Archive id: arxiv:2606.09073
- Published: 2026-06-08
- arXiv: https://arxiv.org/abs/2606.09073
- Citations: 0
- Source URL: https://arxiv.org/abs/2606.09073
- PDF: https://arxiv.org/pdf/2606.09073
- Provenance: arxiv

## Abstract

Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural mitigation is pessimism: lowering rewards in regions where the RM is uncertain. However, standard scalar RMs provide no principled notion of uncertainty. We argue that the right object is a distributional reward model $p(r\\mid x,y)$. Under either a Bayesian inference or a KL-distributionally robust optimization (KL-DRO) lens, the KL-regularized RLHF objective admits a closed-form effective reward $\\tilde r(x,y) = \\pmβ\\log\\mathbb{E}\_p\[e^{\\pm r/β}\]$. The pessimistic branch unifies the prior heuristics for RM ensemble aggregation: mean aggregation, worst-case optimization (WCO), and uncertainty-weighted optimization (UWO) all emerge as limits or truncations of this single expression. This also clarifies the implicit assumptions of each existing rule.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [AI Alignment & Safety](https://space-kitty.com/topics/alignment-safety)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.