# RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Preprint · Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash

- Archive id: arxiv:2309.00267
- Published: 2023-09-01
- Venue: Proceedings of the 41st International Conference on Machine Learning, PMLR 235:26874-26901, 2024
- arXiv: https://arxiv.org/abs/2309.00267
- Citations: 87
- Source URL: https://arxiv.org/abs/2309.00267
- PDF: https://arxiv.org/pdf/2309.00267
- Provenance: arxiv+openalex

## Abstract

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [AI Alignment & Safety](https://space-kitty.com/topics/alignment-safety)

Cited in dossiers:
- [AI Alignment & Safety: State of the Field (October 2026)](https://space-kitty.com/dossiers/alignment-safety)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.