# The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization

Preprint · Sian Gooding, Hassan Mansoor

- Archive id: arxiv:2311.04919
- Published: 2023-11-02
- arXiv: https://arxiv.org/abs/2311.04919
- Source URL: https://arxiv.org/abs/2311.04919
- PDF: https://arxiv.org/pdf/2311.04919
- Provenance: arxiv

## Abstract

Reinforcement Learning from Human Feedback (RLHF) can be used to capture complex and nuanced properties of text generation quality. As a result, the task of text summarization has been identified as a good candidate for this process. In this paper, we explore how preference agreement impacts the efficacy of RLHF for summarization. We show that sampling human preferences to include a range of annotator agreement results in (1) higher accuracy reward models and (2) alters the characteristics of quality captured. We additionally show improvements in downstream generation when using a reward model trained with a range of preference agreements. Our contributions have implications for the design of synthetic datasets as well as the importance of considering quality differentials in comparison-based data.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.