# Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedback

Preprint · Huiying Zhong, Tianwei Gao, Zhiwei Steven Wu, Linjun Zhang, Weijie J. Su, Zhun Deng

- Archive id: arxiv:2403.05006
- Published: 2024-03-08
- arXiv: https://arxiv.org/abs/2403.05006
- Source URL: https://arxiv.org/abs/2403.05006
- PDF: https://arxiv.org/pdf/2403.05006
- Provenance: arxiv

## Abstract

Pluralistic alignment requires learning from feedback that reflects persistent and potentially conflicting stakeholder preferences while ultimately selecting a single collective policy. We study this problem in offline reinforcement learning from human feedback (RLHF), where the party associated with each comparison is observed. Under a shared low-rank linear reward model, we jointly estimate party-specific rewards and perform pessimistic policy optimization under Nash, Utilitarian, and Egalitarian social-welfare objectives. We establish nonasymptotic bounds for party-specific reward estimation and the resulting policy suboptimality under offline coverage conditions. We further consider general pairwise preferences that need not admit a scalar reward representation and may exhibit cycles. In this setting, we construct a pessimistic von Neumann winner policy and derive corresponding performance guarantees. Under these models, our results provide a unified finite-sample solution to a central challenge in pluralistic alignment: learning from limited, heterogeneous, and potentially cyclic feedback, and producing a single policy with explicit collective-welfare guarantees. Our framework thereby makes preference aggregation an explicit and statistically analyzable design choice rather than an implicit consequence of pooling human feedback.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.