# The Crucial Role of Samplers in Online Direct Preference Optimization

Preprint · Ruizhe Shi, Runlong Zhou, Simon S. Du

- Archive id: arxiv:2409.19605
- Published: 2024-09-29
- arXiv: https://arxiv.org/abs/2409.19605
- Source URL: https://arxiv.org/abs/2409.19605
- PDF: https://arxiv.org/pdf/2409.19605
- Provenance: arxiv

## Abstract

Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $\\textbf{linear}$ convergence, while our proposed online sampler achieves $\\textbf{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over $7.4$% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.