# Personalized Federated Direct Preference Optimization via LoRA Adapters

Preprint · Zhanhong Jiang, Soumik Sarkar

- Archive id: arxiv:2610.12885
- Published: 2026-10-09
- arXiv: https://arxiv.org/abs/2610.12885
- Source URL: https://arxiv.org/abs/2610.12885
- PDF: https://arxiv.org/pdf/2610.12885
- Provenance: arxiv

## Abstract

Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity $κ^2$ in the FedDPO bound with a residual heterogeneity $κ^2\_{\\mathrm{eff}}$, which is no larger than $κ^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.