# Energy-Weighted Flow Matching for Offline Reinforcement Learning

Preprint · Shiyuan Zhang, Weitong Zhang, Quanquan Gu

- Archive id: arxiv:2503.04975
- Published: 2025-03-06
- arXiv: https://arxiv.org/abs/2503.04975
- Source URL: https://arxiv.org/abs/2503.04975
- PDF: https://arxiv.org/pdf/2503.04975
- Provenance: arxiv

## Abstract

This paper investigates energy guidance in generative modeling, where the target distribution is defined as $q(\\mathbf x) \\propto p(\\mathbf x)\\exp(-β\\mathcal E(\\mathbf x))$, with $p(\\mathbf x)$ being the data distribution and $\\mathcal E(\\mathcal x)$ as the energy function. To comply with energy guidance, existing methods often require auxiliary procedures to learn intermediate guidance during the diffusion process. To overcome this limitation, we explore energy-guided flow matching, a generalized form of the diffusion process. We introduce energy-weighted flow matching (EFM), a method that directly learns the energy-guided flow without the need for auxiliary models. Theoretical analysis shows that energy-weighted flow matching accurately captures the guided flow. Additionally, we extend this methodology to energy-weighted diffusion models and apply it to offline reinforcement learning (RL) by proposing the Q-weighted Iterative Policy Optimization (QIPO). Empirically, we demonstrate that the proposed QIPO algorithm improves performance in offline RL tasks. Notably, our algorithm is the first energy-guided diffusion model that operates independently of auxiliary models and the first exact energy-guided flow matching model in the literature.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.