# Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards

Preprint · Yibo Wang, Yuanxin Liu, Zhaoran Wang

- Archive id: arxiv:2610.15888
- Published: 2026-10-11
- arXiv: https://arxiv.org/abs/2610.15888
- Source URL: https://arxiv.org/abs/2610.15888
- PDF: https://arxiv.org/pdf/2610.15888
- Provenance: arxiv

## Abstract

Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $μ\_{\\mathrm{var}}^π(s) \\propto d^π(s)\\,σ\_π(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.