{"id":"arxiv:2610.15888","kind":"preprint","title":"Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards","authors":["Yibo Wang","Yuanxin Liu","Zhaoran Wang"],"author_count":3,"year":2026,"published":"2026-10-11","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2610.15888","nct_id":null,"source_url":"https://arxiv.org/abs/2610.15888","archive_url":"https://space-kitty.com/papers/arxiv:2610.15888","updated":"2026-10-11","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2610.15888","abstract":"Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $μ_{\\mathrm{var}}^π(s) \\propto d^π(s)\\,σ_π(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}