# A Sign Law for Entropy Drift under Sampler Mismatch in RLVR

Preprint · Zhiyuan Wang, Jacob Steeves, Chang Liu

- Archive id: arxiv:2610.34044
- Published: 2026-10-25
- arXiv: https://arxiv.org/abs/2610.34044
- Citations: 0
- Source URL: https://arxiv.org/abs/2610.34044
- PDF: https://arxiv.org/pdf/2610.34044
- Provenance: arxiv

## Abstract

Entropy in reinforcement learning with verifiable rewards (RLVR) moves up or down depending on choices that should be neutral, such as whether over-length responses are masked or counted as failures. We trace this to sampler mismatch, where rollouts are drawn with top-k/top-p truncation or at a temperature the learner does not apply, and prove that a constant advantage shift then changes the entropy of a softmax policy, at first order and in expectation, with a sign fixed by the sampler: positive shifts sharpen the policy under any sampler whose likelihood ratio to the policy increases with token probability, including top-k, top-p, min-p, and temperatures below one, and flatten it under flatter samplers. In group-based RLVR with token-level losses, the baseline-controlled part of this drift is set by the token-weighted net advantage, which masking and failing over-length responses can push to opposite signs; the remainder, a covariance between advantages and the sampler's sharpness gap, can feed back over long horizons. On a 2B model, the entropy slope is linear in shifts of the net advantage under truncated sampling and indistinguishable from flat under exact sampling, and token-level importance ratios leave 98% of the drift. In 100-step runs, a net-zero baseline only delays an entropy explosion that exact sampling does not show, a fixed positive shift delays it further, and a baseline weighted by the sharpness gap keeps entropy stable without changing the learner. On a 9B reasoning model over ten steps, the uncorrected rule loses held-out accuracy, whereas zeroing the net advantage or renormalizing the learner gains 8.2 and 8.8 points over it, with no held-out effect under exact sampling.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Reasoning Models & Test-Time Compute](https://space-kitty.com/topics/reasoning-models)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.