# Ranking Reasoning LLMs under Test-Time Scaling

Preprint · Mohsen Hariri, Michael Hinczewski, Jing Ma, Vipin Chaudhary

- Archive id: arxiv:2603.10960
- Published: 2026-03-11
- Venue: The 64th Annual Meeting of the Association for Computational Linguistics (ACL), 2026
- arXiv: https://arxiv.org/abs/2603.10960
- Citations: 0
- Source URL: https://arxiv.org/abs/2603.10960
- PDF: https://arxiv.org/pdf/2603.10960
- Provenance: arxiv

## Abstract

Test-time scaling evaluates reasoning LLMs by sampling multiple outputs per prompt, but ranking models in this regime remains underexplored. We formalize dense benchmark ranking under test-time scaling and introduce Scorio, a library that implements statistical ranking methods such as paired-comparison models, item response theory (IRT) models, voting rules, and graph- and spectral-based methods. Across $20$ reasoning models on four Olympiad-style math benchmarks (AIME'24, AIME'25, HMMT'25, and BrUMO'25; up to $N=80$ trials), most full-trial rankings agree closely with the Bayesian gold standard $\\mathrm{Bayes}\_{\\mathcal{U}}@80$ (mean Kendall's $τ\_b = 0.93$--$0.95$), and $19$--$34$ methods recover exactly the same ordering. In the single-trial regime, the best methods reach $τ\_b \\approx 0.86$. Using greedy decoding as an empirical prior ($\\mathrm{Bayes}\_{\\mathbf{R}\_0}@N$) reduces variance at $N=1$ by $16$--$52\\%$, but can bias rankings when greedy and stochastic sampling disagree. These results identify reliable ranking methods for both high- and low-budget test-time scaling. We release Scorio as an open-source library at https://github.com/mohsenhariri/scorio.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Reasoning Models & Test-Time Compute](https://space-kitty.com/topics/reasoning-models)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.