# From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Models

Preprint · Shubhra Mishra, Gabriel Poesia, Noah D. Goodman

- Archive id: arxiv:2407.00900
- Published: 2024-07-01
- Venue: Conference on Language Modeling (COLM), Montreal, Canada, October 7-10, 2025
- arXiv: https://arxiv.org/abs/2407.00900
- Source URL: https://arxiv.org/abs/2407.00900
- PDF: https://arxiv.org/pdf/2407.00900
- Provenance: arxiv

## Abstract

Large Language Models (LLMs) solely trained on next-token prediction learn to solve a wide range of problems involving mathematical reasoning. But how does this ability evolve during training? We show the first analysis of how mathematical reasoning abilities of several open-weight LLMs develop during pre-training and post-training. To this end, we construct MathCAMPS, a synthetic dataset of novel mathematical reasoning problems grounded in 44 fine-grained skills taken from the Common Core curriculum from K to 8th grades. In one experiment, we show that mathematical skills are learned during pre-training in an order that measurably correlates with the human-designed curriculum, even though training data are randomly ordered. We also show a detailed analysis of which mathematical abilities benefit from instruction tuning, a widely used post-training method and, in contrast, which skills suffer. Our work paves the way for an empirical understanding of LLM training dynamics in relation to reasoning.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.