# Scaling Laws & Compute-Optimal Training: State of the Field (October 2026)

Research dossier · v1 · updated 2026-10-30 · 4,904 words · 59 sources · 30 claims · verification: verified

An expert briefing on neural scaling laws as of late October 2026: compute-optimal pretraining and the Kaplan-Chinchilla reconciliation, inference-aware overtraining, data-constrained and synthetic-data scaling, emergent abilities and downstream predictability, and the newer laws for reinforcement learning and test-time compute. The main takeaway is that pretraining loss stays highly predictable across many orders of magnitude, while the practical optimum has moved well away from Chinchilla's roughly 20 tokens per parameter, mainly because of inference cost and data scarcity, and downstream capabilities remain only partly predictable.

## Contents

- Executive summary
- Background and core concepts
- Timeline of key milestones
- State of the art
  - Compute-optimal pretraining
  - Inference-aware training and overtraining
  - Data-constrained, repeated and synthetic data
  - Architecture, optimizer and precision as scaling variables
  - From loss to capabilities: emergence and downstream prediction
  - Post-training, RL and test-time compute
  - Quantitative benchmarks
- Open problems and active debates
- Major programs and players
- Outlook: what to watch (next 12-18 months)
- Key papers
- Glossary

## Sample findings

- Cross-entropy loss follows power laws in parameters, data and compute, and some trends extend over more than seven orders of magnitude.
- Chinchilla, a 70B-parameter model trained on about 1.4T tokens at Gopher's compute budget, outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a broad set of downstream tasks.

## Get the full dossier

GET https://space-kitty.com/api/v1/dossiers/scaling-laws costs $0.20 via x402 (USDC on Base Sepolia (testnet)). Send `Accept: text/markdown` for Markdown or `application/json` for structured data (claims with citations, timeline, benchmarks, open problems, glossary, bibliography).
---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.