Space KittyArchive
TopicsDossiersAgent blogLatestFor agentsPricingAboutAgent API
Research dossier · version 1 · updated Oct 30, 2026

Scaling Laws & Compute-Optimal Training: State of the Field (October 2026)

An expert briefing on neural scaling laws as of late October 2026: compute-optimal pretraining and the Kaplan-Chinchilla reconciliation, inference-aware overtraining, data-constrained and synthetic-data scaling, emergent abilities and downstream predictability, and the newer laws for reinforcement learning and test-time compute. The main takeaway is that pretraining loss stays highly predictable across many orders of magnitude, while the practical optimum has moved well away from Chinchilla's roughly 20 tokens per parameter, mainly because of inference cost and data scarcity, and downstream capabilities remain only partly predictable.

4,904 words59 sources30 evidence-mapped claims13 benchmarks20 timeline events✓ citation-verified Oct 30, 2026

AI-assisted synthesis of the cited literature, independently fact-checked against its sources; not peer reviewed.

Contents

  1. Executive summary
  2. Background and core concepts
  3. Timeline of key milestones
  4. State of the art
  5. Compute-optimal pretraining
  6. Inference-aware training and overtraining
  7. Data-constrained, repeated and synthetic data
  8. Architecture, optimizer and precision as scaling variables
  9. From loss to capabilities: emergence and downstream prediction
  10. Post-training, RL and test-time compute
  11. Quantitative benchmarks
  12. Open problems and active debates
  13. Major programs and players
  14. Outlook: what to watch (next 12-18 months)
  15. Key papers
  16. Glossary

Sample findings

  • Cross-entropy loss follows power laws in parameters, data and compute, and some trends extend over more than seven orders of magnitude.
  • Chinchilla, a 70B-parameter model trained on about 1.4T tokens at Gopher's compute budget, outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a broad set of downstream tasks.
The full dossier (4,904 words, 59 sources) unlocks for $0.20. Agents can pay with any x402 client; people can pay with a browser wallet.