{"slug":"scaling-laws","topic":"scaling-laws","title":"Scaling Laws & Compute-Optimal Training: State of the Field (October 2026)","summary":"An expert briefing on neural scaling laws as of late October 2026: compute-optimal pretraining and the Kaplan-Chinchilla reconciliation, inference-aware overtraining, data-constrained and synthetic-data scaling, emergent abilities and downstream predictability, and the newer laws for reinforcement learning and test-time compute. The main takeaway is that pretraining loss stays highly predictable across many orders of magnitude, while the practical optimum has moved well away from Chinchilla's roughly 20 tokens per parameter, mainly because of inference cost and data scarcity, and downstream capabilities remain only partly predictable.","version":1,"published_at":"2026-10-30","updated_at":"2026-10-30","stats":{"words":4904,"citations":59,"claims":30,"timeline":20,"benchmarks":13,"open_problems":8,"key_papers":15,"glossary":14},"verification":{"status":"verified","method":"independent AI fact-check against cited sources (evidence pack + web checks); automated citation resolution","claims_checked":129,"issues_fixed":24,"verified_at":"2026-10-30"},"ai_assisted":true,"price":{"usd":"0.20","protocol":"x402","network":"eip155:84532","asset":"USDC"},"url":"https://space-kitty.com/dossiers/scaling-laws","preview_url":"https://space-kitty.com/api/v1/dossiers/scaling-laws/preview","purchase_url":"https://space-kitty.com/api/v1/dossiers/scaling-laws","toc":[{"level":2,"text":"Executive summary","anchor":"executive-summary"},{"level":2,"text":"Background and core concepts","anchor":"background-and-core-concepts"},{"level":2,"text":"Timeline of key milestones","anchor":"timeline-of-key-milestones"},{"level":2,"text":"State of the art","anchor":"state-of-the-art"},{"level":3,"text":"Compute-optimal pretraining","anchor":"compute-optimal-pretraining"},{"level":3,"text":"Inference-aware training and overtraining","anchor":"inference-aware-training-and-overtraining"},{"level":3,"text":"Data-constrained, repeated and synthetic data","anchor":"data-constrained-repeated-and-synthetic-data"},{"level":3,"text":"Architecture, optimizer and precision as scaling variables","anchor":"architecture-optimizer-and-precision-as-scaling-variables"},{"level":3,"text":"From loss to capabilities: emergence and downstream prediction","anchor":"from-loss-to-capabilities-emergence-and-downstream-prediction"},{"level":3,"text":"Post-training, RL and test-time compute","anchor":"post-training-rl-and-test-time-compute"},{"level":3,"text":"Quantitative benchmarks","anchor":"quantitative-benchmarks"},{"level":2,"text":"Open problems and active debates","anchor":"open-problems-and-active-debates"},{"level":2,"text":"Major programs and players","anchor":"major-programs-and-players"},{"level":2,"text":"Outlook: what to watch (next 12-18 months)","anchor":"outlook-what-to-watch-next-12-18-months"},{"level":2,"text":"Key papers","anchor":"key-papers"},{"level":2,"text":"Glossary","anchor":"glossary"}],"sample_claims":["Cross-entropy loss follows power laws in parameters, data and compute, and some trends extend over more than seven orders of magnitude.","Chinchilla, a 70B-parameter model trained on about 1.4T tokens at Gopher's compute budget, outperformed Gopher, GPT-3, Jurassic-1 and Megatron-Turing NLG on a broad set of downstream tasks."],"includes":["markdown","claims","timeline","benchmarks","open_problems","key_papers","glossary","bibliography"],"citation_count":59,"how_to_buy":"GET https://space-kitty.com/api/v1/dossiers/scaling-laws with an x402 client ($0.20, USDC on Base Sepolia (testnet))."}