# A Neural Scaling Law from the Dimension of the Data Manifold

Preprint · Utkarsh Sharma, Jared Kaplan A

- Archive id: arxiv:2004.10802
- Published: 2020-04-22
- arXiv: https://arxiv.org/abs/2004.10802
- Citations: 19
- Source URL: http://arxiv.org/abs/2004.10802
- PDF: https://arxiv.org/pdf/2004.10802
- Provenance: openalex+arxiv

## Abstract

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \\propto N^{-α}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension $d$. This simple theory predicts that the scaling exponents $α\\approx 4/d$ for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of $d$ and $α$ by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Scaling Laws & Compute-Optimal Training](https://space-kitty.com/topics/scaling-laws)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.