Scaling Laws & Compute-Optimal Training
Empirical laws relating loss and capability to parameters, data and compute; data-constrained and synthetic-data scaling; emergent abilities and their measurement.
Most influential
- Scaling Laws for Neural Language Models
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range.…
Preprint1,751 citationsarXiv:2001.08361 - Twenty‐first century technologies and language education: Charting a path forwardArticle82 citationsdoi:10.1111/modl.12924
- Reproducible scaling laws for contrastive language-image learning
Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily us…
Article672 citationsarXiv:2212.07143doi:10.1109/cvpr52729.2023.00276 - The Potential of Neural Network PotentialsReview64 citationsdoi:10.1021/acsphyschemau.4c00004
- Integrating artificial intelligence into health care through data access: can the GDPR act as a beacon for policymakers?
The potential of artificial intelligence (AI) to promote better health care has taken the centre stage in modern debates on public health and health policy. Although AI is considered a contemporary innovation, it has been in development for more than a half century. AI research began in the 1950s, when Alan Turing raised the idea that machines could 1 day th…
Abstract excerpt by Mélanie Bourassa Forcier et al., CC-BY, https://doi.org/10.1093/jlb/lsz013; truncated.
Article135 citationsdoi:10.1093/jlb/lsz013 - The architecture of language: Understanding the mechanics behind LLMs
Abstract Large language models (LLMs) have significantly advanced artificial intelligence (AI) and natural language processing (NLP) by excelling in tasks like text generation, machine translation, question answering and sentiment analysis, often rivaling human performance. This paper reviews LLMs’ foundations, advancements and applications, beginning with t…
Abstract excerpt by Andrea Filippo Ferraris et al., CC-BY, https://doi.org/10.1017/cfl.2024.16; truncated.
Article34 citationsdoi:10.1017/cfl.2024.16 - Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embed…
Preprint21 citationsarXiv:2601.07372 - Language writ large: LLMs, ChatGPT, meaning, and understanding
Apart from what (little) OpenAI may be concealing from us, we all know (roughly) how Large Language Models (LLMs) such as ChatGPT work (their vast text databases, statistics, vector representations, and huge number of parameters, next-word training, etc.). However, none of us can say (hand on heart) that we are not surprised by what ChatGPT has proved to be …
Abstract excerpt by Stevan R. Harnad, CC-BY, https://doi.org/10.3389/frai.2024.1490698; truncated.
Article29 citationsdoi:10.3389/frai.2024.1490698 - Scaling Laws for Autoregressive Generative Modeling
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scali…
Preprint150 citationsarXiv:2010.14701 - Scaling Laws for Transfer
We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language d…
Preprint51 citationsarXiv:2102.01293 - CodeGen2: Lessons for Training LLMs on Programming and Natural Languages
Large language models (LLMs) have demonstrated remarkable abilities in representation learning for program synthesis and understanding tasks. The quality of the learned representations appears to be dictated by the neural scaling laws as a function of the number of model parameters and observations, while imposing upper bounds on the model performance by the…
Preprint35 citationsarXiv:2305.02309 - Hyperbolic Graph Neural Networks: A Review of Methods and Applications
Graph representation learning in Euclidean space, despite its widespread adoption and proven utility in many domains, often struggles to effectively capture the inherent hierarchical and complex relational structures prevalent in real-world data, particularly for datasets exhibiting a highly non-Euclidean latent anatomy or power-law distributions. Hyperbolic…
Preprint33 citationsarXiv:2202.13852 - Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design
Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitiv…
Article13 citationsarXiv:2305.13035doi:10.52202/075280-0722 - A Tale of Tails: Model Collapse as a Change of Scaling Laws
As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesiz…
Preprint20 citationsarXiv:2402.07043 - Thoughts About Disordered Thinking: Measuring and Quantifying the Laws of Order and DisorderArticle31 citationsdoi:10.1093/schbul/sbx040
Newest work
- Developing a Speech Language Model for BasquePreprintarXiv:2610.36911
- Text Generation in Low-Resource Languages Using Human-Curated VocabulariesPreprintarXiv:2610.37105
- Scaling Laws for Tool-Use Agents via Controllable Environment SynthesisPreprintarXiv:2610.37648
- RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric JudgePreprintarXiv:2610.34875
- Synthetic Child-Adult Interactions Are Not Developmentally PlausiblePreprintarXiv:2610.35029
- Sequential versus fully conditional specification for tree-based synthesis of health dataPreprintarXiv:2610.35163
- Accelerating SO(2)-Equivariant Materials Models through Low-Precision EmulationPreprintarXiv:2610.35164
- Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear NetworksPreprintarXiv:2610.35514
- Dirichlet Process Mixtures within Semiparametric Bayesian Networks: A Novel Conditional Density Estimation ApproachPreprintarXiv:2610.36110
- Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMsPreprintarXiv:2610.36195