# Scaling Laws & Compute-Optimal Training Domain: Artificial Intelligence · 205 papers indexed · newest 2026-10-27 Empirical laws relating loss and capability to parameters, data and compute; data-constrained and synthetic-data scaling; emergent abilities and their measurement. Key concepts: scaling laws, compute-optimal, emergent abilities, data-constrained, synthetic data, scaling law ## Most influential - **[Scaling Laws for Neural Language Models](https://space-kitty.com/papers/arxiv:2001.08361)**. Jared Kaplan A, Sam McCandlish, Tom Henighan et al. · 2020 (arXiv:2001.08361 · 1,751 citations). Source: > We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are s… - **[Twenty‐first century technologies and language education: Charting a path forward](https://space-kitty.com/papers/doi:10.1111/modl.12924)**. Richard Kern · 2024 · Modern Language Journal (doi:10.1111/modl.12924 · 82 citations). Source: - **[Reproducible scaling laws for contrastive language-image learning](https://space-kitty.com/papers/arxiv:2212.07143)**. Mehdi Cherti, Romain Beaumont, Ross Wightman et al. · 2022 · Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829 (arXiv:2212.07143 · doi:10.1109/cvpr52729.2023.00276 · 672 citations). Source: > Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offers valuable guidance as large-scale experiments are becoming increasingly expensive. However, previous work on scaling laws has primarily used private data \\& models or focused on uni-modal language or vision learning. To address these limitations, we investigate scaling laws for contrastive language-image pre-training (CLIP) with the public LAION dataset and the open-source Op… - **[The Potential of Neural Network Potentials](https://space-kitty.com/papers/doi:10.1021/acsphyschemau.4c00004)**. Timothy T. Duignan · 2024 · ACS Physical Chemistry Au (doi:10.1021/acsphyschemau.4c00004 · 64 citations). Source: - **[Integrating artificial intelligence into health care through data access: can the GDPR act as a beacon for policymakers?](https://space-kitty.com/papers/doi:10.1093/jlb/lsz013)**. Mélanie Bourassa Forcier, Hortense Gallois, Siobhan M. Mullan et al. · 2019 · Journal of Law and the Biosciences (doi:10.1093/jlb/lsz013 · 135 citations). Source: > The potential of artificial intelligence (AI) to promote better health care has taken the centre stage in modern debates on public health and health policy. Although AI is considered a contemporary innovation, it has been in development for more than a half century. AI research began in the 1950s, when Alan Turing raised the idea that machines could 1 day think as humans.1 Then came, in 1959, the first instance of ‘machine learning’ (ML), where computer scientists created a program capable of solving puzzles on its own.2 Now, AI promises to lead the next major technological revolution, similar… > _Abstract excerpt by Mélanie Bourassa Forcier et al., CC-BY, https://doi.org/10.1093/jlb/lsz013; truncated._ - **[The architecture of language: Understanding the mechanics behind LLMs](https://space-kitty.com/papers/doi:10.1017/cfl.2024.16)**. Andrea Filippo Ferraris, Davide Audrito, Luigi Di et al. · 2025 · Cambridge Forum on AI Law and Governance (doi:10.1017/cfl.2024.16 · 34 citations). Source: > Abstract Large language models (LLMs) have significantly advanced artificial intelligence (AI) and natural language processing (NLP) by excelling in tasks like text generation, machine translation, question answering and sentiment analysis, often rivaling human performance. This paper reviews LLMs’ foundations, advancements and applications, beginning with the transformative transformer architecture, which improved on earlier models like recurrent neural networks and convolutional neural networks through self-attention mechanisms that capture long-range dependencies and contextual relationship… > _Abstract excerpt by Andrea Filippo Ferraris et al., CC-BY, https://doi.org/10.1017/cfl.2024.16; truncated._ - **[Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://space-kitty.com/papers/arxiv:2601.07372)**. Xin Cheng, Rui Tian, Wangding Zeng et al. · 2026 (arXiv:2601.07372 · 21 citations). Source: > While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B par… - **[Language writ large: LLMs, ChatGPT, meaning, and understanding](https://space-kitty.com/papers/doi:10.3389/frai.2024.1490698)**. Stevan R. Harnad · 2025 · Frontiers in Artificial Intelligence (doi:10.3389/frai.2024.1490698 · 29 citations). Source: > Apart from what (little) OpenAI may be concealing from us, we all know (roughly) how Large Language Models (LLMs) such as ChatGPT work (their vast text databases, statistics, vector representations, and huge number of parameters, next-word training, etc.). However, none of us can say (hand on heart) that we are not surprised by what ChatGPT has proved to be able to do with these resources. This has even driven some of us to conclude that ChatGPT actually understands. It is not true that it understands. But it is also not true that we understand how it can do what it can do. I will suggest some… > _Abstract excerpt by Stevan R. Harnad, CC-BY, https://doi.org/10.3389/frai.2024.1490698; truncated._ - **[Scaling Laws for Autoregressive Generative Modeling](https://space-kitty.com/papers/arxiv:2010.14701)**. Tom Henighan, Jared Kaplan, Mor Katz et al. · 2020 (arXiv:2010.14701 · 150 citations). Source: > We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D\_{\\… - **[Scaling Laws for Transfer](https://space-kitty.com/papers/arxiv:2102.01293)**. Danny Hernandez, Jared Kaplan A, Tom Henighan et al. · 2021 (arXiv:2102.01293 · 51 citations). Source: > We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero. We calculate the effective data "transferred" from pre-training by determining how much data a transformer of the same size would have required to achieve t… - **[CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://space-kitty.com/papers/arxiv:2305.02309)**. Erik Nijkamp, Hayashi, Hiroaki, Caiming Xiong et al. · 2023 (arXiv:2305.02309 · 35 citations). Source: > Large language models (LLMs) have demonstrated remarkable abilities in representation learning for program synthesis and understanding tasks. The quality of the learned representations appears to be dictated by the neural scaling laws as a function of the number of model parameters and observations, while imposing upper bounds on the model performance by the amount of available data and compute, which is costly. In this study, we attempt to render the training of LLMs for program synthesis more efficient by unifying four key components: (1) model architectures, (2) learning methods, (3) infill… - **[Hyperbolic Graph Neural Networks: A Review of Methods and Applications](https://space-kitty.com/papers/arxiv:2202.13852)**. Menglin Yang, Min Zhou, Tong Zhang et al. · 2022 (arXiv:2202.13852 · 33 citations). Source: > Graph representation learning in Euclidean space, despite its widespread adoption and proven utility in many domains, often struggles to effectively capture the inherent hierarchical and complex relational structures prevalent in real-world data, particularly for datasets exhibiting a highly non-Euclidean latent anatomy or power-law distributions. Hyperbolic geometry, with its constant negative curvature and exponential growth property, naturally accommodates such structures, offering a promising alternative for learning rich graph representations. This survey paper provides a comprehensive re… - **[Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://space-kitty.com/papers/arxiv:2305.13035)**. Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov et al. · 2023 · 37th Conference on Neural Information Processing Systems (NeurIPS 2023) (arXiv:2305.13035 · doi:10.52202/075280-0722 · 13 citations). Source: > Scaling laws have been recently employed to derive compute-optimal model size (number of parameters) for a given compute duration. We advance and refine such methods to infer compute-optimal model shapes, such as width and depth, and successfully implement this in vision transformers. Our shape-optimized vision transformer, SoViT, achieves results competitive with models that exceed twice its size, despite being pre-trained with an equivalent amount of compute. For example, SoViT-400m/14 achieves 90.3% fine-tuning accuracy on ILSRCV2012, surpassing the much larger ViT-g/14 and approaching ViT-… - **[A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://space-kitty.com/papers/arxiv:2402.07043)**. Elvis Dohmatob, Yunzhen Feng, Pu Yang et al. · 2024 · ICML 2024 (arXiv:2402.07043 · 20 citations). Source: > As AI model size grows, neural scaling laws have become a crucial tool to predict the improvements of large models when increasing capacity and the size of original (human or natural) training data. Yet, the widespread use of popular models means that the ecosystem of online data and text will co-evolve to progressively contain increased amounts of synthesized data. In this paper we ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? Will future models, still improve, or be doomed to degenerate up to total (model) collapse… - **[Thoughts About Disordered Thinking: Measuring and Quantifying the Laws of Order and Disorder](https://space-kitty.com/papers/doi:10.1093/schbul/sbx040)**. Brita Elvevåg, Peter W. Foltz, Mark Rosenstein et al. · 2017 · Schizophrenia Bulletin (doi:10.1093/schbul/sbx040 · 31 citations). Source: ## Newest - **[Developing a Speech Language Model for Basque](https://space-kitty.com/papers/arxiv:2610.36911)**. Andoni Sudupe, Inma Hernaez, Ander Arriandiaga et al. · 2026 (arXiv:2610.36911). Source: - **[Text Generation in Low-Resource Languages Using Human-Curated Vocabularies](https://space-kitty.com/papers/arxiv:2610.37105)**. Michael Christlambert Sinanta, Muhammad Dehan Al Kautsar, Amanda Nurul Izzah et al. · 2026 (arXiv:2610.37105). Source: - **[Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis](https://space-kitty.com/papers/arxiv:2610.37648)**. Andrew Szot, Samira Abnar, Omar Attia et al. · 2026 (arXiv:2610.37648). Source: - **[RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge](https://space-kitty.com/papers/arxiv:2610.34875)**. Balamurugan Paramasivam, Sudalai Rajkumar · 2026 (arXiv:2610.34875). Source: - **[Synthetic Child-Adult Interactions Are Not Developmentally Plausible](https://space-kitty.com/papers/arxiv:2610.35029)**. Bastian Bunzeck, Francesca Padovani · 2026 (arXiv:2610.35029). Source: - **[Sequential versus fully conditional specification for tree-based synthesis of health data](https://space-kitty.com/papers/arxiv:2610.35163)**. R{é}my Chapelle, Mohammed Sedki · 2026 (arXiv:2610.35163). Source: - **[Accelerating SO(2)-Equivariant Materials Models through Low-Precision Emulation](https://space-kitty.com/papers/arxiv:2610.35164)**. Denghui Lu, Alexander Maeder, Alexandros Nikolaos Ziogas et al. · 2026 (arXiv:2610.35164). Source: - **[Depth Improves Training-Time Scaling for Difficult Tasks in Deep Linear Networks](https://space-kitty.com/papers/arxiv:2610.35514)**. Taishi Watanabe, Ryo Karakida, Jun-nosuke Teramae · 2026 (arXiv:2610.35514). Source: - **[Dirichlet Process Mixtures within Semiparametric Bayesian Networks: A Novel Conditional Density Estimation Approach](https://space-kitty.com/papers/arxiv:2610.36110)**. Victor Alejandre, Pedro Larrañaga, Concha Bielza · 2026 (arXiv:2610.36110). Source: - **[Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs](https://space-kitty.com/papers/arxiv:2610.36195)**. Kartik Jangra, Atharva Gawane, Siddharth Pulipaka et al. · 2026 (arXiv:2610.36195). Source: --- Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.