# Asymmetric Scaling Laws from Sparse Features

Preprint · John Sous, Michael Winer

- Archive id: arxiv:2605.23591
- Published: 2026-05-22
- arXiv: https://arxiv.org/abs/2605.23591
- Citations: 0
- Source URL: https://arxiv.org/abs/2605.23591
- PDF: https://arxiv.org/pdf/2605.23591
- Provenance: arxiv

## Abstract

We introduce a model for neural scaling laws under sparse activations. In the model, test loss is often dominated by rare coordinates that are never observed in the training input. This mechanism induces a novel bottleneck absent from dense models. We derive the asymptotic population loss in both the underparameterized and overparameterized regimes, and show that the loss exhibits a double-descent peak near the interpolation threshold -- where the number of parameters is just sufficient to fit the training data -- resulting in a loss curve governed by two distinct scaling exponents -- one for the overparameterized regime and one for the underparameterized regime -- with a gap determined by the degree of sparsity. Additionally, we derive a compute-optimal frontier that favors increasing dataset size over model capacity under fixed compute budgets. We also analyze gradient-descent dynamics and identify a scaling law for the probability that fixed-step gradient descent becomes unstable. We further show that the sparsity-induced effect persists under nonlinear activations.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Scaling Laws & Compute-Optimal Training](https://space-kitty.com/topics/scaling-laws)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.