# Geometric Limits of Knowledge Distillation: A Minimum-Width Theorem via Superposition Theory

Preprint · Nilesh Sarkar, Dawar Jyoti Deka

- Archive id: arxiv:2604.04037
- Published: 2026-04-05
- arXiv: https://arxiv.org/abs/2604.04037
- Source URL: https://arxiv.org/abs/2604.04037
- PDF: https://arxiv.org/pdf/2604.04037
- Provenance: arxiv

## Abstract

Knowledge distillation compresses large teachers into smaller students, but performance saturates at a loss floor that persists across training methods and objectives. We argue this floor is geometric: neural networks represent far more features than dimensions through superposition, and a student of width $d\_S$ can encode at most $d\_S \\cdot g(α)$ features, where $g(α) = 1/((1-α)\\ln\\frac{1}{1-α})$ is a sparsity-dependent capacity function. Features beyond this budget are permanently lost, yielding an importance-weighted loss floor. We validate on a toy model (48 configurations, median accuracy \>93%) and on Pythia-410M, where sparse autoencoders measure $F \\approx 28{,}700$ features at $α\\approx 0.992$ (critical width $d\_S^\* \\approx 1{,}065$). Distillation into five student widths confirms the predicted monotonic floor ordering. The observed floor decomposes into a geometric component and a width-independent architectural baseline ($R^2 = 0.993$). Linear probing shows coarse concepts survive even 88% feature loss, revealing the floor arises from aggregate loss of fine-grained features in the importance distribution's long tail. Our results connect representation geometry to distillation limits and provide a practical tool for predicting distillation performance from SAE measurements alone.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.