# Toy Models of Superposition

Preprint · Elhage, Nelson, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan A, Dario Amodei, MARTIN P. WATTENBERG, Christopher Olah

- Archive id: arxiv:2209.10652
- Published: 2022-09-21
- arXiv: https://arxiv.org/abs/2209.10652
- Citations: 50
- Source URL: http://arxiv.org/abs/2209.10652
- PDF: https://arxiv.org/pdf/2209.10652
- Provenance: openalex+arxiv

## Abstract

Neural networks often pack many unrelated concepts into a single neuron - a puzzling phenomenon known as 'polysemanticity' which makes interpretability much more challenging. This paper provides a toy model where polysemanticity can be fully understood, arising as a result of models storing additional sparse features in "superposition." We demonstrate the existence of a phase change, a surprising connection to the geometry of uniform polytopes, and evidence of a link to adversarial examples. We also discuss potential implications for mechanistic interpretability.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Mechanistic Interpretability](https://space-kitty.com/topics/interpretability)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.