# The Emergence and Sparsification of a Syntactic Circuit

Preprint · Qing Yao, Sasha Boguraev, Tiago Pimentel, Kyle Mahowald

- Archive id: arxiv:2610.33299
- Published: 2026-10-23
- arXiv: https://arxiv.org/abs/2610.33299
- Source URL: https://arxiv.org/abs/2610.33299
- PDF: https://arxiv.org/pdf/2610.33299
- Provenance: arxiv

## Abstract

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.