{"id":"arxiv:2610.33299","kind":"preprint","title":"The Emergence and Sparsification of a Syntactic Circuit","authors":["Qing Yao","Sasha Boguraev","Tiago Pimentel","Kyle Mahowald"],"author_count":4,"year":2026,"published":"2026-10-23","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2610.33299","nct_id":null,"source_url":"https://arxiv.org/abs/2610.33299","archive_url":"https://space-kitty.com/papers/arxiv:2610.33299","updated":"2026-10-23","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2610.33299","abstract":"To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}