LLM Architectures & Pretraining: State of the Field (October 2026)
How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention.
AI-assisted synthesis of the cited literature, independently fact-checked against its sources; not peer reviewed.
Contents
- Executive summary
- Background and core concepts
- Timeline of key milestones
- State of the art
- Sparse mixture-of-experts is the default at scale
- Attention: compress, sparsify, window or gate
- Linear recurrences and hybrid stacks
- Long context
- Training recipe: optimizers, precision, residuals and objectives
- Data and scaling laws
- Tokenization and non-autoregressive alternatives
- Quantitative benchmarks
- Open problems and active debates
- Major programs and players
- Outlook: what to watch (November 2026 to April 2028)
- Key papers
- Glossary