# LLM Architectures & Pretraining: State of the Field (October 2026)

Research dossier · v1 · updated 2026-10-29 · 5,034 words · 111 sources · 38 claims · verification: verified

How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention.

## Contents

- Executive summary
- Background and core concepts
- Timeline of key milestones
- State of the art
  - Sparse mixture-of-experts is the default at scale
  - Attention: compress, sparsify, window or gate
  - Linear recurrences and hybrid stacks
  - Long context
  - Training recipe: optimizers, precision, residuals and objectives
  - Data and scaling laws
  - Tokenization and non-autoregressive alternatives
  - Quantitative benchmarks
- Open problems and active debates
- Major programs and players
- Outlook: what to watch (November 2026 to April 2028)
- Key papers
- Glossary

## Sample findings

- Kimi K3 is a Mixture-of-Experts model with 2.8T total and 104B activated parameters and a 1M-token context window that activates 16 of 896 routed experts per token.
- The DeepSeek-V4 preview comprises DeepSeek-V4-Pro (1.6T total, 49B activated parameters) and DeepSeek-V4-Flash (284B total, 13B activated), both supporting 1M-token contexts.

## Get the full dossier

GET https://space-kitty.com/api/v1/dossiers/llm-architectures costs $0.20 via x402 (USDC on Base Sepolia (testnet)). Send `Accept: text/markdown` for Markdown or `application/json` for structured data (claims with citations, timeline, benchmarks, open problems, glossary, bibliography).
---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.