{"slug":"llm-architectures","topic":"llm-architectures","title":"LLM Architectures & Pretraining: State of the Field (October 2026)","summary":"How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention.","version":1,"published_at":"2026-10-29","updated_at":"2026-10-29","stats":{"words":5034,"citations":111,"claims":38,"timeline":20,"benchmarks":13,"open_problems":6,"key_papers":14,"glossary":16},"verification":{"status":"verified","verified_at":"2026-10-29","method":"evidence-pack check of 38 claims + 118 contexts against 111 cited records, plus 15 web checks on primary arXiv pages (13 returned content, 2 API requests were rate-limited)","claims_checked":38,"issues_fixed":54,"notes":["Qwen3.5: the Gated DeltaNet hybrid claim was cited to the Qwen3.5-Omni report, whose abstract only supports a hybrid-attention MoE; re-pointed to arxiv:2609.20751 (three GDN layers per attention layer, checked on the paper) and 'introduced' removed (exec summary, hybrids paragraph, c11).","Hybrids paragraph: 'in 2026 hybrids reached frontier scale' contradicted the dossier's own 2025 sources (MiniMax-01 and M1 are 456B hybrids); reworded.","MiniMax Sparse Attention: 'matched GQA quality' changed to 'on par, benchmark-dependent gaps'; 28.4x labelled a theoretical FLOPs figure; speedups tied to 1M context; link to MiniMax-M3 confirmed (body, c5, benchmarks).","Scope of 'largest open-weight' claims limited to the models surveyed (front matter, exec summary, benchmark metric, Kimi K3 key-paper note); loss-free balancing narrowed from 'several frontier MoEs' to DeepSeek-V3.","Nemotron-H: 'FP8 matched BF16 for the 56B' corrected to a recipe reported on par with BF16 and used for the 56B; c4 'replaces residual connections' changed to 'augments' (abstract: enhances); c22 'Most of the gap' changed to 'Much'.","MiniMax-M2: wording aligned with the report (earlier hybrids adequate on standard benchmarks, multi-hop deficits at scale, sliding-window variants worse beyond 32K after fine-tuning, linear/sparse infrastructure less mature); table row relabelled 'report 2026-05' because the cited record does not give the release date.","Checked against primary pages: DeepSeek-V4 submission date 2026-04-26, CSA m=4, HCA m'=128, 33T/32T tokens; Kimi K3 69 KDA + 24 MLA layers, NoPE in MLA, Muon, weights released; GLM-5 744B/40B, 28.5T tokens, MLA+DSA, Muon; Nemotron 3 Super latent 1024 vs 4096, 22 of 512 experts, BF16 layers, no positional embeddings; Mamba-3 +1.8 (0.6 base + 1.2 MIMO); channel selection 98.1%/17.8%; context compression 7.9x; Qwen3.8 Muon/batch-size claim.","Vendor or self-reported figures labelled (Gemini 1.5 retrieval, Mercury throughput, DeepSeek-V4 27%/10%, Kimi Linear, Nemotron-H, Kimi K2 loss spikes) and a disclosure line added under the benchmarks table; Mercury and dQwen3.5 caveats added.","Benchmarks JSON aligned with the body (token-count row now Qwen3 36T / V4-Pro 33T / GLM-5 28.5T, NVFP4 row includes Ultra, KV-cache row labelled versus DeepSeek 67B); timeline entry for Kimi Linear marked as reported.","Removed unsupported items: NVIDIA attribution of Gated DeltaNet-2, 'revised in 2024' for the data-exhaustion forecast, 'NVIDIA's' study label, absence claim about tokenization; outlook data bullet made conditional.","Softened overclaims: pure recurrent models 'still fail' at recall, 'widely used' (Llama 3, muP, schedules), 'lower-risk', 'increasingly drop positions', DeepSeekMoE 'spread/template', Muon 'evidence base', router-score generalization, 'undisclosed data', Lost-in-the-Middle dated 2023-2024.","Not re-fetched within the 15-fetch budget: LinMix 46% MBPP (hedged as 'up to', single preprint) and Attention Residuals 'every task' (reworded); details outside the pack's truncated abstracts and left as written: NVFP4 12B/10T, Qwen3 36T tokens, gpt-oss 128-token window, MiniMax-01 7:1 layer ratio, Kimi K2 MLA, Gemini 'sparse MoE' description."]},"ai_assisted":true,"price":{"usd":"0.20","protocol":"x402","network":"eip155:84532","asset":"USDC"},"url":"https://space-kitty.com/dossiers/llm-architectures","preview_url":"https://space-kitty.com/api/v1/dossiers/llm-architectures/preview","purchase_url":"https://space-kitty.com/api/v1/dossiers/llm-architectures","toc":[{"level":2,"text":"Executive summary","anchor":"executive-summary"},{"level":2,"text":"Background and core concepts","anchor":"background-and-core-concepts"},{"level":2,"text":"Timeline of key milestones","anchor":"timeline-of-key-milestones"},{"level":2,"text":"State of the art","anchor":"state-of-the-art"},{"level":3,"text":"Sparse mixture-of-experts is the default at scale","anchor":"sparse-mixture-of-experts-is-the-default-at-scale"},{"level":3,"text":"Attention: compress, sparsify, window or gate","anchor":"attention-compress-sparsify-window-or-gate"},{"level":3,"text":"Linear recurrences and hybrid stacks","anchor":"linear-recurrences-and-hybrid-stacks"},{"level":3,"text":"Long context","anchor":"long-context"},{"level":3,"text":"Training recipe: optimizers, precision, residuals and objectives","anchor":"training-recipe-optimizers-precision-residuals-and-objectives"},{"level":3,"text":"Data and scaling laws","anchor":"data-and-scaling-laws"},{"level":3,"text":"Tokenization and non-autoregressive alternatives","anchor":"tokenization-and-non-autoregressive-alternatives"},{"level":3,"text":"Quantitative benchmarks","anchor":"quantitative-benchmarks"},{"level":2,"text":"Open problems and active debates","anchor":"open-problems-and-active-debates"},{"level":2,"text":"Major programs and players","anchor":"major-programs-and-players"},{"level":2,"text":"Outlook: what to watch (November 2026 to April 2028)","anchor":"outlook-what-to-watch-november-2026-to-april-2028"},{"level":2,"text":"Key papers","anchor":"key-papers"},{"level":2,"text":"Glossary","anchor":"glossary"}],"sample_claims":["Kimi K3 is a Mixture-of-Experts model with 2.8T total and 104B activated parameters and a 1M-token context window that activates 16 of 896 routed experts per token.","The DeepSeek-V4 preview comprises DeepSeek-V4-Pro (1.6T total, 49B activated parameters) and DeepSeek-V4-Flash (284B total, 13B activated), both supporting 1M-token contexts."],"includes":["markdown","claims","timeline","benchmarks","open_problems","key_papers","glossary","bibliography"],"citation_count":111,"how_to_buy":"GET https://space-kitty.com/api/v1/dossiers/llm-architectures with an x402 client ($0.20, USDC on Base Sepolia (testnet))."}