{"topic":{"slug":"llm-architectures","name":"LLM Architectures & Pretraining","domain":"ai","description":"How frontier language models are built: transformer variants, mixture-of-experts, state-space and hybrid models, long-context attention, tokenization and pretraining recipes.","keywords":["large language model architecture","mixture of experts","state space model","long context","linear attention","pretraining","Mamba","sparse mixture-of-experts"],"paper_count":309,"latest_published":"2026-10-27","url":"https://space-kitty.com/topics/llm-architectures","api_url":"https://space-kitty.com/api/v1/topics/llm-architectures"},"dossier":{"slug":"llm-architectures","topic":"llm-architectures","title":"LLM Architectures & Pretraining: State of the Field (October 2026)","summary":"How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention.","version":1,"published_at":"2026-10-29","updated_at":"2026-10-29","stats":{"words":5034,"citations":111,"claims":38,"timeline":20,"benchmarks":13,"open_problems":6,"key_papers":14,"glossary":16},"verification":{"status":"verified","verified_at":"2026-10-29","method":"evidence-pack check of 38 claims + 118 contexts against 111 cited records, plus 15 web checks on primary arXiv pages (13 returned content, 2 API requests were rate-limited)","claims_checked":38,"issues_fixed":54,"notes":["Qwen3.5: the Gated DeltaNet hybrid claim was cited to the Qwen3.5-Omni report, whose abstract only supports a hybrid-attention MoE; re-pointed to arxiv:2609.20751 (three GDN layers per attention layer, checked on the paper) and 'introduced' removed (exec summary, hybrids paragraph, c11).","Hybrids paragraph: 'in 2026 hybrids reached frontier scale' contradicted the dossier's own 2025 sources (MiniMax-01 and M1 are 456B hybrids); reworded.","MiniMax Sparse Attention: 'matched GQA quality' changed to 'on par, benchmark-dependent gaps'; 28.4x labelled a theoretical FLOPs figure; speedups tied to 1M context; link to MiniMax-M3 confirmed (body, c5, benchmarks).","Scope of 'largest open-weight' claims limited to the models surveyed (front matter, exec summary, benchmark metric, Kimi K3 key-paper note); loss-free balancing narrowed from 'several frontier MoEs' to DeepSeek-V3.","Nemotron-H: 'FP8 matched BF16 for the 56B' corrected to a recipe reported on par with BF16 and used for the 56B; c4 'replaces residual connections' changed to 'augments' (abstract: enhances); c22 'Most of the gap' changed to 'Much'.","MiniMax-M2: wording aligned with the report (earlier hybrids adequate on standard benchmarks, multi-hop deficits at scale, sliding-window variants worse beyond 32K after fine-tuning, linear/sparse infrastructure less mature); table row relabelled 'report 2026-05' because the cited record does not give the release date.","Checked against primary pages: DeepSeek-V4 submission date 2026-04-26, CSA m=4, HCA m'=128, 33T/32T tokens; Kimi K3 69 KDA + 24 MLA layers, NoPE in MLA, Muon, weights released; GLM-5 744B/40B, 28.5T tokens, MLA+DSA, Muon; Nemotron 3 Super latent 1024 vs 4096, 22 of 512 experts, BF16 layers, no positional embeddings; Mamba-3 +1.8 (0.6 base + 1.2 MIMO); channel selection 98.1%/17.8%; context compression 7.9x; Qwen3.8 Muon/batch-size claim.","Vendor or self-reported figures labelled (Gemini 1.5 retrieval, Mercury throughput, DeepSeek-V4 27%/10%, Kimi Linear, Nemotron-H, Kimi K2 loss spikes) and a disclosure line added under the benchmarks table; Mercury and dQwen3.5 caveats added.","Benchmarks JSON aligned with the body (token-count row now Qwen3 36T / V4-Pro 33T / GLM-5 28.5T, NVFP4 row includes Ultra, KV-cache row labelled versus DeepSeek 67B); timeline entry for Kimi Linear marked as reported.","Removed unsupported items: NVIDIA attribution of Gated DeltaNet-2, 'revised in 2024' for the data-exhaustion forecast, 'NVIDIA's' study label, absence claim about tokenization; outlook data bullet made conditional.","Softened overclaims: pure recurrent models 'still fail' at recall, 'widely used' (Llama 3, muP, schedules), 'lower-risk', 'increasingly drop positions', DeepSeekMoE 'spread/template', Muon 'evidence base', router-score generalization, 'undisclosed data', Lost-in-the-Middle dated 2023-2024.","Not re-fetched within the 15-fetch budget: LinMix 46% MBPP (hedged as 'up to', single preprint) and Attention Residuals 'every task' (reworded); details outside the pack's truncated abstracts and left as written: NVFP4 12B/10T, Qwen3 36T tokens, gpt-oss 128-token window, MiniMax-01 7:1 layer ratio, Kimi K2 MLA, Gemini 'sparse MoE' description."]},"ai_assisted":true,"price":{"usd":"0.20","protocol":"x402","network":"eip155:84532","asset":"USDC"},"url":"https://space-kitty.com/dossiers/llm-architectures","preview_url":"https://space-kitty.com/api/v1/dossiers/llm-architectures/preview","purchase_url":"https://space-kitty.com/api/v1/dossiers/llm-architectures"},"top_papers":[{"id":"arxiv:2005.14165","kind":"preprint","title":"Language Models are Few-Shot Learners","authors":["Brown, Tom B.","Benjamin Mann","Nick Ryder","Melanie Subbiah","Jared Kaplan A","Prafulla Dhariwal"],"author_count":31,"year":2020,"published":"2020-05-28","venue":null,"cited_by":23014,"doi":null,"arxiv_id":"2005.14165","nct_id":null,"source_url":"http://arxiv.org/abs/2005.14165","archive_url":"https://space-kitty.com/papers/arxiv:2005.14165"},{"id":"arxiv:2303.18223","kind":"article","title":"A Survey of Large Language Models","authors":["Wayne Xin Zhao","Kun Zhou","Junyi Li","Tianyi Tang","Zican Dong","Yupeng Hou"],"author_count":23,"year":2026,"published":"2026-05-09","venue":"Frontiers of Computer Science","cited_by":2085,"doi":"10.1007/s11704-026-60308-3","arxiv_id":"2303.18223","nct_id":null,"source_url":"https://doi.org/10.1007/s11704-026-60308-3","archive_url":"https://space-kitty.com/papers/arxiv:2303.18223"},{"id":"arxiv:1901.08746","kind":"article","title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","authors":["Jinhyuk Lee","Wonjin Yoon","Sungdong Kim","Donghyeon Kim","Sunkyu Kim","Chan Ho So"],"author_count":7,"year":2019,"published":"2019-09-05","venue":"Bioinformatics","cited_by":7208,"doi":"10.1093/bioinformatics/btz682","arxiv_id":"1901.08746","nct_id":null,"source_url":"https://doi.org/10.1093/bioinformatics/btz682","archive_url":"https://space-kitty.com/papers/arxiv:1901.08746"},{"id":"arxiv:2101.01169","kind":"article","title":"Transformers in Vision: A Survey","authors":["Salman Hameed Khan","Muzammal Naseer","Munawar Hayat","Syed Waqas Zamir","Fahad Shahbaz Khan","Mubarak A. Shah"],"author_count":6,"year":2022,"published":"2022-01-06","venue":"ACM Computing Surveys","cited_by":3593,"doi":"10.1145/3505244","arxiv_id":"2101.01169","nct_id":null,"source_url":"https://doi.org/10.1145/3505244","archive_url":"https://space-kitty.com/papers/arxiv:2101.01169"},{"id":"arxiv:2106.09685","kind":"preprint","title":"LoRA: Low-Rank Adaptation of Large Language Models","authors":["J. Edward Hu","Yelong Shen","Phillip Wallis","Zeyuan Allen-Zhu","Yuanzhi Li","Shean Wang"],"author_count":8,"year":2021,"published":"2021-06-17","venue":null,"cited_by":2501,"doi":null,"arxiv_id":"2106.09685","nct_id":null,"source_url":"http://arxiv.org/abs/2106.09685","archive_url":"https://space-kitty.com/papers/arxiv:2106.09685"},{"id":"arxiv:2307.06435","kind":"article","title":"A Comprehensive Overview of Large Language Models","authors":["Humza Naveed","Asad Ullah Khan","Shi Qiu","Muhammad Saqib","Saeed Anwar","Muhammad Usman"],"author_count":9,"year":2025,"published":"2025-06-18","venue":"ACM Transactions on Intelligent Systems and Technology","cited_by":1255,"doi":"10.1145/3744746","arxiv_id":"2307.06435","nct_id":null,"source_url":"https://doi.org/10.1145/3744746","archive_url":"https://space-kitty.com/papers/arxiv:2307.06435"},{"id":"arxiv:1908.02265","kind":"preprint","title":"ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks","authors":["Jiasen Lu","Dhruv Batra","Devi Parikh","Stefan Lee"],"author_count":4,"year":2019,"published":"2019-08-06","venue":null,"cited_by":2111,"doi":null,"arxiv_id":"1908.02265","nct_id":null,"source_url":"http://arxiv.org/abs/1908.02265","archive_url":"https://space-kitty.com/papers/arxiv:1908.02265"},{"id":"arxiv:1807.10854","kind":"review","title":"A Survey of the Usages of Deep Learning for Natural Language Processing","authors":["Daniel W. Otter","Julian Richard Medina","Jugal Kumar Kalita"],"author_count":3,"year":2020,"published":"2020-04-21","venue":"IEEE Transactions on Neural Networks and Learning Systems","cited_by":1836,"doi":"10.1109/tnnls.2020.2979670","arxiv_id":"1807.10854","nct_id":null,"source_url":"https://doi.org/10.1109/tnnls.2020.2979670","archive_url":"https://space-kitty.com/papers/arxiv:1807.10854"},{"id":"arxiv:1506.00019","kind":"preprint","title":"A Critical Review of Recurrent Neural Networks for Sequence Learning","authors":["Zachary C. Lipton","John A. Berkowitz","Charles Elkan"],"author_count":3,"year":2015,"published":"2015-05-29","venue":null,"cited_by":2074,"doi":null,"arxiv_id":"1506.00019","nct_id":null,"source_url":"http://arxiv.org/abs/1506.00019","archive_url":"https://space-kitty.com/papers/arxiv:1506.00019"},{"id":"arxiv:2001.08361","kind":"preprint","title":"Scaling Laws for Neural Language Models","authors":["Jared Kaplan A","Sam McCandlish","Tom Henighan","Brown, Tom B.","Benjamin Chess","Rewon Child"],"author_count":10,"year":2020,"published":"2020-01-23","venue":null,"cited_by":1751,"doi":null,"arxiv_id":"2001.08361","nct_id":null,"source_url":"http://arxiv.org/abs/2001.08361","archive_url":"https://space-kitty.com/papers/arxiv:2001.08361"},{"id":"arxiv:2204.14198","kind":"preprint","title":"Flamingo: a Visual Language Model for Few-Shot Learning","authors":["Jean-Baptiste Alayrac","Jeff Donahue","Pauline Luc","Antoine Miech","Barr, Iain","Yana Hasson"],"author_count":27,"year":2022,"published":"2022-04-29","venue":null,"cited_by":1555,"doi":null,"arxiv_id":"2204.14198","nct_id":null,"source_url":"http://arxiv.org/abs/2204.14198","archive_url":"https://space-kitty.com/papers/arxiv:2204.14198"},{"id":"arxiv:2111.01243","kind":"article","title":"Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey","authors":["Bonan Min","Hayley Ross","Elior Sulem","Amir Pouran Ben Veyseh","Thien Huu Nguyen","Oscar Sainz"],"author_count":9,"year":2023,"published":"2023-06-27","venue":"ACM Computing Surveys","cited_by":1386,"doi":"10.1145/3605943","arxiv_id":"2111.01243","nct_id":null,"source_url":"https://doi.org/10.1145/3605943","archive_url":"https://space-kitty.com/papers/arxiv:2111.01243"},{"id":"doi:10.3390/electronics8030292","kind":"article","title":"A State-of-the-Art Survey on Deep Learning Theory and Architectures","authors":["Md Zahangir Alom","Tarek M. Taha","Chris Yakopcic","Stefan K.E. Westberg","Paheding Sidike","Mst Shamima Nasrin"],"author_count":10,"year":2019,"published":"2019-03-05","venue":"Electronics","cited_by":1622,"doi":"10.3390/electronics8030292","arxiv_id":null,"nct_id":null,"source_url":"http://dx.doi.org/10.3390/electronics8030292","archive_url":"https://space-kitty.com/papers/doi:10.3390/electronics8030292"},{"id":"doi:10.1109/access.2024.3365742","kind":"article","title":"A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges","authors":["Mohaimenul Azam Khan Raiaan","Md. Saddam Hossain Mukta","Kaniz Fatema","Nur Mohammad Fahad","Sadman Sakib","Most. Marufatul Jannat Mim"],"author_count":9,"year":2024,"published":"2024-01-01","venue":"IEEE Access","cited_by":933,"doi":"10.1109/access.2024.3365742","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/access.2024.3365742","archive_url":"https://space-kitty.com/papers/doi:10.1109/access.2024.3365742"},{"id":"arxiv:1612.08083","kind":"preprint","title":"Language Modeling with Gated Convolutional Networks","authors":["Yann Dauphin","Angela Fan","Michael Auli","David Grangier"],"author_count":4,"year":2016,"published":"2016-12-23","venue":null,"cited_by":1577,"doi":null,"arxiv_id":"1612.08083","nct_id":null,"source_url":"http://arxiv.org/abs/1612.08083","archive_url":"https://space-kitty.com/papers/arxiv:1612.08083"}],"newest_papers":[{"id":"arxiv:2610.36538","kind":"preprint","title":"Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization","authors":["Zhiheng Qian","Junhao Du","Jing Peng","Hanqi Li","Kunyang Peng","Feng Xue"],"author_count":9,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36538","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36538","archive_url":"https://space-kitty.com/papers/arxiv:2610.36538"},{"id":"arxiv:2610.36583","kind":"preprint","title":"MM-Net: A Multimodal Multi-Task Network for Sleep Staging and Sleep Apnea Assessment After Ischemic Stroke","authors":["Md Wahiduzzaman Suva","Esm E Moula Chowdhury Abha","Md Imtiaj Alam Sajin","Md Iftekharul Mobin","Jia Uddin"],"author_count":5,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36583","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36583","archive_url":"https://space-kitty.com/papers/arxiv:2610.36583"},{"id":"arxiv:2610.36617","kind":"preprint","title":"Balanced State Subspaces for Mamba-2 State Compression","authors":["Shang Tingyu","Cao Wenfei"],"author_count":2,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36617","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36617","archive_url":"https://space-kitty.com/papers/arxiv:2610.36617"},{"id":"arxiv:2610.36778","kind":"preprint","title":"Does LLM Pre-Training Typically Occur at the Edge of Stability?","authors":["Yuhang Cai","Haodong Wen","Haofeng Huang","Deyi Liu","Yiyuan Ma","Kaifeng Lyu"],"author_count":6,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36778","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36778","archive_url":"https://space-kitty.com/papers/arxiv:2610.36778"},{"id":"arxiv:2610.36794","kind":"preprint","title":"Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time Training","authors":["Yue Wu","Yang Zhang","Weinan Gan","Chongming Gao","Xinsen Guo","Xiaoyuan Li"],"author_count":7,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36794","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36794","archive_url":"https://space-kitty.com/papers/arxiv:2610.36794"},{"id":"arxiv:2610.36823","kind":"preprint","title":"TAPESTRY: A Single Backbone for Video Generation and Robot Control","authors":["Jeremy A. Collins","Seungjae Lee","Rachanon Wachakorn","Krishnan Srinivasan","Animesh Garg","Vitor Campagnolo Guizilini"],"author_count":7,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36823","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36823","archive_url":"https://space-kitty.com/papers/arxiv:2610.36823"},{"id":"arxiv:2610.36877","kind":"preprint","title":"GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents","authors":["Bingchao Wang","Yongcheng Jing","Shunyu Liu","Chengyu Wang","Jun Huang","Jie Yang"],"author_count":8,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36877","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36877","archive_url":"https://space-kitty.com/papers/arxiv:2610.36877"},{"id":"arxiv:2610.36896","kind":"preprint","title":"PREEN: Exemplar-Based Color Style Transfer with Smartphone Editor Presets","authors":["Beomjin Ahn"],"author_count":1,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36896","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36896","archive_url":"https://space-kitty.com/papers/arxiv:2610.36896"},{"id":"arxiv:2610.36950","kind":"preprint","title":"DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models","authors":["Egor Maximov","Aleksei Merkelov","Vasiliy Alekseev","Ivan Krivorotov","Valentin Beliaev","Nikita Arsenin"],"author_count":8,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.36950","nct_id":null,"source_url":"https://arxiv.org/abs/2610.36950","archive_url":"https://space-kitty.com/papers/arxiv:2610.36950"},{"id":"arxiv:2610.37037","kind":"preprint","title":"Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal Perception","authors":["Lai Wei","Yatai Li","Liangbo He","Jun Lan","Renjie Gu","Linhan Cao"],"author_count":13,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.37037","nct_id":null,"source_url":"https://arxiv.org/abs/2610.37037","archive_url":"https://space-kitty.com/papers/arxiv:2610.37037"}],"related_topics":[{"slug":"scaling-laws","name":"Scaling Laws & Compute-Optimal Training","api_url":"https://space-kitty.com/api/v1/topics/scaling-laws"},{"slug":"reasoning-models","name":"Reasoning Models & Test-Time Compute","api_url":"https://space-kitty.com/api/v1/topics/reasoning-models"},{"slug":"alignment-safety","name":"AI Alignment & Safety","api_url":"https://space-kitty.com/api/v1/topics/alignment-safety"},{"slug":"interpretability","name":"Mechanistic Interpretability","api_url":"https://space-kitty.com/api/v1/topics/interpretability"},{"slug":"ai-agents","name":"LLM Agents & Tool Use","api_url":"https://space-kitty.com/api/v1/topics/ai-agents"},{"slug":"generative-multimodal","name":"Generative & Multimodal Models","api_url":"https://space-kitty.com/api/v1/topics/generative-multimodal"},{"slug":"efficient-ai","name":"Efficient Training & Inference","api_url":"https://space-kitty.com/api/v1/topics/efficient-ai"},{"slug":"ai-for-science","name":"AI for Science","api_url":"https://space-kitty.com/api/v1/topics/ai-for-science"},{"slug":"robotics-embodied","name":"Robot Learning & Embodied AI","api_url":"https://space-kitty.com/api/v1/topics/robotics-embodied"}],"premium":{"dossier":{"url":"https://space-kitty.com/api/v1/dossiers/llm-architectures","price_usd":"0.20"},"research_pack":{"url":"https://space-kitty.com/api/v1/topics/llm-architectures/pack","price_usd":"0.20"}},"more":"https://space-kitty.com/api/v1/topics/llm-architectures/papers?sort=top&limit=100"}