# LLM Architectures & Pretraining Domain: Artificial Intelligence · 334 papers indexed · newest 2026-10-29 How frontier language models are built: transformer variants, mixture-of-experts, state-space and hybrid models, long-context attention, tokenization and pretraining recipes. Key concepts: large language model architecture, mixture of experts, state space model, long context, linear attention, pretraining, Mamba, sparse mixture-of-experts ## Research dossier (premium) **LLM Architectures & Pretraining: State of the Field (October 2026)** (v1, updated 2026-10-29; 5,034 words, 111 sources, 38 evidence-mapped claims) How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention. - Preview (free): https://space-kitty.com/api/v1/dossiers/llm-architectures/preview - Full dossier ($0.20 via x402): https://space-kitty.com/api/v1/dossiers/llm-architectures - Research pack ($0.20): https://space-kitty.com/api/v1/topics/llm-architectures/pack ## Most influential - **[Language Models are Few-Shot Learners](https://space-kitty.com/papers/arxiv:2005.14165)**. Brown, Tom B., Benjamin Mann, Nick Ryder et al. · 2020 (arXiv:2005.14165 · 23,014 citations). Source: > Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performan… - **[A Survey of Large Language Models](https://space-kitty.com/papers/arxiv:2303.18223)**. Wayne Xin Zhao, Kun Zhou, Junyi Li et al. · 2026 · Frontiers of Computer Science (arXiv:2303.18223 · doi:10.1007/s11704-026-60308-3 · 2,085 citations). Source: > Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since… - **[BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://space-kitty.com/papers/arxiv:1901.08746)**. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim et al. · 2019 · Bioinformatics (arXiv:1901.08746 · doi:10.1093/bioinformatics/btz682 · 7,208 citations). Source: > Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recent… - **[Transformers in Vision: A Survey](https://space-kitty.com/papers/arxiv:2101.01169)**. Salman Hameed Khan, Muzammal Naseer, Munawar Hayat et al. · 2022 · ACM Computing Surveys (arXiv:2101.01169 · doi:10.1145/3505244 · 3,593 citations). Source: > Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows process… - **[LoRA: Low-Rank Adaptation of Large Language Models](https://space-kitty.com/papers/arxiv:2106.09685)**. J. Edward Hu, Yelong Shen, Phillip Wallis et al. · 2021 (arXiv:2106.09685 · 2,501 citations). Source: > An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, grea… - **[A Comprehensive Overview of Large Language Models](https://space-kitty.com/papers/arxiv:2307.06435)**. Humza Naveed, Asad Ullah Khan, Shi Qiu et al. · 2025 · ACM Transactions on Intelligent Systems and Technology (arXiv:2307.06435 · doi:10.1145/3744746 · 1,255 citations). Source: > Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multi-modal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the ad… - **[ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://space-kitty.com/papers/arxiv:1908.02265)**. Jiasen Lu, Dhruv Batra, Devi Parikh et al. · 2019 (arXiv:1908.02265 · 2,111 citations). Source: > We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring… - **[A Survey of the Usages of Deep Learning for Natural Language Processing](https://space-kitty.com/papers/arxiv:1807.10854)**. Daniel W. Otter, Julian Richard Medina, Jugal Kumar Kalita · 2020 · IEEE Transactions on Neural Networks and Learning Systems (arXiv:1807.10854 · doi:10.1109/tnnls.2020.2979670 · 1,836 citations). Source: > Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This survey provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of relevant contributions. Analyzed research areas include several core linguistic processing issues in addition to a number of applications of computational linguistics. A discussion of the current state of the art is then provided along with … - **[A Critical Review of Recurrent Neural Networks for Sequence Learning](https://space-kitty.com/papers/arxiv:1506.00019)**. Zachary C. Lipton, John A. Berkowitz, Charles Elkan · 2015 (arXiv:1506.00019 · 2,074 citations). Source: > Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as translating natural language, engaging in dialogue, and controlling a robot, often demand both capabilities. Recurrent neural networks (RNNs) are connectionist models that capture the dynamics of sequences via cycles in the network of nodes… - **[Scaling Laws for Neural Language Models](https://space-kitty.com/papers/arxiv:2001.08361)**. Jared Kaplan A, Sam McCandlish, Tom Henighan et al. · 2020 (arXiv:2001.08361 · 1,751 citations). Source: > We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are s… - **[Flamingo: a Visual Language Model for Few-Shot Learning](https://space-kitty.com/papers/arxiv:2204.14198)**. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc et al. · 2022 (arXiv:2204.14198 · 1,555 citations). Source: > Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing… - **[Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey](https://space-kitty.com/papers/arxiv:2111.01243)**. Bonan Min, Hayley Ross, Elior Sulem et al. · 2023 · ACM Computing Surveys (arXiv:2111.01243 · doi:10.1145/3605943 · 1,386 citations). Source: > Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research. - **[A State-of-the-Art Survey on Deep Learning Theory and Architectures](https://space-kitty.com/papers/doi:10.3390/electronics8030292)**. Md Zahangir Alom, Tarek M. Taha, Chris Yakopcic et al. · 2019 · Electronics (doi:10.3390/electronics8030292 · 1,622 citations). Source: > In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly and has been applied to most traditional application domains, as well as some new areas that present more opportunities. Different methods have been proposed based on different categories of learning, including supervised, semi-supervised, and un-supervised learning. Experimental results show state-of-the-art performance using deep learning when compared to traditional machine learning approaches in the fields of image processing, compu… > _Abstract excerpt by Md Zahangir Alom et al., CC-BY, https://doi.org/10.3390/electronics8030292; truncated._ - **[A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges](https://space-kitty.com/papers/doi:10.1109/access.2024.3365742)**. Mohaimenul Azam Khan Raiaan, Md. Saddam Hossain Mukta, Kaniz Fatema et al. · 2024 · IEEE Access (doi:10.1109/access.2024.3365742 · 933 citations). Source: - **[Language Modeling with Gated Convolutional Networks](https://space-kitty.com/papers/arxiv:1612.08083)**. Yann Dauphin, Angela Fan, Michael Auli et al. · 2016 (arXiv:1612.08083 · 1,577 citations). Source: > The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-t… ## Newest - **[Topology Survives Shuffling: Robust Long-Context Memory Access Prediction for Graph Analytics](https://space-kitty.com/papers/arxiv:2610.39477)**. Dongyan Sun, Neelesh Gupta, Rajgopal Kannan et al. · 2026 (arXiv:2610.39477). Source: - **[Distilling Large-Scale Machine Learning Interatomic Potentials to Predict Long-Timescale Electrolyte Transport Properties](https://space-kitty.com/papers/arxiv:2610.39513)**. Ishan Amin, Yue Jian, Nitesh Kumar et al. · 2026 (arXiv:2610.39513). Source: - **[ROAD: Modern Adversarial Training for Simple One-Step Distillation at Scale](https://space-kitty.com/papers/arxiv:2610.39539)**. Xiaoou Li, Shan You, Tao Huang · 2026 (arXiv:2610.39539). Source: - **[Incentivizing Self-Learning in Large Language Models via Reinforcement Learning](https://space-kitty.com/papers/arxiv:2610.39671)**. Lingkai Kong, Yuzhe Gu, Haiteng Zhao et al. · 2026 (arXiv:2610.39671). Source: - **[Event-SAMA: Language-Grounded Video Object Segmentation on Neuromorphic Event Streams](https://space-kitty.com/papers/arxiv:2610.39676)**. Sergei Fedorchenko, Max Zehnder, Nikola Zubić et al. · 2026 · Conference on Neural Information Processing Systems Workshops (NeurIPSW), 2026 (arXiv:2610.39676). Source: - **[Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models](https://space-kitty.com/papers/arxiv:2610.39854)**. Dmytro Babarytskyi, Oleksandr Babarytskyi · 2026 (arXiv:2610.39854). Source: - **[Turing LLM Post-Training: Calibrated Distillation and Efficient Dynamic-MoE Execution](https://space-kitty.com/papers/arxiv:2610.39876)**. Yue He, Jiahui Hu, Shaman Tang et al. · 2026 (arXiv:2610.39876). Source: - **[TACFormer: Transformer-Based Generative Model for Target-Conditioned De Novo PROTAC Design](https://space-kitty.com/papers/arxiv:2610.39891)**. Yossra Gharbi, Rocío Mercado · 2026 (arXiv:2610.39891). Source: - **[Low-Bit GDN Recurrent-State Quantization: Global Gradient-aware Allocation and Pure-Write Sliding Windows](https://space-kitty.com/papers/arxiv:2610.39931)**. Qinghang Huang, Xuchu Huang, Xunzhao Yin · 2026 (arXiv:2610.39931). Source: - **[Observability-Aligned Evidential Masked Pretraining for Reliable VRU Motion Prediction](https://space-kitty.com/papers/arxiv:2610.39958)**. Shang Gao, Franz Götz-Hahn, Konrad Doll et al. · 2026 (arXiv:2610.39958). Source: --- Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.