LLM Architectures & Pretraining
How frontier language models are built: transformer variants, mixture-of-experts, state-space and hybrid models, long-context attention, tokenization and pretraining recipes.
LLM Architectures & Pretraining: State of the Field (October 2026)
How frontier language models are built and pretrained as of October 2026: sparse mixture-of-experts scaling, the competing full, sparse-compressed and hybrid linear attention designs, million-token context, optimizers, low-precision training, residual-stream redesigns, data and scaling laws. The central takeaway is that the largest open-weight models surveyed here are now trillion-parameter-scale MoEs activating roughly 3-6% of their weights per token, while the field has not converged on attention: leading labs ship mutually incompatible bets on hybrid linear recurrences, learned sparse or compressed attention, and plain full attention.
Most influential
- Language Models are Few-Shot Learners
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perf…
Preprint23,014 citationsarXiv:2005.14165 - A Survey of Large Language Models
Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from sta…
Article2,085 citationsarXiv:2303.18223doi:10.1007/s11704-026-60308-3 - BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. How…
Article7,208 citationsarXiv:1901.08746doi:10.1093/bioinformatics/btz682 - Transformers in Vision: A Survey
Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long s…
Article3,593 citationsarXiv:2101.01169doi:10.1145/3505244 - LoRA: Low-Rank Adaptation of Large Language Models
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, eac…
Preprint2,501 citationsarXiv:2106.09685 - A Comprehensive Overview of Large Language Models
Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tunin…
Article1,255 citationsarXiv:2307.06435doi:10.1145/3744746 - ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain …
Preprint2,111 citationsarXiv:1908.02265 - A Survey of the Usages of Deep Learning for Natural Language Processing
Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This survey provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of r…
Review1,836 citationsarXiv:1807.10854doi:10.1109/tnnls.2020.2979670 - A Critical Review of Recurrent Neural Networks for Sequence Learning
Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. Interactive tasks, such as …
Preprint2,074 citationsarXiv:1506.00019 - Scaling Laws for Neural Language Models
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range.…
Preprint1,751 citationsarXiv:2001.08361 - Flamingo: a Visual Language Model for Few-Shot Learning
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models…
Preprint1,555 citationsarXiv:2204.14198 - Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained languag…
Article1,386 citationsarXiv:2111.01243doi:10.1145/3605943 - A State-of-the-Art Survey on Deep Learning Theory and Architectures
In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly and has been applied to most traditional application domains, as well as some new areas that present more opportunities. Different methods have been proposed based on different categories of learning,…
Abstract excerpt by Md Zahangir Alom et al., CC-BY, https://doi.org/10.3390/electronics8030292; truncated.
Article1,622 citationsdoi:10.3390/electronics8030292 - A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and ChallengesArticle933 citationsdoi:10.1109/access.2024.3365742
- Language Modeling with Gated Convolutional Networks
The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose…
Preprint1,577 citationsarXiv:1612.08083
Newest work
- Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven OptimizationPreprintarXiv:2610.36538
- MM-Net: A Multimodal Multi-Task Network for Sleep Staging and Sleep Apnea Assessment After Ischemic StrokePreprintarXiv:2610.36583
- Balanced State Subspaces for Mamba-2 State CompressionPreprintarXiv:2610.36617
- Does LLM Pre-Training Typically Occur at the Edge of Stability?PreprintarXiv:2610.36778
- Select-TTT: Query-Conditioned Selective Fast Weight Updates for In-Place Test-Time TrainingPreprintarXiv:2610.36794
- TAPESTRY: A Single Backbone for Video Generation and Robot ControlPreprintarXiv:2610.36823
- GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI AgentsPreprintarXiv:2610.36877
- PREEN: Exemplar-Based Color Style Transfer with Smartphone Editor PresetsPreprintarXiv:2610.36896
- DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language ModelsPreprintarXiv:2610.36950
- Zooming without Zooming v2: From Seeing Details to Connecting Evidence for Generalizable Multimodal PerceptionPreprintarXiv:2610.37037