# Efficient Training & Inference Domain: Artificial Intelligence · 253 papers indexed · newest 2026-10-28 Quantization, distillation, sparsity, speculative decoding, KV-cache compression and IO-aware kernels that make frontier models cheaper to train and serve. Key concepts: quantization, speculative decoding, KV cache, FlashAttention, knowledge distillation, low-rank adaptation, efficient inference, pruning ## Most influential - **[AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration](https://space-kitty.com/papers/doi:10.1145/3714983.3714987)**. Ji Lin, Jiaming Tang, Haotian Tang et al. · 2025 · GetMobile Mobile Computing and Communications (doi:10.1145/3714983.3714987 · 274 citations). Source: - **[A Survey on Model Compression for Large Language Models](https://space-kitty.com/papers/arxiv:2308.07633)**. Xunyu Zhu, Jian Li, Yong Liu et al. · 2024 · Transactions of the Association for Computational Linguistics (arXiv:2308.07633 · doi:10.1162/tacl_a_00704 · 272 citations). Source: > Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offer… - **[EdgeShard: Efficient LLM Inference via Collaborative Edge Computing](https://space-kitty.com/papers/arxiv:2405.14371)**. Mingjin Zhang, Xiaoming Shen, Jiannong Cao et al. · 2024 · IEEE Internet of Things Journal (arXiv:2405.14371 · doi:10.1109/jiot.2024.3524255 · 229 citations). Source: > Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, … - **[Video (language) modeling: a baseline for generative models of natural videos](https://space-kitty.com/papers/arxiv:1412.6604)**. Marc’Aurelio Ranzato, Arthur D. Szlam, Joan Bruna et al. · 2014 (arXiv:1412.6604 · 325 citations). Source: > We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed from the language modeling literature, and adapted to the vision domain by quantizing the space of image patches into a large dictionary. We demonstrate the approach on both a filling and a generation task. For the first time, we show that, a… - **[SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://space-kitty.com/papers/arxiv:2211.10438)**. Guangxuan Xiao, Ji Lin, Mickaël Seznec et al. · 2022 (arXiv:2211.10438 · 247 citations). Source: > Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization… - **[Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation](https://space-kitty.com/papers/arxiv:2004.09602)**. Hao Wu, Patrick Judd, Xiaojie Zhang et al. · 2020 (arXiv:2004.09602 · 240 citations). Source: > Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain acc… - **[GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://space-kitty.com/papers/arxiv:2210.17323)**. Elias Frantar, Saleh Ashkboos, Torsten Hoefler et al. · 2022 (arXiv:2210.17323 · 197 citations). Source: > Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models… - **[I-BERT: Integer-only BERT Quantization](https://space-kitty.com/papers/arxiv:2101.01321)**. Sehoon Kim, Amir Gholami, Zhewei Yao et al. · 2021 · ICML 2021 (Oral) (arXiv:2101.01321 · 186 citations). Source: > Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we p… - **[A survey on knowledge distillation: Recent advancements](https://space-kitty.com/papers/doi:10.1016/j.mlwa.2024.100605)**. Amir Moslemi, Anna Briskina, Zubeka Dang et al. · 2024 · Machine Learning with Applications (doi:10.1016/j.mlwa.2024.100605 · 112 citations). Source: - **[SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://space-kitty.com/papers/arxiv:2301.00774)**. Elias Frantar, Dan Alistarh · 2023 (arXiv:2301.00774 · 131 citations). Source: > We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these mo… - **[The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures](https://space-kitty.com/papers/arxiv:2104.10640)**. Sushant Kumar Singh, Ausif Mahmood · 2021 · IEEE Access (arXiv:2104.10640 · doi:10.1109/access.2021.3077350 · 158 citations). Source: > In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architecture… - **[Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://space-kitty.com/papers/arxiv:2303.01037)**. Yu Zhang, Han, Wei, James Qin et al. · 2023 (arXiv:2303.01037 · 110 citations). Source: > We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled … - **[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://space-kitty.com/papers/arxiv:2306.00978)**. Ji Lin, Jiaming Tang, Haotian Tang et al. · 2023 (arXiv:2306.00978 · 103 citations). Source: > Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ finds that not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To… - **[Chronos: Learning the Language of Time Series](https://space-kitty.com/papers/arxiv:2403.07815)**. Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Türkmen et al. · 2024 (arXiv:2403.07815 · 85 citations). Source: > We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark c… - **[Measuring and Improving the Energy Efficiency of Large Language Models Inference](https://space-kitty.com/papers/doi:10.1109/access.2024.3409745)**. Mauricio Fadel Argerich, Marta Patiño-Martı́nez · 2024 · IEEE Access (doi:10.1109/access.2024.3409745 · 84 citations). Source: > Recent improvements in the accuracy of machine learning (ML) models in the language domain have propelled their use in a multitude of products and services, touching millions of lives daily. These new levels of accuracy have been attained mainly through exponential growth in model size, creating a new category of models known as Large Language Models (LLMs) and leading to a substantial increase in computing and energy demands. While recent studies have focused on measuring and improving the energy consumption of LLMs during training, inference has received little attention. In this article, we… > _Abstract excerpt by Mauricio Fadel Argerich et al., CC-BY, https://doi.org/10.1109/access.2024.3409745; truncated._ ## Newest - **[Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI](https://space-kitty.com/papers/arxiv:2610.38095)**. NVIDIA, :, Aastha Jhunjhunwala et al. · 2026 (arXiv:2610.38095). Source: - **[The Role of Quantization in Loss Spikes](https://space-kitty.com/papers/arxiv:2610.38277)**. Zaiqian Chen, Aditya Somasundaram, Sophie Li et al. · 2026 (arXiv:2610.38277). Source: - **[Multi-Token Prediction Done Right](https://space-kitty.com/papers/arxiv:2610.38331)**. Jayden Teoh · 2026 (arXiv:2610.38331). Source: - **[FlatCake: A Flat KVCache Storage Architecture for LLM Serving](https://space-kitty.com/papers/arxiv:2610.38355)**. Xin Huang, Teng Ma, Zhuoran Li et al. · 2026 (arXiv:2610.38355). Source: - **[CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained Optimization](https://space-kitty.com/papers/arxiv:2610.38357)**. Zhiyuan Xiang, Xiang Deng, Wanjie Tao et al. · 2026 (arXiv:2610.38357). Source: - **[Towards Reliable, Scalable, and Value-Aligned Generative Recommendation for Large-Scale Advertising](https://space-kitty.com/papers/arxiv:2610.38378)**. Yatong Xiao, Yibo Sheng, Lixiang Wang et al. · 2026 (arXiv:2610.38378). Source: - **[REVBRAIN: Real-time Probabilistic EEG Forecasting Using Residual Vector Quantization](https://space-kitty.com/papers/arxiv:2610.38558)**. Hanna Pankka, Jaakko Lehtinen, Timo Roine · 2026 (arXiv:2610.38558). Source: - **[Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning Models](https://space-kitty.com/papers/arxiv:2610.38681)**. Yecheng Wu, Song Han, Han Cai · 2026 (arXiv:2610.38681). Source: - **[Whole-Body Humanoid Control via Stochastic Motion Tokenization](https://space-kitty.com/papers/arxiv:2610.36452)**. Michael Drolet, Oleg Arenz, Jan Peters · 2026 (arXiv:2610.36452). Source: - **[Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard Kernels](https://space-kitty.com/papers/arxiv:2610.36520)**. Utkarsh Kanwat · 2026 (arXiv:2610.36520). Source: --- Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.