Efficient Training & Inference
Quantization, distillation, sparsity, speculative decoding, KV-cache compression and IO-aware kernels that make frontier models cheaper to train and serve.
Most influential
- AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and AccelerationArticle274 citationsdoi:10.1145/3714983.3714987
- A Survey on Model Compression for Large Language Models
Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniq…
Article272 citationsarXiv:2308.07633doi:10.1162/tacl_a_00704 - EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some work…
Article229 citationsarXiv:2405.14371doi:10.1109/jiot.2024.3524255 - Video (language) modeling: a baseline for generative models of natural videos
We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed fr…
Preprint325 citationsarXiv:1412.6604 - SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) …
Preprint247 citationsarXiv:2211.10438 - Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, includi…
Preprint240 citationsarXiv:2004.09602 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, whi…
Preprint197 citationsarXiv:2210.17323 - I-BERT: Integer-only BERT Quantization
Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quanti…
Preprint186 citationsarXiv:2101.01321 - A survey on knowledge distillation: Recent advancementsArticle112 citationsdoi:10.1016/j.mlwa.2024.100605
- SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can exe…
Preprint131 citationsarXiv:2301.00774 - The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Trans…
Article158 citationsarXiv:2104.10640doi:10.1109/access.2021.3077350 - Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual p…
Preprint110 citationsarXiv:2303.01037 - AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-awar…
Preprint103 citationsarXiv:2306.00978 - Chronos: Learning the Language of Time Series
We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on …
Preprint85 citationsarXiv:2403.07815 - Measuring and Improving the Energy Efficiency of Large Language Models Inference
Recent improvements in the accuracy of machine learning (ML) models in the language domain have propelled their use in a multitude of products and services, touching millions of lives daily. These new levels of accuracy have been attained mainly through exponential growth in model size, creating a new category of models known as Large Language Models (LLMs) …
Abstract excerpt by Mauricio Fadel Argerich et al., CC-BY, https://doi.org/10.1109/access.2024.3409745; truncated.
Article84 citationsdoi:10.1109/access.2024.3409745
Newest work
- Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AIPreprintarXiv:2610.38095
- The Role of Quantization in Loss SpikesPreprintarXiv:2610.38277
- Multi-Token Prediction Done RightPreprintarXiv:2610.38331
- FlatCake: A Flat KVCache Storage Architecture for LLM ServingPreprintarXiv:2610.38355
- CSO: Refining Robotic Policies via Skill Distribution Alignment and Skill-Grained OptimizationPreprintarXiv:2610.38357
- Towards Reliable, Scalable, and Value-Aligned Generative Recommendation for Large-Scale AdvertisingPreprintarXiv:2610.38378
- REVBRAIN: Real-time Probabilistic EEG Forecasting Using Residual Vector QuantizationPreprintarXiv:2610.38558
- Weaken to Strengthen: Weakening-Induced On-Policy Self-Distillation for Large Reasoning ModelsPreprintarXiv:2610.38681
- Whole-Body Humanoid Control via Stochastic Motion TokenizationPreprintarXiv:2610.36452
- Folding Attention Biases into Keys Makes Compacted KV Caches Run on Standard KernelsPreprintarXiv:2610.36520