{"topic":"efficient-ai","sort":"top","limit":25,"offset":0,"count":25,"papers":[{"id":"doi:10.1145/3714983.3714987","kind":"article","title":"AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration","authors":["Ji Lin","Jiaming Tang","Haotian Tang","Shang Wen Yang","Guangxuan Xiao","Song Han"],"author_count":6,"year":2025,"published":"2025-01-20","venue":"GetMobile Mobile Computing and Communications","cited_by":274,"doi":"10.1145/3714983.3714987","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1145/3714983.3714987","archive_url":"https://space-kitty.com/papers/doi:10.1145/3714983.3714987","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4406650295","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2308.07633","kind":"article","title":"A Survey on Model Compression for Large Language Models","authors":["Xunyu Zhu","Jian Li","Yong Liu","Can Ma","Weiping Wang"],"author_count":5,"year":2024,"published":"2024-01-01","venue":"Transactions of the Association for Computational Linguistics","cited_by":272,"doi":"10.1162/tacl_a_00704","arxiv_id":"2308.07633","nct_id":null,"source_url":"https://doi.org/10.1162/tacl_a_00704","archive_url":"https://space-kitty.com/papers/arxiv:2308.07633","updated":"2024-07-30","pmid":null,"pmcid":null,"openalex_id":"W4404918643","pdf_url":"https://arxiv.org/pdf/2308.07633","abstract":"Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2405.14371","kind":"article","title":"EdgeShard: Efficient LLM Inference via Collaborative Edge Computing","authors":["Mingjin Zhang","Xiaoming Shen","Jiannong Cao","Zeyang Cui","Shan Jiang"],"author_count":5,"year":2024,"published":"2024-12-31","venue":"IEEE Internet of Things Journal","cited_by":229,"doi":"10.1109/jiot.2024.3524255","arxiv_id":"2405.14371","nct_id":null,"source_url":"https://doi.org/10.1109/jiot.2024.3524255","archive_url":"https://space-kitty.com/papers/arxiv:2405.14371","updated":"2024-05-23","pmid":null,"pmcid":null,"openalex_id":"W4405934565","pdf_url":"https://arxiv.org/pdf/2405.14371","abstract":"Large language models (LLMs) have shown great potential in natural language processing and content generation. However, current LLMs heavily rely on cloud computing, leading to prolonged latency, high bandwidth cost, and privacy concerns. Edge computing is promising to address such concerns by deploying LLMs on edge devices, closer to data sources. Some works try to leverage model quantization to reduce the model size to fit the resource-constraint edge devices, but they lead to accuracy loss. Other works use cloud-edge collaboration, suffering from unstable network connections. In this work, we leverage collaborative edge computing to facilitate the collaboration among edge devices and cloud servers for jointly performing efficient LLM inference. We propose a general framework to partition the LLM model into shards and deploy on distributed devices. To achieve efficient LLM inference, we formulate an adaptive joint device selection and model partition problem and design an efficient dynamic programming algorithm to optimize the inference latency and throughput, respectively. Experiments of Llama2 serial models on a heterogeneous physical prototype demonstrate that EdgeShard achieves up to 50% latency reduction and 2x throughput improvement over baseline methods.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["IoT and Edge/Fog Computing"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:1412.6604","kind":"preprint","title":"Video (language) modeling: a baseline for generative models of natural videos","authors":["Marc’Aurelio Ranzato","Arthur D. Szlam","Joan Bruna","Michaël Mathieu","Ronan Collobert","Sumit Chopra"],"author_count":6,"year":2014,"published":"2014-12-20","venue":null,"cited_by":325,"doi":null,"arxiv_id":"1412.6604","nct_id":null,"source_url":"http://arxiv.org/abs/1412.6604","archive_url":"https://space-kitty.com/papers/arxiv:1412.6604","updated":"2016-05-04","pmid":null,"pmcid":null,"openalex_id":"W1568514080","pdf_url":"https://arxiv.org/pdf/1412.6604","abstract":"We propose a strong baseline model for unsupervised feature learning using video data. By learning to predict missing frames or extrapolate future frames from an input video sequence, the model discovers both spatial and temporal correlations which are useful to represent complex deformations and motion patterns. The models we propose are largely borrowed from the language modeling literature, and adapted to the vision domain by quantizing the space of image patches into a large dictionary. We demonstrate the approach on both a filling and a generation task. For the first time, we show that, after training on natural videos, such a model can predict non-trivial motions over short video sequences.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Generative Adversarial Networks and Image Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2211.10438","kind":"preprint","title":"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models","authors":["Guangxuan Xiao","Ji Lin","Mickaël Seznec","Wu, Hao","Julien Demouth","Song Han"],"author_count":6,"year":2022,"published":"2022-11-18","venue":null,"cited_by":247,"doi":null,"arxiv_id":"2211.10438","nct_id":null,"source_url":"http://arxiv.org/abs/2211.10438","archive_url":"https://space-kitty.com/papers/arxiv:2211.10438","updated":"2024-03-29","pmid":null,"pmcid":null,"openalex_id":"W4309591680","pdf_url":"https://arxiv.org/pdf/2211.10438","abstract":"Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, existing methods cannot maintain accuracy and hardware efficiency at the same time. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node. Our work offers a turn-key solution that reduces hardware costs and democratizes LLMs. Code is available at https://github.com/mit-han-lab/smoothquant.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2004.09602","kind":"preprint","title":"Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation","authors":["Hao Wu","Patrick Judd","Xiaojie Zhang","Mikhail Isaev","Paulius Micikevicius"],"author_count":5,"year":2020,"published":"2020-04-20","venue":null,"cited_by":240,"doi":null,"arxiv_id":"2004.09602","nct_id":null,"source_url":"http://arxiv.org/abs/2004.09602","archive_url":"https://space-kitty.com/papers/arxiv:2004.09602","updated":"2020-04-20","pmid":null,"pmcid":null,"openalex_id":"W3017746288","pdf_url":"https://arxiv.org/pdf/2004.09602","abstract":"Quantization techniques can reduce the size of Deep Neural Networks and improve inference latency and throughput by taking advantage of high throughput integer instructions. In this paper we review the mathematical aspects of quantization parameters and evaluate their choices on a wide range of neural network models for different application domains, including vision, speech, and language. We focus on quantization techniques that are amenable to acceleration by processors with high-throughput integer math pipelines. We also present a workflow for 8-bit quantization that is able to maintain accuracy within 1% of the floating-point baseline on all networks studied, including models that are more difficult to quantize, such as MobileNets and BERT-large.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2210.17323","kind":"preprint","title":"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers","authors":["Elias Frantar","Saleh Ashkboos","Torsten Hoefler","Dan Alistarh"],"author_count":4,"year":2022,"published":"2022-10-31","venue":null,"cited_by":197,"doi":null,"arxiv_id":"2210.17323","nct_id":null,"source_url":"http://arxiv.org/abs/2210.17323","archive_url":"https://space-kitty.com/papers/arxiv:2210.17323","updated":"2023-03-22","pmid":null,"pmcid":null,"openalex_id":"W4307934016","pdf_url":"https://arxiv.org/pdf/2210.17323","abstract":"Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. Moreover, we also show that our method can still provide reasonable accuracy in the extreme quantization regime, in which weights are quantized to 2-bit or even ternary quantization levels. We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation is available at https://github.com/IST-DASLab/gptq.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2101.01321","kind":"preprint","title":"I-BERT: Integer-only BERT Quantization","authors":["Sehoon Kim","Amir Gholami","Zhewei Yao","Michael W. Mahoney","Kurt Keutzer"],"author_count":5,"year":2021,"published":"2021-01-05","venue":"ICML 2021 (Oral)","cited_by":186,"doi":null,"arxiv_id":"2101.01321","nct_id":null,"source_url":"http://arxiv.org/abs/2101.01321","archive_url":"https://space-kitty.com/papers/arxiv:2101.01321","updated":"2021-06-08","pmid":null,"pmcid":null,"openalex_id":"W3118568258","pdf_url":"https://arxiv.org/pdf/2101.01321","abstract":"Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has been open-sourced.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1016/j.mlwa.2024.100605","kind":"article","title":"A survey on knowledge distillation: Recent advancements","authors":["Amir Moslemi","Anna Briskina","Zubeka Dang","Jason H. Li"],"author_count":4,"year":2024,"published":"2024-11-10","venue":"Machine Learning with Applications","cited_by":112,"doi":"10.1016/j.mlwa.2024.100605","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1016/j.mlwa.2024.100605","archive_url":"https://space-kitty.com/papers/doi:10.1016/j.mlwa.2024.100605","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4404212170","pdf_url":null,"abstract":null,"abstract_license":null,"license":"cc-by-nc-nd","is_open_access":true,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2301.00774","kind":"preprint","title":"SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot","authors":["Elias Frantar","Dan Alistarh"],"author_count":2,"year":2023,"published":"2023-01-02","venue":null,"cited_by":131,"doi":null,"arxiv_id":"2301.00774","nct_id":null,"source_url":"http://arxiv.org/abs/2301.00774","archive_url":"https://space-kitty.com/papers/arxiv:2301.00774","updated":"2023-03-22","pmid":null,"pmcid":null,"openalex_id":"W4313484599","pdf_url":"https://arxiv.org/pdf/2301.00774","abstract":"We show for the first time that large-scale generative pretrained transformer (GPT) family models can be pruned to at least 50% sparsity in one-shot, without any retraining, at minimal loss of accuracy. This is achieved via a new pruning method called SparseGPT, specifically designed to work efficiently and accurately on massive GPT-family models. We can execute SparseGPT on the largest available open-source models, OPT-175B and BLOOM-176B, in under 4.5 hours, and can reach 60% unstructured sparsity with negligible increase in perplexity: remarkably, more than 100 billion weights from these models can be ignored at inference time. SparseGPT generalizes to semi-structured (2:4 and 4:8) patterns, and is compatible with weight quantization approaches. The code is available at: https://github.com/IST-DASLab/sparsegpt.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2104.10640","kind":"article","title":"The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures","authors":["Sushant Kumar Singh","Ausif Mahmood"],"author_count":2,"year":2021,"published":"2021-01-01","venue":"IEEE Access","cited_by":158,"doi":"10.1109/access.2021.3077350","arxiv_id":"2104.10640","nct_id":null,"source_url":"https://doi.org/10.1109/access.2021.3077350","archive_url":"https://space-kitty.com/papers/arxiv:2104.10640","updated":"2021-04-24","pmid":null,"pmcid":null,"openalex_id":"W3163832451","pdf_url":"https://arxiv.org/pdf/2104.10640","abstract":"In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architectures have utilized concepts of transfer learning, pruning, quantization, and knowledge distillation to achieve moderate model sizes while keeping nearly similar performances as achieved by their predecessors. Additionally, to mitigate the data size challenge raised by language models from a knowledge extraction perspective, Knowledge Retrievers have been built to extricate explicit data documents from a large corpus of databases with greater efficiency and accuracy. Recent research has also focused on superior inference by providing efficient attention to longer input sequences. In this paper, we summarize and examine the current state-of-the-art (SOTA) NLP models that have been employed for numerous NLP tasks for optimal performance and efficiency. We provide a detailed understanding and functioning of the different architectures, a taxonomy of NLP designs, comparative evaluations, and future directions in NLP.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by-nc-nd","is_open_access":true,"peer_reviewed":null,"categories":["Topic Modeling"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2303.01037","kind":"preprint","title":"Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages","authors":["Yu Zhang","Han, Wei","James Qin","Yongqiang Wang","Ankur Bapna","Zhehuai Chen","Nanxin Chen","Li, Bo","Vera Axelrod","Gary Wang","Zhong Meng","Ke Hu","Rosenberg, Andrew","Rohit Prabhavalkar","Daniel Park","Parisa Haghani","Jason Riesa","Ginger Perng","Hagen Soltau","Trevor Strohman","Bhuvana Ramabhadran","Tara N. Sainath","Moreno, Pedro","Chung‐Cheng Chiu","Johan Schalkwyk"],"author_count":27,"year":2023,"published":"2023-03-02","venue":null,"cited_by":110,"doi":null,"arxiv_id":"2303.01037","nct_id":null,"source_url":"http://arxiv.org/abs/2303.01037","archive_url":"https://space-kitty.com/papers/arxiv:2303.01037","updated":"2023-09-25","pmid":null,"pmcid":null,"openalex_id":"W4323066695","pdf_url":"https://arxiv.org/pdf/2303.01037","abstract":"We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"public-domain","is_open_access":true,"peer_reviewed":false,"categories":["Speech Recognition and Synthesis"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2306.00978","kind":"preprint","title":"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration","authors":["Ji Lin","Jiaming Tang","Haotian Tang","Yang Shang","Wei-Ming Chen","Wei-Chen Wang","Guangxuan Xiao","Dang, Xingyu","Chuang Gan","Song Han"],"author_count":10,"year":2023,"published":"2023-06-01","venue":null,"cited_by":103,"doi":null,"arxiv_id":"2306.00978","nct_id":null,"source_url":"https://arxiv.org/abs/2306.00978","archive_url":"https://space-kitty.com/papers/arxiv:2306.00978","updated":"2026-04-25","pmid":null,"pmcid":null,"openalex_id":"W4379260375","pdf_url":"https://arxiv.org/pdf/2306.00978","abstract":"Large language models (LLMs) have transformed numerous AI applications. On-device LLM is becoming increasingly important: running LLMs locally on edge devices can reduce the cloud computing cost and protect users' privacy. However, the astronomical model size and the limited hardware resource pose significant deployment challenges. We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ finds that not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error. To identify salient weight channels, we should refer to the activation distribution, not weights. To avoid the hardware-inefficient mix-precision quantization, we mathematically derive that scaling up the salient channels can reduce the quantization error. AWQ employs an equivalent transformation to scale the salient weight channels to protect them. The scale is determined by collecting the activation statistics offline. AWQ does not rely on any backpropagation or reconstruction, so it generalizes to different domains and modalities without overfitting the calibration set. AWQ outperforms existing work on various language modeling and domain-specific benchmarks (coding and math). Thanks to better generalization, it achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. Alongside AWQ, we implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs. With kernel fusion and platform-aware weight packing, TinyChat offers more than 3x speedup over the Huggingface FP16 implementation on both desktop and mobile GPUs. It also democratizes the deployment of the 70B Llama-2 model on mobile GPUs.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2403.07815","kind":"preprint","title":"Chronos: Learning the Language of Time Series","authors":["Abdul Fatir Ansari","Lorenzo Stella","Ali Caner Türkmen","Xiyuan Zhang","Mercado, Pedro","Huibin Shen","Oleksandr Shchur","Syama Sundar Rangapuram","Sebastian Pineda Arango","Shubham Kapoor","Jasper Zschiegner","Danielle C. Maddix","Hao Wang","Michael W. Mahoney","Kari Torkkola","Andrew Wilson","Michael Bohlke‐Schneider","Wang, Yuyang"],"author_count":18,"year":2024,"published":"2024-03-12","venue":null,"cited_by":85,"doi":null,"arxiv_id":"2403.07815","nct_id":null,"source_url":"http://arxiv.org/abs/2403.07815","archive_url":"https://space-kitty.com/papers/arxiv:2403.07815","updated":"2024-11-04","pmid":null,"pmcid":null,"openalex_id":"W4392781393","pdf_url":"https://arxiv.org/pdf/2403.07815","abstract":"We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"public-domain","is_open_access":true,"peer_reviewed":false,"categories":["Time Series Analysis and Forecasting"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1109/access.2024.3409745","kind":"article","title":"Measuring and Improving the Energy Efficiency of Large Language Models Inference","authors":["Mauricio Fadel Argerich","Marta Patiño-Martı́nez"],"author_count":2,"year":2024,"published":"2024-01-01","venue":"IEEE Access","cited_by":84,"doi":"10.1109/access.2024.3409745","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/access.2024.3409745","archive_url":"https://space-kitty.com/papers/doi:10.1109/access.2024.3409745","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4399449628","pdf_url":null,"abstract":"Recent improvements in the accuracy of machine learning (ML) models in the language domain have propelled their use in a multitude of products and services, touching millions of lives daily. These new levels of accuracy have been attained mainly through exponential growth in model size, creating a new category of models known as Large Language Models (LLMs) and leading to a substantial increase in computing and energy demands. While recent studies have focused on measuring and improving the energy consumption of LLMs during training, inference has received little attention. In this article, we present an approach to profile the energy consumption of LLMs during inference and leverage it to improve energy efficiency. For this, we deploy several state-of-the-art LLMs and observe how model size, number of layers, parallelized attention, and even vocabulary size affect their energy consumption. In addition, we leverage input batch size and different quantization levels to optimize their inference energy efficiency and latency.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Green IT and Sustainability"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Measuring and Improving the Energy Efficiency of Large Language Models Inference\" by Mauricio Fadel Argerich, Marta Patiño-Martı́nez, https://doi.org/10.1109/access.2024.3409745, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:2308.10462","kind":"article","title":"Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models","authors":["Martin Weyssow","Xin Zhou","Kisub Kim","David Lo","Houari A. Sahraoui"],"author_count":5,"year":2025,"published":"2025-01-22","venue":"ACM Transactions on Software Engineering and Methodology","cited_by":69,"doi":"10.1145/3714461","arxiv_id":"2308.10462","nct_id":null,"source_url":"https://doi.org/10.1145/3714461","archive_url":"https://space-kitty.com/papers/arxiv:2308.10462","updated":"2024-12-27","pmid":null,"pmcid":null,"openalex_id":"W4406688098","pdf_url":"https://arxiv.org/pdf/2308.10462","abstract":"Large language models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in a zero-shot manner, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored in-context learning (ICL) and retrieval-augmented generation (RAG) as strategies to guide the LLM generative process with task-specific prompt examples. However, ICL and RAG introduce inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee parameter-efficient fine-tuning (PEFT) as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs in the context of automated code generation. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL and RAG across a diverse set of LLMs and three representative Python code generation datasets: Conala, CodeAlpacaPy, and APPS. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at https://github.com/martin-wey/peft-llm-code/.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Software Engineering Research"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.3390/app14177445","kind":"article","title":"Advances in the Neural Network Quantization: A Comprehensive Review","authors":["Wei Lu","Zhong Ma","Chaojie Yang","Qin Yao"],"author_count":4,"year":2024,"published":"2024-08-23","venue":"Applied Sciences","cited_by":78,"doi":"10.3390/app14177445","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.3390/app14177445","archive_url":"https://space-kitty.com/papers/doi:10.3390/app14177445","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4401889011","pdf_url":"https://www.mdpi.com/2076-3417/14/17/7445/pdf?version=1724397266","abstract":"Artificial intelligence technologies based on deep convolutional neural networks and large language models have made significant breakthroughs in many tasks, such as image recognition, target detection, semantic segmentation, and natural language processing, but also face a conflict between the high computational capacity of the algorithms and limited deployment resources. Quantization, which converts floating-point neural networks into low-bit-width integer networks, is an important and essential technique for efficient deployment and cost reduction in edge computing. This paper analyzes various existing quantization methods, showcases the deployment accuracy of advanced techniques, and discusses the future challenges and trends in this domain.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Advances in the Neural Network Quantization: A Comprehensive Review\" by Wei Lu, Zhong Ma, Chaojie Yang et al., https://doi.org/10.3390/app14177445, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"doi:10.1109/twc.2024.3497923","kind":"article","title":"Beyond the Cloud: Edge Inference for Generative Large Language Models in Wireless Networks","authors":["Xinyuan Zhang","Jiangtian Nie","Yudong Huang","Gaochang Xie","Zehui Xiong","Jiang Liu","Dusit Tao Niyato","Xuemin Shen"],"author_count":8,"year":2024,"published":"2024-11-20","venue":"IEEE Transactions on Wireless Communications","cited_by":77,"doi":"10.1109/twc.2024.3497923","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/twc.2024.3497923","archive_url":"https://space-kitty.com/papers/doi:10.1109/twc.2024.3497923","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4404562740","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["IoT and Edge/Fog Computing"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.2139/ssrn.4853709","kind":"article","title":"Generative Ai and Large Language Models for Cyber Security: All Insights You Need","authors":["Mohamed Amine Ferrag","Fatima Alwahedi","Ammar Ayman Battah","Bilel Cherif","Abdechakour Mechri","Norbert Tihanyi"],"author_count":6,"year":2024,"published":"2024-01-01","venue":"SSRN Electronic Journal","cited_by":77,"doi":"10.2139/ssrn.4853709","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.2139/ssrn.4853709","archive_url":"https://space-kitty.com/papers/doi:10.2139/ssrn.4853709","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4399338566","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Adversarial Robustness in Machine Learning"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"doi:10.1109/tcad.2024.3483089","kind":"article","title":"RTLCoder: Fully Open-Source and Efficient LLM-Assisted RTL Code Generation Technique","authors":["Shang Liu","Wenji Fang","Yao Lu","Jing Wang","Qijun Zhang","Hongce Zhang","Zhiyao Xie"],"author_count":7,"year":2024,"published":"2024-10-17","venue":"IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems","cited_by":75,"doi":"10.1109/tcad.2024.3483089","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1109/tcad.2024.3483089","archive_url":"https://space-kitty.com/papers/doi:10.1109/tcad.2024.3483089","updated":null,"pmid":null,"pmcid":null,"openalex_id":"W4403511190","pdf_url":null,"abstract":null,"abstract_license":null,"license":null,"is_open_access":false,"peer_reviewed":null,"categories":["Embedded Systems Design Techniques"],"details":null,"provenance":"openalex","attribution":"Metadata and citation counts: OpenAlex (CC0)."},{"id":"arxiv:2502.07855","kind":"article","title":"Vision-Language Models for Edge Networks: A Comprehensive Survey","authors":["Ahmed Sharshar","Latif Ullah Khan","Waseem Ullah","Mohsen Guizani"],"author_count":4,"year":2025,"published":"2025-06-12","venue":"IEEE Internet of Things Journal","cited_by":61,"doi":"10.1109/jiot.2025.3579032","arxiv_id":"2502.07855","nct_id":null,"source_url":"https://doi.org/10.1109/jiot.2025.3579032","archive_url":"https://space-kitty.com/papers/arxiv:2502.07855","updated":"2025-06-13","pmid":null,"pmcid":null,"openalex_id":"W4411232024","pdf_url":"https://arxiv.org/pdf/2502.07855","abstract":"Vision Large Language Models (VLMs) combine visual understanding with natural language processing, enabling tasks like image captioning, visual question answering, and video analysis. While VLMs show impressive capabilities across domains such as autonomous vehicles, smart surveillance, and healthcare, their deployment on resource-constrained edge devices remains challenging due to processing power, memory, and energy limitations. This survey explores recent advancements in optimizing VLMs for edge environments, focusing on model compression techniques, including pruning, quantization, knowledge distillation, and specialized hardware solutions that enhance efficiency. We provide a detailed discussion of efficient training and fine-tuning methods, edge deployment challenges, and privacy considerations. Additionally, we discuss the diverse applications of lightweight VLMs across healthcare, environmental monitoring, and autonomous systems, illustrating their growing impact. By highlighting key design strategies, current challenges, and offering recommendations for future directions, this survey aims to inspire further research into the practical deployment of VLMs, ultimately making advanced AI accessible in resource-limited settings.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.3390/s25051318","kind":"article","title":"Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions","authors":["Ismail Lamaakal","Yassine Maleh","Khalid El Makkaoui","Ibrahim Ouahbi","Paweł Pławiak","Osama Alfarraj","May Almousa","Ahmed A. Abd El‐Latif"],"author_count":8,"year":2025,"published":"2025-02-21","venue":"Sensors","cited_by":60,"doi":"10.3390/s25051318","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.3390/s25051318","archive_url":"https://space-kitty.com/papers/doi:10.3390/s25051318","updated":null,"pmid":"40096098","pmcid":null,"openalex_id":"W4407809258","pdf_url":"https://www.mdpi.com/1424-8220/25/5/1318/pdf?version=1740130455","abstract":"Large Language Models (LLMs), like GPT and BERT, have significantly advanced Natural Language Processing (NLP), enabling high performance on complex tasks. However, their size and computational needs make LLMs unsuitable for deployment on resource-constrained devices, where efficiency, speed, and low power consumption are critical. Tiny Language Models (TLMs), also known as BabyLMs, offer compact alternatives by using advanced compression and optimization techniques to function effectively on devices such as smartphones, Internet of Things (IoT) systems, and embedded platforms. This paper provides a comprehensive survey of TLM architectures and methodologies, including key techniques such as knowledge distillation, quantization, and pruning. Additionally, it explores potential and emerging applications of TLMs in automation and control, covering areas such as edge computing, IoT, industrial automation, and healthcare. The survey discusses challenges unique to TLMs, such as trade-offs between model size and accuracy, limited generalization, and ethical considerations in deployment. Future research directions are also proposed, focusing on hybrid compression techniques, application-specific adaptations, and context-aware TLMs optimized for hardware-specific constraints. This paper aims to serve as a foundational resource for advancing TLMs capabilities across diverse real-world applications.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions\" by Ismail Lamaakal, Yassine Maleh, Khalid El Makkaoui et al., https://doi.org/10.3390/s25051318, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."},{"id":"arxiv:1802.00150","kind":"preprint","title":"Alternating Multi-bit Quantization for Recurrent Neural Networks","authors":["Chen Xu","Jianqiang Yao","Zhouchen Lin","Wenwu Ou","Yuanbin Cao","Zhirong Wang","Hongbin Zha"],"author_count":7,"year":2018,"published":"2018-02-01","venue":null,"cited_by":86,"doi":null,"arxiv_id":"1802.00150","nct_id":null,"source_url":"http://arxiv.org/abs/1802.00150","archive_url":"https://space-kitty.com/papers/arxiv:1802.00150","updated":"2018-02-01","pmid":null,"pmcid":null,"openalex_id":"W2786951478","pdf_url":"https://arxiv.org/pdf/1802.00150","abstract":"Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied. We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration. Both results beat the exiting quantization works with large margins. We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves excellent performance.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":false,"categories":["Advanced Neural Network Applications"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"arxiv:2310.08279","kind":"article","title":"Enhancing text-based knowledge graph completion with zero-shot large language models: A focus on semantic enhancement","authors":["Rui Yang","Jiahao Zhu","Jianping Man","Fang Li","Yi Jun Zhou"],"author_count":5,"year":2024,"published":"2024-06-24","venue":"Knowledge-Based Systems","cited_by":63,"doi":"10.1016/j.knosys.2024.112155","arxiv_id":"2310.08279","nct_id":null,"source_url":"https://doi.org/10.1016/j.knosys.2024.112155","archive_url":"https://space-kitty.com/papers/arxiv:2310.08279","updated":"2024-06-27","pmid":null,"pmcid":null,"openalex_id":"W4399957366","pdf_url":"https://arxiv.org/pdf/2310.08279","abstract":"The design and development of text-based knowledge graph completion (KGC) methods leveraging textual entity descriptions are at the forefront of research. These methods involve advanced optimization techniques such as soft prompts and contrastive learning to enhance KGC models. The effectiveness of text-based methods largely hinges on the quality and richness of the training data. Large language models (LLMs) can utilize straightforward prompts to alter text data, thereby enabling data augmentation for KGC. Nevertheless, LLMs typically demand substantial computational resources. To address these issues, we introduce a framework termed constrained prompts for KGC (CP-KGC). This CP-KGC framework designs prompts that adapt to different datasets to enhance semantic richness. Additionally, CP-KGC employs a context constraint strategy to effectively identify polysemous entities within KGC datasets. Through extensive experimentation, we have verified the effectiveness of this framework. Even after quantization, the LLM (Qwen-7B-Chat-int4) still enhances the performance of text-based KGC methods \\footnote{Code and datasets are available at \\href{https://github.com/sjlmg/CP-KGC}{https://github.com/sjlmg/CP-KGC}}. This study extends the performance limits of existing models and promotes further integration of KGC with LLMs.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":true,"peer_reviewed":null,"categories":["Advanced Graph Neural Networks"],"details":null,"provenance":"openalex+arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv."},{"id":"doi:10.1038/s43856-025-00808-8","kind":"article","title":"Comprehensive testing of large language models for extraction of structured data in pathology","authors":["Bastian Grothey","Jan Odenkirchen","Adnan Brkic","Birgid Schömig‐Markiefka","Alexander Quaas","Reinhard Büttner","Yuri Tolkach"],"author_count":7,"year":2025,"published":"2025-03-31","venue":"Communications Medicine","cited_by":54,"doi":"10.1038/s43856-025-00808-8","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1038/s43856-025-00808-8","archive_url":"https://space-kitty.com/papers/doi:10.1038/s43856-025-00808-8","updated":null,"pmid":"40164789","pmcid":null,"openalex_id":"W4409011223","pdf_url":null,"abstract":"BACKGROUND: Pathology departments generate large volumes of unstructured data as free-text diagnostic reports. Converting these reports into structured formats for analytics or artificial intelligence projects requires substantial manual effort by specialized personnel. While recent studies show promise in using advanced language models for structuring pathology data, they primarily rely on proprietary models, raising cost and privacy concerns. Additionally, important aspects such as prompt engineering and model quantization for deployment on consumer-grade hardware remain unaddressed. METHODS: We created a dataset of 579 annotated pathology reports in German and English versions. Six language models (proprietary: GPT-4; open-source: Llama2 13B, Llama2 70B, Llama3 8B, Llama3 70B, and Qwen2.5 7B) were evaluated for their ability to extract eleven key parameters from these reports. Additionally, we investigated model performance across different prompt engineering strategies and model quantization techniques to assess practical deployment scenarios. RESULTS: Here we show that open-source language models extract structured data from pathology reports with high precision, matching the accuracy of proprietary GPT-4 model. The precision varies significantly across different models and configurations. These variations depend on specific prompt engineering strategies and quantization methods used during model deployment. CONCLUSIONS: Open-source language models demonstrate comparable performance to proprietary solutions in structuring pathology report data. This finding has significant implications for healthcare institutions seeking cost-effective, privacy-preserving data structuring solutions. The variations in model performance across different configurations provide valuable insights for practical deployment in pathology departments. Our publicly available bilingual dataset serves as both a benchmark and a resource for future research.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"peer_reviewed":null,"categories":["Machine Learning in Healthcare"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Comprehensive testing of large language models for extraction of structured data in pathology\" by Bastian Grothey, Jan Odenkirchen, Adnan Brkic et al., https://doi.org/10.1038/s43856-025-00808-8, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified."}],"next":"https://space-kitty.com/api/v1/topics/efficient-ai/papers?sort=top&limit=25&offset=25"}