# A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

Article · Alessio Devoto, Yu Zhao, Simone Scardapane, Pasquale Minervini

- Archive id: arxiv:2406.11430
- Published: 2024-06-17
- Venue: Conference on Empirical Methods in Natural Language Processing (EMNLP)
- DOI: https://doi.org/10.18653/v1/2024.emnlp-main.1027
- arXiv: https://arxiv.org/abs/2406.11430
- Citations: 6
- Source URL: https://arxiv.org/abs/2406.11430
- PDF: https://arxiv.org/pdf/2406.11430
- Provenance: arxiv

## Abstract

The deployment of large language models (LLMs) is often hindered by the extensive memory requirements of the Key-Value (KV) cache, especially as context lengths increase. Existing approaches to reduce the KV cache size involve either fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce the sequence length. We analyse the attention distributions in decoder-only Transformers-based models and observe that attention allocation patterns stay consistent across most layers. Surprisingly, we find a clear correlation between the $L\_2$ and the attention scores over cached KV pairs, where a low $L\_2$ of a key embedding usually leads to a high attention score during decoding. This finding indicates that the influence of a KV pair is potentially determined by the key embedding itself before being queried. Based on this observation, we compress the KV cache based on the $L\_2$ of key embeddings. Our experimental results show that this simple strategy can reduce the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy. Moreover, without relying on the attention scores, this approach remains compatible with FlashAttention, enabling broader applicability.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Efficient Training & Inference](https://space-kitty.com/topics/efficient-ai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.