# Lossless KV Cache Compression to 2%

Preprint · Zhen Yang, J. N. Han, Kan Wu, Ruobing Xie, An Wang, Xingwu Sun, Zhanhui Kang

- Archive id: arxiv:2410.15252
- Published: 2024-10-20
- arXiv: https://arxiv.org/abs/2410.15252
- Citations: 0
- Source URL: https://arxiv.org/abs/2410.15252
- PDF: https://arxiv.org/pdf/2410.15252
- Provenance: arxiv

## Abstract

Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up inference, maintaining a key-value (KV) cache memory is essential. Nonetheless, the growing demands for KV cache memory create significant hurdles for efficient implementation. This work introduces a novel architecture, Cross-Layer Latent Attention (CLLA), aimed at compressing the KV cache to less than 2% of its original size while maintaining comparable performance levels. CLLA integrates multiple aspects of KV cache compression, including attention head/dimension reduction, layer sharing, and quantization techniques, into a cohesive framework. Our extensive experiments demonstrate that CLLA achieves lossless performance on most tasks while utilizing minimal KV cache, marking a significant advancement in practical KV cache compression.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Efficient Training & Inference](https://space-kitty.com/topics/efficient-ai)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.