{"id":"arxiv:2506.11418","kind":"preprint","title":"CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering","authors":["Jie Hu","Shengnan Wang","Yutong He","Ping Gong","Jiawei Yi","Juncheng Zhang","Youhui Bai","Renhai Chen","Gong Zhang","Cheng Li","Kun Yuan"],"author_count":11,"year":2025,"published":"2025-06-13","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2506.11418","nct_id":null,"source_url":"https://arxiv.org/abs/2506.11418","archive_url":"https://space-kitty.com/papers/arxiv:2506.11418","updated":"2026-06-14","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2506.11418","abstract":"Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for long-context LLMs poses significant deployment challenges. Existing approaches either discard potentially critical information needed for future generations or offer limited efficiency gains due to high computational overhead. In this paper, we introduce CentroidKV, a simple yet effective framework for online KV cache clustering. Our approach is based on the observation that key states exhibit high similarity along the sequence dimension. To enable efficient clustering, we divide the sequence into chunks and propose Chunked Soft Matching, which employs an alternating partition strategy within each chunk and identifies clusters based on similarity. CentroidKV then merges the KV cache within each cluster into a single centroid. Additionally, we provide a theoretical analysis of the computational complexity and the optimality of the intra-chunk partitioning strategy. Extensive experiments across various models and long-context benchmarks demonstrate that CentroidKV achieves up to 75% reduction in KV cache memory usage while maintaining comparable model performance. Moreover, with minimal computational overhead, CentroidKV accelerates the decoding stage of inference by up to $1.92\\times$ and increases the serving throughput by up to $4\\times$.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}