{"id":"arxiv:2608.02901","kind":"preprint","title":"AnchorKV: Anchor-Residual KV Cache Compression","authors":["Malik Khalaf","Yara Shamshoum","Nitzan Hodos","Yuval Sieradzki","Assaf Schuster"],"author_count":5,"year":2026,"published":"2026-08-03","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2608.02901","nct_id":null,"source_url":"https://arxiv.org/abs/2608.02901","archive_url":"https://space-kitty.com/papers/arxiv:2608.02901","updated":"2026-08-03","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2608.02901","abstract":"The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by $20\\times$ without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}