# Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection

Preprint · Hengshuai Yao, Xing Chen, Ahmed Murtadha, Guan Wang

- Archive id: arxiv:2603.04427
- Published: 2026-02-16
- arXiv: https://arxiv.org/abs/2603.04427
- Source URL: https://arxiv.org/abs/2603.04427
- PDF: https://arxiv.org/pdf/2603.04427
- Provenance: arxiv

## Abstract

Standard Transformer attention uses identical dimensionality for queries, keys, and values, yet these components serve different roles: queries and keys produce scalar attention weights (selection), while values carry rich representations (value transfer). We show that selection requires only $O(\\log N)$ dimensions to distinguish among $N$ relevant token categories (e.g., syntactic roles, semantic clusters, positional patterns) -- far fewer than value transfer needs. We introduce factored keys, which exploit this asymmetry to physically shrink the KV cache of any pretrained model without retraining from scratch -- unlike Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA), which must be designed into the architecture before pretraining. We factorize each key projection $W\_K \\approx A\_{d \\times r} B\_{r \\times d}$ via truncated singular value decomposition (SVD) (where $r$ is the chosen compression dimension), set $W\_K' = A$ as the new key projection producing compact $r$-dimensional keys for the cache, and absorb $B^\\top$ into the query projection ($W\_Q' = W\_Q B^\\top$) at zero cost -- since queries are never cached. At the 7B scale, training from scratch with $r = d/4$ (where $d$ is the model dimension) matches full-attention perplexity ($9.24$ vs $9.25$ PPL after 20B tokens, mean over two seeds) while using 12% fewer parameters and training 8% faster. For existing models, SVD followed by QK fine-tuning (3 epochs, less than 1% of pretraining data) achieves 75% key cache savings at roughly 2% quality cost on both GPT-2 and Mistral-7B. The approach composes with GQA and quantization for up to $16\\times$ combined key cache compression. For a 7B model serving a 128K context, factored keys save 25 GB of KV cache per user, enabling roughly 60% more concurrent users on identical hardware.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.