# PiKV: KV Cache Management System for Mixture of Experts

Preprint · Dong Liu, Yanxuan Yu, Ben Lengerich, Ying Nian Wu

- Archive id: arxiv:2508.06526
- Published: 2025-08-02
- arXiv: https://arxiv.org/abs/2508.06526
- Source URL: https://arxiv.org/abs/2508.06526
- PDF: https://arxiv.org/pdf/2508.06526
- Provenance: arxiv

## Abstract

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce \\textbf{PiKV}, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages \\textit{expert-sharded KV storage} to partition caches across GPUs, \\textit{PiKV routing} to reduce token-to-KV access, and a \\textit{PiKV Scheduling} to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates \\textit{PiKV Compression} modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: \\href{https://github.com/NoakLiu/PiKV}{https://github.com/NoakLiu/PiKV}. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.