# FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework

Preprint · Jianian Zhu, Hang Wu, Haojie Wang, Yinghui Li, Biao Hou, Ruixuan Li, Jidong Zhai

- Archive id: arxiv:2503.08461
- Published: 2025-03-11
- arXiv: https://arxiv.org/abs/2503.08461
- Source URL: https://arxiv.org/abs/2503.08461
- PDF: https://arxiv.org/pdf/2503.08461
- Provenance: arxiv

## Abstract

Multi-modal Large Language Models (MLLMs) serving systems commonly employ KV-cache compression to reduce memory footprint. However, existing compression methods introduce significant processing overhead and queuing delays, particularly in concurrent serving scenarios. We present \\texttt{FastCache}, a novel serving framework that effectively addresses these challenges through two key innovations: (1) a dynamic batching strategy that optimizes request scheduling across prefill, compression, and decode stages, and (2) an efficient KV-cache memory pool mechanism that eliminates memory fragmentation while maintaining high GPU utilization. Our comprehensive experiments on the GQA and MileBench datasets demonstrate that \\texttt{FastCache} achieves up to 19.3$\\times$ reduction in Time-To-First-Token (TTFT) and 12.1$\\times$ improvement in throughput compared to state-of-the-art baselines. The system maintains stable performance under high-concurrency scenarios (up to 40 req/s) while reducing average memory consumption by 20\\%. These results establish \\texttt{FastCache} as an efficient solution for real-world LLM serving systems with KV-cache compression.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.