{"id":"arxiv:2609.09768","kind":"preprint","title":"Fine-Tuning a KV Cache Concatenation-Aware Model or Recomputing KV Caches? Why Not Both?","authors":["Fumihiko Tachibana","Daisuke Miyashita","Jun Deguchi"],"author_count":3,"year":2026,"published":"2026-09-09","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2609.09768","nct_id":null,"source_url":"https://arxiv.org/abs/2609.09768","archive_url":"https://space-kitty.com/papers/arxiv:2609.09768","updated":"2026-09-09","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2609.09768","abstract":"In Retrieval-Augmented Generation (RAG) systems, a large number of retrieved chunks are concatenated to form the input context so that users can receive high-quality responses based on external knowledge. As a result, the input context length increases substantially, leading to a larger prefill workload and, in turn, a longer time to first token (TTFT). While previous works that reuse precomputed key-value (KV) caches effectively reduce TTFT for long-context inputs, it remains unclear whether response quality is preserved when the input context becomes very long. In this paper, we propose a combined approach that (i) fine-tunes the model while taking KV cache concatenation into account and (ii) selectively recomputes a subset of the KV caches. By applying both techniques, we demonstrate improved accuracy for long-context inputs. Experiments on the RULER benchmark show that, for a 124k-token input, our method improves the RULER score by 9.7 point over the baseline that recomputes KV caches only. Moreover, TTFT is reduced by 80% compared with full attention.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}