# Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding

Preprint · Aayush Gautam, Susav Shrestha, Narasimha Reddy

- Archive id: arxiv:2504.00030
- Published: 2025-03-28
- arXiv: https://arxiv.org/abs/2504.00030
- Source URL: https://arxiv.org/abs/2504.00030
- PDF: https://arxiv.org/pdf/2504.00030
- Provenance: arxiv

## Abstract

Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce \\textit{GammaTune} and \\textit{GammaTune+}, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15\\% ($\\pm$5\\%) with \\textit{GammaTune} and 16\\% ($\\pm$3\\%) with \\textit{GammaTune+}, while reducing performance variance. This makes \\textit{GammaTune} a robust and efficient solution for real-world deployment.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.