{"id":"arxiv:2510.09942","kind":"preprint","title":"Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding","authors":["Payel Bhattacharjee","Fengwei Tian","Meiyu Zhong","Guangyi Zhang","Osvaldo Simeone","Ravi Tandon"],"author_count":6,"year":2025,"published":"2025-10-11","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2510.09942","nct_id":null,"source_url":"https://arxiv.org/abs/2510.09942","archive_url":"https://space-kitty.com/papers/arxiv:2510.09942","updated":"2025-10-11","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2510.09942","abstract":"Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.LG","cs.AI","cs.IT"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}