{"id":"arxiv:2510.22876","kind":"preprint","title":"Correctness Forensics for Batch Speculative Decoding: Diagnosing the Ragged Tensor Problem","authors":["Ranran Haoran Zhang","Soumik Dey","Ashirbad Mishra","Hansi Wu","Binbin Li","Rui Zhang"],"author_count":6,"year":2025,"published":"2025-10-26","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2510.22876","nct_id":null,"source_url":"https://arxiv.org/abs/2510.22876","archive_url":"https://space-kitty.com/papers/arxiv:2510.22876","updated":"2026-08-31","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2510.22876","abstract":"Inference optimizations are routinely evaluated by throughput alone, without verifying output correctness. We conduct a forensic analysis of batch speculative decoding and find that several widely-used implementations silently produce corrupted outputs (repetitive tokens, symbols) while reporting competitive speed; failures invisible to metrics like ROUGE. We trace the root cause to the ragged tensor problem: variable token acceptance desynchronizes position IDs, attention masks, and KV-cache across a batch. We formalize the synchronization invariants (rectangular alignment and position-ID contiguity) that valid batched inference must preserve and show that maintaining them incurs superlinear alignment overhead under contiguous layouts. EQSPEC enforces the invariants without custom kernels; EXSPEC schedules same-length sequences to bypass realignment. On SpecBench across three model families, EXSPEC reaches 3 x throughput at batch size 8 with 95% exact match to standard decoding; residual divergence traces to floating-point non-determinism, not synchronization error. Code:https://github.com/eBay/spec_dec","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CL","cs.AI"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}