{"id":"arxiv:2610.37266","kind":"preprint","title":"Watermark Basins: Black-Box Detection of Distilled LLMs","authors":["Hongyan Chang","Hady Elsahar","Tom Sander","Pierre Fernandez","Saeed Mahloujifar","Sylvestre-Alvise Rebuffi","Tomáš Souček","Tuan Tran","Valeriu Lacatusu","Surya Parimi","Rashel Moritz","Vanessa Stark","Alexandre Mourachko"],"author_count":13,"year":2026,"published":"2026-10-27","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2610.37266","nct_id":null,"source_url":"https://arxiv.org/abs/2610.37266","archive_url":"https://space-kitty.com/papers/arxiv:2610.37266","updated":"2026-10-27","pmid":null,"pmcid":null,"openalex_id":"W7224115392","pdf_url":"https://arxiv.org/pdf/2610.37266","abstract":"Proprietary language models and their reasoning traces are vulnerable to API-based distillation, where an adversary trains a student model on the teacher model's outputs. To obtain evidence of distillation, model providers can watermark the teacher's outputs, aiming to leave a radioactive signal in the student's outputs. Existing black-box detectors search for such radioactivity using token-level statistics of the suspect model's outputs. However, token-level signals are not always detectable, for instance when the distillation traces are sampled at a low temperature. In this work, we discover the watermark basin, a persistent signal that enables more reliable watermark detection: during fine-tuning, watermarked teacher traces embed a distinct, key-specific geometric signal in parameter space that lies orthogonal to the ordinary task-specific fine-tuning subspace. Building on this, we introduce Watermark Basin Alignment (WM-Basin), a black-box watermark detector that queries the suspect model through its API, without access to its weights or logits. WM-Basin maps generated text to gradient representations using a fixed clean probe model, projects out the ordinary fine-tuning task directions, and tests whether the residual gradient aligns with the watermark basin induced by the watermark key. We evaluate WM-Basin extensively across diverse teacher models and student architectures (from 135M to 8.2B parameters) distilled on mathematical reasoning. At a strict 0.1% false-positive rate, WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3, substantially improving over the state-of-the-art token-level baseline (29% and 1.3%, respectively). Because the watermark signal resides in a persistent basin, it survives post-distillation modifications including quantization, pruning, and reinforcement-learning post-training.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CR","cs.CL","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"ai-mathematics","name":"AI for Mathematics & Formal Proof","domain_slug":"math-cs"},{"slug":"efficient-ai","name":"Efficient Training & Inference","domain_slug":"ai"},{"slug":"interpretability","name":"Mechanistic Interpretability","domain_slug":"ai"}],"cited_in_dossiers":[]}