{"id":"arxiv:2602.07013","kind":"preprint","title":"Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models","authors":["Jiaxi Yang","Shicheng Liu","Yuchen Yang","Dongwon Lee"],"author_count":4,"year":2026,"published":"2026-01-31","venue":null,"cited_by":null,"doi":null,"arxiv_id":"2602.07013","nct_id":null,"source_url":"https://arxiv.org/abs/2602.07013","archive_url":"https://space-kitty.com/papers/arxiv:2602.07013","updated":"2026-01-31","pmid":null,"pmcid":null,"openalex_id":null,"pdf_url":"https://arxiv.org/pdf/2602.07013","abstract":"With the rapid advancement of Vision Language Models (VLMs), refusal mechanisms have become a critical component for ensuring responsible and safe model behavior. However, existing refusal strategies are largely \\textit{one-size-fits-all} and fail to adapt to diverse user needs and contextual constraints, leading to either under-refusal or over-refusal. In this work, we firstly explore the challenges mentioned above and develop \\textbf{C}onfigurable \\textbf{R}efusal in \\textbf{VLM}s (\\textbf{CR-VLM}), a robust and efficient approach for {\\em configurable} refusal based on activation steering. CR-VLM consists of three integrated components: (1) extracting a configurable refusal vector via a teacher-forced mechanism to amplify the refusal signal; (2) introducing a gating mechanism that mitigates over-refusal by preserving acceptance for in-scope queries; and (3) designing a counterfactual vision enhancement module that aligns visual representations with refusal requirements. Comprehensive experiments across multiple datasets and various VLMs demonstrate that CR-VLM achieves effective, efficient, and robust configurable refusals, offering a scalable path toward user-adaptive safety alignment in VLMs.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CV","cs.AI","cs.LG"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[],"cited_in_dossiers":[]}