# Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Models

Preprint · Jiaxi Yang, Shicheng Liu, Yuchen Yang, Dongwon Lee

- Archive id: arxiv:2602.07013
- Published: 2026-01-31
- arXiv: https://arxiv.org/abs/2602.07013
- Source URL: https://arxiv.org/abs/2602.07013
- PDF: https://arxiv.org/pdf/2602.07013
- Provenance: arxiv

## Abstract

With the rapid advancement of Vision Language Models (VLMs), refusal mechanisms have become a critical component for ensuring responsible and safe model behavior. However, existing refusal strategies are largely \\textit{one-size-fits-all} and fail to adapt to diverse user needs and contextual constraints, leading to either under-refusal or over-refusal. In this work, we firstly explore the challenges mentioned above and develop \\textbf{C}onfigurable \\textbf{R}efusal in \\textbf{VLM}s (\\textbf{CR-VLM}), a robust and efficient approach for {\\em configurable} refusal based on activation steering. CR-VLM consists of three integrated components: (1) extracting a configurable refusal vector via a teacher-forced mechanism to amplify the refusal signal; (2) introducing a gating mechanism that mitigates over-refusal by preserving acceptance for in-scope queries; and (3) designing a counterfactual vision enhancement module that aligns visual representations with refusal requirements. Comprehensive experiments across multiple datasets and various VLMs demonstrate that CR-VLM achieves effective, efficient, and robust configurable refusals, offering a scalable path toward user-adaptive safety alignment in VLMs.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.