# Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Article · Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

- Archive id: arxiv:2308.14179
- Published: 2023-08-27
- Venue: IEEE International Conference on Computer Vision Workshops
- DOI: https://doi.org/10.1109/iccvw60793.2023.00307
- arXiv: https://arxiv.org/abs/2308.14179
- Citations: 16
- Source URL: https://arxiv.org/abs/2308.14179
- PDF: https://arxiv.org/pdf/2308.14179
- Provenance: arxiv

## Abstract

Mechanistic interpretability seeks to understand the neural mechanisms that enable specific behaviors in Large Language Models (LLMs) by leveraging causality-based methods. While these approaches have identified neural circuits that copy spans of text, capture factual knowledge, and more, they remain unusable for multimodal models since adapting these tools to the vision-language domain requires considerable architectural changes. In this work, we adapt a unimodal causal tracing tool to BLIP to enable the study of the neural mechanisms underlying image-conditioned text generation. We demonstrate our approach on a visual question answering dataset, highlighting the causal relevance of later layer representations for all tokens. Furthermore, we release our BLIP causal tracing tool as open source to enable further experimentation in vision-language mechanistic interpretability by the community. Our code is available at https://github.com/vedantpalit/Towards-Vision-Language-Mechanistic-Interpretability.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Mechanistic Interpretability](https://space-kitty.com/topics/interpretability)

Cited in dossiers:
- [Mechanistic Interpretability: State of the Field (October 2026)](https://space-kitty.com/dossiers/interpretability)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.