# From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Models

Preprint · Changming Xiao, Qi Yang, Feng Zhou, Changshui Zhang

- Archive id: arxiv:2309.04109
- Published: 2023-09-08
- Venue: Neurocomputing, Volume 610, 2024, 128437
- DOI: https://doi.org/10.1016/j.neucom.2024.128437
- arXiv: https://arxiv.org/abs/2309.04109
- Source URL: https://arxiv.org/abs/2309.04109
- PDF: https://arxiv.org/pdf/2309.04109
- Provenance: arxiv

## Abstract

Diffusion models have revolted the field of text-to-image generation recently. The unique way of fusing text and image information contributes to their remarkable capability of generating highly text-related images. From another perspective, these generative models imply clues about the precise correlation between words and pixels. In this work, a simple but effective method is proposed to utilize the attention mechanism in the denoising network of text-to-image diffusion models. Without re-training nor inference-time optimization, the semantic grounding of phrases can be attained directly. We evaluate our method on Pascal VOC 2012 and Microsoft COCO 2014 under weakly-supervised semantic segmentation setting and our method achieves superior performance to prior methods. In addition, the acquired word-pixel correlation is found to be generalizable for the learned text embedding of customized generation methods, requiring only a few modifications. To validate our discovery, we introduce a new practical task called "personalized referring image segmentation" with a new dataset. Experiments in various situations demonstrate the advantages of our method compared to strong baselines on this task. In summary, our work reveals a novel way to extract the rich multi-modal knowledge hidden in diffusion models for segmentation.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.