{"id":"arxiv:2512.02536","kind":"preprint","title":"WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens","authors":["Jian Yang","Dacheng Yin","Xiaoxuan He","Yong Li","Fengyun Rao","Jing Lyu","Wei Zhai","Yang Cao","Zheng-Jun Zha"],"author_count":9,"year":2025,"published":"2025-12-02","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2512.02536","nct_id":null,"source_url":"https://arxiv.org/abs/2512.02536","archive_url":"https://space-kitty.com/papers/arxiv:2512.02536","updated":"2025-12-02","pmid":null,"pmcid":null,"openalex_id":"W4417006787","pdf_url":"https://arxiv.org/pdf/2512.02536","abstract":"Recent progress in multimodal large language models (MLLMs) has highlighted the challenge of efficiently bridging pre-trained Vision-Language Models (VLMs) with Diffusion Models. While methods using a fixed number of learnable query tokens offer computational efficiency, they suffer from task generalization collapse, failing to adapt to new tasks that are distant from their pre-training tasks. To overcome this, we propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization, enhancing continual learning. Additionally, we introduce a VAE branch with linear projection to recover fine-grained image details. Experimental results confirm our approach mitigates generalization collapse and enables stable continual learning across diverse tasks.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CV"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"generative-multimodal","name":"Generative & Multimodal Models","domain_slug":"ai"}],"cited_in_dossiers":[]}