{"id":"arxiv:2608.19825","kind":"preprint","title":"Towards Clinically Faithful Medical Image Captioning via Enhanced Vision-Language Alignment","authors":["Yunseo Lee","Hyun Jun Kim","Heeseung Shin","Changwon Lim"],"author_count":4,"year":2026,"published":"2026-08-20","venue":null,"cited_by":0,"doi":null,"arxiv_id":"2608.19825","nct_id":null,"source_url":"https://arxiv.org/abs/2608.19825","archive_url":"https://space-kitty.com/papers/arxiv:2608.19825","updated":"2026-08-20","pmid":null,"pmcid":null,"openalex_id":"W7203927025","pdf_url":"https://arxiv.org/pdf/2608.19825","abstract":"Medical image captioning is a technique that accelerates early-stage diagnostic workflows and enhances the interpretability of medical diagnostic AI systems. However, unlike general image captioning, clinically reliable captioning remains challenging due to grayscale-based modalities, subtle anatomical cues, specialized medical phrasing, and variations in data quality. Despite recent advances in large vision-language models, fluent outputs do not necessarily guarantee sufficient alignment with clinical concept spaces or evaluation criteria. To address this issue, we propose a framework that strengthens clinical alignment by separating and enhancing training-time alignment and inference-time alignment. We build a medical image captioning pipeline that integrates single/dual vision encoders based on BioMedCLIP and SigLIP2, a Q-Former, and a LLaMA-based decoder, and examine the contribution of auxiliary learning for UMLS concept/type prediction. At inference, we apply single-embedding-based reranking to select the best caption among candidates, while at training we introduce MedPAIR-SCST, which combines clinically relevant rewards to shift the generative distribution toward improved clinical alignment. Our experiments show that complementary visual representations with a multi-encoder design and concept-level auxiliary learning help preserve clinically meaningful information. Furthermore, inference-time reranking provides a practical way to improve semantic and clinical alignment without additional training, whereas MedPAIR-SCST goes beyond selection by directly improving the model's distribution to generate more consistent and clinically grounded captions. These findings suggest that jointly leveraging selection-based alignment and reinforcement-learning-based alignment can promote more trustworthy medical image captioning even in data-constrained settings.","abstract_license":"CC0-1.0 (arXiv descriptive metadata)","license":null,"is_open_access":false,"retracted":false,"peer_reviewed":false,"categories":["cs.CV","cs.CL"],"details":null,"provenance":"arxiv","attribution":"Abstract: arXiv descriptive metadata (CC0). Thank you to arXiv for use of its open access interoperability. This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.","topics":[{"slug":"ai-medicine","name":"AI in Clinical Medicine","domain_slug":"medicine"}],"cited_in_dossiers":[]}