# GPT-4V Cannot Generate Radiology Reports Yet

Article · Yuyang Jiang, Chacha Chen, Dang Nguyen, Benjamin M. Mervak, Chenhao Tan

- Archive id: arxiv:2407.12176
- Published: 2024-07-16
- Venue: Findings of the Association for Computational Linguistics: NAACL
- DOI: https://doi.org/10.18653/v1/2025.findings-naacl.113
- arXiv: https://arxiv.org/abs/2407.12176
- Citations: 6
- Source URL: https://arxiv.org/abs/2407.12176
- PDF: https://arxiv.org/pdf/2407.12176
- Provenance: arxiv

## Abstract

GPT-4V's purported strong multimodal abilities raise interests in using it to automate radiology report writing, but there lacks thorough evaluations. In this work, we perform a systematic evaluation of GPT-4V in generating radiology reports on two chest X-ray report datasets: MIMIC-CXR and IU X-Ray. We attempt to directly generate reports using GPT-4V through different prompting strategies and find that it fails terribly in both lexical metrics and clinical efficacy metrics. To understand the low performance, we decompose the task into two steps: 1) the medical image reasoning step of predicting medical condition labels from images; and 2) the report synthesis step of generating reports from (groundtruth) conditions. We show that GPT-4V's performance in image reasoning is consistently low across different prompts. In fact, the distributions of model-predicted labels remain constant regardless of which groundtruth conditions are present on the image, suggesting that the model is not interpreting chest X-rays meaningfully. Even when given groundtruth conditions in report synthesis, its generated reports are less correct and less natural-sounding than a finetuned LLaMA-2. Altogether, our findings cast doubt on the viability of using GPT-4V in a radiology workflow.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [AI in Clinical Medicine](https://space-kitty.com/topics/ai-medicine)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.