# Evaluating the ChatGPT family of models for biomedical reasoning and classification

Article · Shan Chen, Yingya Li, Sheng Fu Lu, Hoang Van, Hugo J.W.L. Aerts, Guergana Savova, Danielle S. Bitterman

- Archive id: arxiv:2304.02496
- Published: 2023-12-19
- Venue: Journal of the American Medical Informatics Association
- DOI: https://doi.org/10.1093/jamia/ocad256
- arXiv: https://arxiv.org/abs/2304.02496
- PMID: 38261400
- Citations: 91
- Source URL: https://doi.org/10.1093/jamia/ocad256
- PDF: https://academic.oup.com/jamia/advance-article-pdf/doi/10.1093/jamia/ocad256/56334268/ocad256.pdf
- Provenance: openalex+arxiv

## Abstract

Recent advances in large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates the performance of LLMs such as the ChatGPT family of models (GPT-3.5s, GPT-4) in biomedical tasks beyond question-answering. Because no patient data can be passed to the OpenAI API public interface, we evaluated model performance with over 10000 samples as proxies for two fundamental tasks in the clinical domain - classification and reasoning. The first task is classifying whether statements of clinical and policy recommendations in scientific literature constitute health advice. The second task is causal relation detection from the biomedical literature. We compared LLMs with simpler models, such as bag-of-words (BoW) with logistic regression, and fine-tuned BioBERT models. Despite the excitement around viral ChatGPT, we found that fine-tuning for two fundamental NLP tasks remained the best strategy. The simple BoW model performed on par with the most complex LLM prompting. Prompt engineering required significant investment.

_Abstract reproduced under CC0-1.0 (arXiv descriptive metadata)._

Topics: [Reasoning Models & Test-Time Compute](https://space-kitty.com/topics/reasoning-models)

---
Space Kitty Archive · https://space-kitty.com · API: https://space-kitty.com/api/v1 · MCP: https://space-kitty.com/mcp · Premium: $0.20/request via x402 (Base Sepolia (testnet)). Metadata: arXiv (CC0), OpenAlex (CC0), Europe PMC (CC BY/CC0 works), ClinicalTrials.gov.