{"id":"doi:10.1038/s41746-025-01830-9","kind":"article","title":"Evaluation of performance of generative large language models for stroke care","authors":["John Tayu Lee","Vincent Cheng-Sheng Li","Jia-Jyun Wu","Hsiao-Hui Chen","Sophia Sin-Yu Su","Brian Pin-Hsuan Chang","Richard Lee Lai","Chi-Hung Liu","Chung-Ting Chen","Valis Tanapima","Toby Kai-Bo Shen","Rifat Ali Atun"],"author_count":12,"year":2025,"published":"2025-07-29","venue":"npj Digital Medicine","cited_by":19,"doi":"10.1038/s41746-025-01830-9","arxiv_id":null,"nct_id":null,"source_url":"https://doi.org/10.1038/s41746-025-01830-9","archive_url":"https://space-kitty.com/papers/doi:10.1038/s41746-025-01830-9","updated":null,"pmid":"40730644","pmcid":null,"openalex_id":"W4412786083","pdf_url":"https://www.nature.com/articles/s41746-025-01830-9.pdf","abstract":"Stroke is a leading cause of global morbidity and mortality, disproportionately impacting lower socioeconomic groups. In this study, we evaluated three generative LLMs-GPT, Claude, and Gemini-across four stages of stroke care: prevention, diagnosis, treatment, and rehabilitation. Using three prompt engineering techniques-Zero-Shot Learning (ZSL), Chain of Thought (COT), and Talking Out Your Thoughts (TOT)-we applied each to realistic stroke scenarios. Clinical experts assessed the outputs across five domains: (1) accuracy; (2) hallucinations; (3) specificity; (4) empathy; and (5) actionability, based on clinical competency benchmarks. Overall, the LLMs demonstrated suboptimal performance with inconsistent scores across domains. Each prompt engineering method showed strengths in specific areas: TOT does well in empathy and actionability, COT was strong in structured reasoning during diagnosis, and ZSL provided concise, accurate responses with fewer hallucinations, especially in the Treatment stage. However, none consistently met high clinical standards across all stroke care stages.","abstract_license":"CC-BY (publisher license, via OpenAlex)","license":"cc-by","is_open_access":true,"retracted":false,"peer_reviewed":null,"categories":["Artificial Intelligence in Healthcare and Education"],"details":null,"provenance":"openalex","attribution":"Abstract of \"Evaluation of performance of generative large language models for stroke care\" by John Tayu Lee, Vincent Cheng-Sheng Li, Jia-Jyun Wu et al., https://doi.org/10.1038/s41746-025-01830-9, used under its Creative Commons Attribution license (CC-BY); markup removed, otherwise unmodified.","topics":[{"slug":"reasoning-models","name":"Reasoning Models & Test-Time Compute","domain_slug":"ai"}],"cited_in_dossiers":[]}