PURPOSE This study assessed the diagnostic accuracy and medical appropriateness of a publicly available large language model (LLM) in triaging common clinical scenarios in breast oncology. METHODS From January through February 2025, seven physicians interacted with OpenAI ChatGPT-4o and ChatGPT-o1pro and impersonated 66 patient complaint scenarios in the early-stage, metastatic, and survivorship settings. Each interaction began with a standardized phrase to prompt ChatGPT to act as a provider. Through iterative questioning, the tool provided a diagnosis, management plan, triage recommendations, and supportive care advice for common oncology clinic triage concerns which were reviewed by the physicians for appropriateness. The primary outcomes were the proportion of scenarios in which LLM arrived at the correct, or acceptable, diagnosis and provided clinically appropriate, or reasonable, triaging recommendations. The secondary end point was appropriateness of LLM's questions during history taking. RESULTS Of 849 LLM-generated questions across 132 simulated interviews, 97% was highly medically appropriate and 3% was reasonable but repetitive. The correct diagnosis was listed as the top choice in 89% of scenarios and was included in the differential diagnosis 98% of the time. Triage recommendations were rated highly appropriate in 92% of scenarios, and no recommendations were considered dangerous. Clinical decisions regarding supportive care were fully appropriate in 86% of cases and reasonable but not optimal in 14%. Overall agreements for ChatGPT triaging performance metrics between physician raters were 77%-86%. CONCLUSION ChatGPT-4o– and ChatGPT-o1pro–driven iterative questioning to triage common outpatient breast oncology concerns was diagnostically accurate and clinically appropriate. These results support further real-world assessment to establish safety of LLM-generated triaging recommendations when interacting with individuals of variable medical vocabulary and health literacy.
Chan et al. (Mon,) studied this question.