Can AI Read Brain MRIs Like Radiologists? A Multimodal Evaluation of ChatGPT and Gemini Using Real Patient Images

Artificial intelligence (AI) models have been investigated for their potential role in clinical diagnosis. The aim of this study was to compare the diagnostic performance of experienced radiologists and contemporary artificial intelligence models in patients with pathologically confirmed brain lesions, using standardized magnetic resonance imaging datasets under controlled evaluation conditions. This retrospective study analyzed 102 patients with pathologically confirmed brain lesions who underwent surgery between January 2024 and June 2025. Eight MRI sequences per patient were independently evaluated by two experienced radiologists and Artificial Intelligence Models (AIMs). Assessments included lesion detection, localization, edema, malignancy, grading, and differential diagnosis. Imaging data from 102 patients with pathologically confirmed brain lesions were evaluated by two experienced radiologists and three artificial intelligence models (ChatGPT-4.0, ChatGPT-4.5, and Gemini 2.5 Pro) using complete JPEG image sets of all MRI sequences along with clinical information. Although the AIMs accurately identified basic MRI sequences and fundamental radiological findings, they performed substantially worse than radiologists in differential diagnosis and in determining the single most likely diagnosis. Agreement with the reference standard was high for both radiologists, whereas the AI models showed only moderate levels of agreement. AIMs showed near-perfect performance in recognizing MRI sequences and identifying lesions within a lesion-enriched cohort. AI may serve as a supportive tool in clinical practice, though not a replacement for expert evaluation.