ChatGPT-assisted kidney pathology: insights into current strengths and limitations

Artificial intelligence (AI) holds great promise for enhancing diagnostic workflows in pathology, yet its true value in complex, image-driven clinical scenarios has not been established. Separately, large language models (LLMs) such as ChatGPT and Gemini have shown strong performance on controlled, exam-style pathology questions.1 However, neither approach has been tested on authentic patient cases that demand seamless integration of clinical history and high-resolution imaging. With recent advances in multimodal large language models, these systems are now capable of processing not only narrative text but also medical images, offering the potential for truly integrated diagnostic reasoning. However, their capacity to synthesize patient vignettes and detailed histological findings into accurate kidney diagnoses remains unproven.2 We therefore set out to evaluate how these models combine clinical and visual inputs when interpreting authentic kidney biopsy cases, and to test the hypothesis that their diagnostic accuracy depends predominantly on the clinical narrative, with only marginal gains from image data alone. This work represents the first rigorous assessment of multimodal large language model-based inference in nephropathology. We retrospectively evaluated the diagnostic accuracy of two multimodal large language models (GPT-4o and GPT-o1, OpenAI) on clinical cases with kidney biopsy images, previously published between August 1, 2007 and January 31, 2025,3 and under three levels of contextual information (redacted clinical vignette; redacted vignette + biopsy image with minimal caption; full vignette + image with detailed legend and histopathological findings). In a supplementary analysis, GPT-o1 generated histological descriptions and diagnostic hypotheses based only on biopsy image with minimal caption. Four expert nephropathologists rated GPT-o1's interpretation across four predefined criteria (recognition of normal structures, detection of pathologic alterations, interpretation coherence, and diagnostic compatibility), which also served as the basis for a systematic review of image-interpretation outputs to identify recurrent error patterns. This research was conducted in accordance with the TRIPOD-LLM guidelines.4 Model runs were conducted on April 24, 2025, and expert pathologist review was carried out in May and June 2025. Full methodological details are provided in the Supplementary Methods. We analyzed 66 published kidney biopsy clinical cases. The case mix comprised 31.8% acute kidney injury, 31.8% glomerular diseases, 19.7% kidney transplant pathology, 13.6% chronic kidney disease and kidney failure, 1.5% electrolyte disorders, and 1.5% tubulointerstitial disease. Geographically, cases originated from Europe (43.9%), North America (36.4%), Asia (12.1%), Oceania (4.5%), and South America (3.0%). Detailed geographic origin and nephropathology themes are presented in Table S1. Each case included at least one kidney histopathology image with its description, and 12.5% also featured additional clinical, laboratory, or radiologic images. When evaluated with only the redacted clinical vignette, both models showed low accuracy (GPT-4o 37.9%; 95% Confidence Interval [CI], 26.2–49.6; GPT-o1 43.9%; 95% CI, 32.0–55.9). Adding the biopsy image with a minimal caption did not significantly improve performance (GPT-4o 30.3%; 95% CI, 19.2–41.4; p = 0.36 vs. redacted text; GPT-o1 51.5%; 95% CI, 39.5–63.6; p = 0.38). In contrast, providing the full histological description markedly increased accuracy (Figure 1A), with GPT-o1 outperforming GPT-4o across all scenarios—achieving 80.3% vs. 57.6% accuracy when full clinical and image details were available (P < 0.001). In the image-only evaluation, GPT-o1 accurately recognized basic histological structures (eg, glomeruli) in 90.9% of cases (95% CI, 84.0–97.8) but detected disease-specific abnormalities in only 42.4% (95% CI, 30.5–54.3). Only 39.4% (95% CI, 27.6–51.2) of its interpretations were judged coherent and pathologically plausible, and its diagnostic hypotheses were compatible with the image findings in only 34.8% of cases (95% CI, 23.4–46.3; Figure 1B). Agreement between nephropathologists was high across all four domains, with pairwise Cohen's κ consistently strong and global Fleiss’ κ ranging from 0.70 to 0.82 (Figure S1). GPT-o1 committed at least one error in 87.9% of image-only cases: it omitted true lesions (“negative hallucinations”) in 40.9%, introduced spurious findings (“positive hallucinations”) in 30.3%, and propagated training-data biases or over-generalizations in 71.2% (Table S2). (A) ChatGPT diagnostic accuracy by model version and information scenario. (B). Expert pathologist evaluation of AI-generated biopsy descriptions. In this study, we evaluated two multimodal ChatGPT models on real-world kidney biopsy cases. Appending a minimal image caption to the clinical vignette did not improve diagnostic accuracy, which remained modest. By contrast, providing the original, detailed histological description markedly boosted performance. GPT-o1, OpenAI's chain-of-thought-enhanced variant, consistently outperformed GPT-4o, reaching 80.3% accuracy when given full clinical and image context. In a separate image-only assessment, GPT-o1 reliably identified normal histological structures but frequently failed to detect disease-specific lesions, suggesting that the 2024 generation multimodal large language models are not yet capable of fully exploiting visual data for accurate diagnostic reasoning. Most errors originated from failure to recognize or characterize lesions, indicating dependence on explicit cues rather than flawed reasoning. While diagnostic logic was consistent once a lesion was identified, subtle or atypical features were often missed. The models tended to default to common glomerular diagnoses (eg, diabetic or membranous nephropathy) in nonspecific contexts, reflecting training-data biases. Certain stains also biased outputs: for instance, GPT-o1 often misidentified membranous nephro­pathy in the presence of Jones silver staining. These findings align with prior work showing strong large language model performance on text-based kidney pathology items but poor results on image-based questions.5,6 Our team previously described similar results in the field of neuroradiology: GPT-4o and Gemini 1.5 Pro gained no accuracy from added image input and often incorrectly described key abnormalities.7 Training with histopathology image sets may help: in one study, GPT-4's accuracy improved from 61.7% to 90.0% after training with 10 images.8 Still, barriers remain, including training data biases. In the future, large language models could also be able to “read” whole-slide images through specialized models. Encouragingly, GPT-o1 showed clear diagnostic gains over GPT-4o despite only a seven-month gap between releases. Our research has several limitations: 1) Generalizability is limited, inherent to using 66 published cases, which cannot represent the full diversity of clinical nephrology. 2) No direct comparison with human pathologists was made in this pilot study. Further studies should explicitly address human-AI interface to better delineate the respective strengths and limitations of AI-assisted pathology. 3) The cases do not mirror day-to-day biopsy workflows, as our assessment relied on journal-quality image fields, limited to a few key stains and lacking the full panel of histochemical and immunohistochemical stains or high-resolution whole-slide scans that are often critical for routine diagnosis. 4) Findings pertain to two proprietary large language models; rapidly advancing open-source models—more suited for confidential patient data—may yield different results. 5) Given the rapidly evolving field, the versions evaluated in this study are no longer the most up to date, which further underscores the exploratory nature of our findings. In conclusion, multimodal large language models showed promising accuracy on complex kidney pathology cases, driven largely by text rather than images. GPT-o1 outperformed GPT-4o with full context but both struggled with image-only diagnosis. Multimodal large language models may augment—but not replace—expert nephropathologists, and further studies on routine material are needed before clinical use. Nothing to declare. Thomas Beaudrey (Investigation, Writing—original draft, Writing—review & editing, Visualization), Raphaël Bentegeac (Methodology, Software, Investigation, Writing—review & editing), Nans Florens (Conceptualization, Writing—review & editing, Supervision), Mehdi Maanaoui (Investigation, Writing—review & editing), Yann Boudard (Investigation, Writing—review & editing), Viviane Gnemmi (Investigation, Writing—review & editing), Jean-Baptiste Gibier (Investigation, Writing—review & editing), Bastien Le Guellec (Methodology, Software, Investigation, Writing—review & editing), and Aghiles Hamroun (Conceptualization, Methodology, Writing—original draft, Writing—review & editing, Supervision) This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The clinical case data analyzed in this study were obtained from previously published articles. The specific dataset compiled for analysis in this manuscript can be made available by the corresponding author upon reasonable request, subject to compliance with relevant ethical and copyright considerations. The analysis code used in this study is also available upon request to the corresponding author. This study involved the retrospective analysis of publicly available, de-identified case materials from previously published articles. As no new data involving human participants were collected, institutional review board (IRB) approval was not required. Not applicable. This study did not involve direct patient contact or the collection of new data from individuals. During the preparation of this work the authors used ChatGPT-o1 in order to improve English readability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication. Supplementary material is available at Journal of Nephrology online.