Artificial intelligence in the physician’s seat: a scenario-based evaluation of scientific accuracy, quality, and readability

To evaluate the scientific accuracy, quality, and readability of artificial intelligence (AI)-generated responses for prenatal counseling after a diagnosis of spina bifida (SB). Twenty frequently asked parental questions regarding prenatal SB diagnosis were submitted in Turkish to ChatGPT, Grok, and Gemini. Responses were independently evaluated by specialists in perinatology and pediatric neurology using the JAMA Benchmark Criteria, Global Quality Score (GQS), Modified DISCERN Scale, and a 5-point Likert scale. Readability was assessed using the Ateşman Readability Formula, and interrater association and absolute agreement were evaluated using Kendall’s Tau-b and intraclass correlation coefficients, respectively. All AI tools received high expert-rated scientific accuracy scores and high overall evaluation scores, and all models achieved full scores on the JAMA criteria. Significant overall differences among the three AI models were observed for GQS, DISCERN, and Likert scores (Friedman test, all p < 0.05). After Bonferroni-corrected pairwise comparisons (Wilcoxon signed-rank test, exact p values), ChatGPT scored significantly higher than Grok for the DISCERN and Likert scales; no other pairwise comparison, including ChatGPT versus Gemini for any scale, reached statistical significance after correction. Readability scores corresponded to a high school reading level. Inter-rater association and absolute agreement varied across models, with stronger alignment for Gemini, lower and non-significant estimates for Grok, and limited interpretability for ChatGPT owing to insufficient score variability. AI tools can generate scientifically accurate and comprehensive prenatal-counseling information for SB, as judged against a standard clinical reference (ACOG Practice Bulletin No. 187). Although they cannot replace physician–patient interaction, they may serve as useful adjuncts in patient education and counseling.