Accuracy and reproducibility of AI chatbots in answering guideline-based questions on pulp therapy in primary and immature permanent teeth

The integration of artificial intelligence into dental education and clinical information seeking is increasing, while concerns regarding the accuracy and reproducibility of AI-generated responses remain. This study aimed to assess the response accuracy and reproducibility of AI chatbots in answering guideline-based questions on pulp therapy in primary and immature permanent teeth. Twenty text-based clinical questions regarding pulp therapy in primary and immature permanent teeth were developed in multiple-choice and dichotomous formats. Seven AI chatbots (ChatGPT 4o, ChatGPT 4.1, DeepSeek R1, Gemini 2.5 Flash, Copilot, Claude 4 Sonnet, and Perplexity AI) were tested over three consecutive days. Response accuracy was analyzed using bias-reduced logistic generalized estimating equations with clinical question as the clustering variable. Reproducibility was assessed using exact agreement, Fleiss’ kappa, Cochran’s Q test, and Holm-corrected exact McNemar tests. No statistically significant overall difference in response accuracy was detected among the evaluated AI chatbots ( p = 0.190). Testing day had a significant effect on response accuracy ( p = 0.016), with accuracy rates of 77.9%, 85.0%, and 83.6% on days 1, 2, and 3, respectively. No significant overall differences were observed according to question type ( p = 0.057) or clinical question category ( p = 0.429). Reproducibility differed significantly among the evaluated AI chatbots ( p = 0.041), with exact agreement ranging from 45.0% to 85.0% and Fleiss’ κ from − 0.025 to 0.567; however, no pairwise comparisons remained statistically significant after Holm correction. The evaluated AI chatbots showed generally high response accuracy in standardized pediatric endodontic scenarios, with no significant overall difference among chatbots. However, response accuracy varied across testing days and reproducibility differed among chatbots, indicating that similar overall response accuracy may not reflect consistent performance over time. Further validation is needed before AI chatbots can be considered for broader clinical use.