What question did this study set out to answer?

July 5, 2026BMC Oral Health0 citationsOpen Access

Can AI chatbots be reliable in dental emergencies? quality assessment of Arabic responses to dental emergency inquiries and public attitudes toward their use

KAKhalid AboalshamatUmm al-Qura University ADAbrar K. DemyatiUmm al-Qura University AGAbdalmalik Osamah GhandourahUmm al-Qura University

Key Points

This study aims to evaluate public attitudes toward AI chatbots in dental emergencies and assess the quality of their responses in Arabic.
Conducted a cross-sectional online survey with 441 Saudi residents aged ≥ 18 years using a 33-item questionnaire in Arabic.
Collected 50 dental emergency inquiries and evaluated responses from five different AI chatbots by two calibrated oral and maxillofacial surgeons.
Measured response quality based on accuracy, clarity, comprehensiveness, relevance, and acceptability using 5-point Likert scales.
Participants had moderately positive attitudes towards AI chatbots, with scores ranging from 2.72 to 3.89 out of 5.
AI chatbots scored high on accuracy (4.08–4.87) and clarity (4.21–4.92), but Grok had significantly lower ratings on multiple measures (p < 0.001).
Inter-rater reliability varied across chatbots, with single-measure ICC values ranging from 0.23 to 0.60 and exact agreement at 69.8%.

Abstract

Abstract Background Artificial intelligence (AI) is increasingly penetrating health and dental fields without sufficient monitoring of its quality and applicability. Aim This study aimed to evaluate public attitudes toward AI chatbots in dental emergencies and assess the quality of Arabic-language responses generated by different AI chatbots for dental emergency inquiries. Methods The study had two parts. Part one: A cross-sectional online survey where 441 Saudi residents aged ≥ 18 years answered a 33-item questionnaire in Arabic that included 14 items to measure attitudes about the use of AI chatbots in dental emergencies with a 5-point Likert scale. Part two: From participant answers and oral and maxillofacial surgeons, we selected 50 dental inquiries about dental emergencies and presented them in Arabic to five AI chatbots (ChatGPT-5.1, Google Gemini 3, Claude Sonnet 4.5, Grok 1.3.40, and DeepSeek 3.2). Responses were evaluated by two calibrated oral and maxillofacial surgeons using 5-point Likert scales for accuracy, clarity, comprehensiveness, relevance, and acceptability. Results Participants showed moderately positive attitudes (2.72–3.89/5) about AI chatbots for dental emergencies. AI chatbots had generally high mean scores for accuracy (4.08–4.87), clarity (4.21–4.92), comprehensiveness (4.10–4.67), relevance (4.11–4.91), and acceptance (3.84–4.89). No significant differences were found among the AI chatbots, except Grok, which scored lower than the others on multiple quality measures (all p < 0.001). Inter-rater reliability varied across chatbots, single-measure ICC values ranging from 0.23 to 0.60; however, exact agreement was 69.8%, and 94.5% of paired ratings differed by no more than one point. Conclusion Saudi public attitudes toward AI chatbots in dental emergencies were moderate. Overall, the quality of Arabic AI chatbot responses was high, although Grok had significantly lower ratings. Human supervision remains essential, and continuous “living” evaluations are needed to track rapidly evolving chatbot performance.

Ask AI

Helpful

Bookmark

View Full Paper