Artificial Intelligence Based Chatbots as Professional Medical Consultants in Oral Surgery: Are they reliable? Comparative Study (Preprint)

BACKGROUND With advances in artificial intelligence (AI) technologies, AI-based chatbots have become promising tools for generating medical information. Given their widespread use and easy accessibility, it is important to comprehensively evaluate the quality, accuracy, and safety of the information they produce to ensure their effectiveness as reliable sources in healthcare. OBJECTIVE The purpose of the study is to examine the reliability of chatbots and their role as professional consultants. METHODS 64 questions were generated, including systemic diseases and medications for which professional consultation is often requested and common conditions that may raise concerns during performing oral surgery. The questions were posed to ChatGPT 3,5 and Claude-instant at 2 sessions with 1 week interval. The answers were recorded and rated by 2 experienced oral surgeons by using 2 evaluation metrics: A modified DISCERN tool, a subset of the original, was used to evaluate the answers in terms of quality, and the Likert scale(LS) was used to evaluate the answers in terms of accuracy (6 point LS) and completeness (3 point LS). Statistical analyses, including intraclass correlation, Mann-Whitney U test, skewness and kurtosis coefficients calculations were conducted to assess and compare the performance of the chatbots. RESULTS In terms of intrarater agreement, ChatGPT demonstrated a high level of quality and accuracy in both sessions. Additionally, quality scores of ChatGPT was found statistically significantly higher than Claude instant. CONCLUSIONS The evaluated chatbots exhibited a remarkable capacity to generate valuable medical content. Our particular view is that, although they are currently insufficient to serve as a single source of information, AI-based chatbots will gain greater acceptance among healthcare professionals in the near future and can provide a quick solution to the high demand for medical care.