Expert Review on the Quality of Responses to the Questions of Multiple Myeloma Patients: A Validation Study of the Medical Artificial Intelligence System “Myelobot”

BACKGROUND. The use of artificial intelligence (AI) in oncology and hematology opens up many possibilities for improving health service systems including communication between physicians and patients with long-standing diseases, such as multiple myeloma (MM). Generative AI based on the large language models is increasingly introduced into clinical practice. However, the issues of the quality of information provided as well as the level of empathy and clinical safety of such systems have until now remained underresearched. AIM. A comprehensive prospective evaluation of the quality of responses to the questions of MM patients provided by the specialized medical AI system “Myelobot”. MATERIALS & METHODS. This study used the scores of accuracy, empathy, and potential harm and additionally analyzed the consistency in reviewers’ ratings. All scores were measured with 5-point Likert scale with lower points corresponding to higher quality, safety, and empathy level of responses. Three hematologists participated in the study, independently and anonymously reviewing 32 AI system responses to patient questions across three scores. RESULTS. The median values of all scores appeared to be significantly lower than empirical threshold of 2.5 points (p < 0.001), suggesting a high quality of responses. At the same time, the Fleiss kappa and Krippendorff alpha coefficients of consistency in reviewers’ ratings were negative, especially on the empathy score, suggesting substantial variability in expert evaluations. CONCLUSION. AI service “Myelobot” demonstrated a high level of accuracy, clinical safety, and ability for empathic communication with MM patients. However, conflicting expert ratings clearly indicate the need for standardization of the scores and calibration of evaluation approaches in future studies. According to medical specialists, AI service system “Myelobot” is a highly effective MM patient support tool with a capacity to carry out the function of physician assistant providing medical information 24 hours a day.