Artificial intelligence as an adjunctive tool for rater qualification for clinical trials: an illustration

Background Rater qualification programs affirm ratings competency and inter-rater reliability for clinical trials. We explored the usefulness of a trained artificial intelligence (AI) program as an adjunctive tool to assess prospective raters prior to their participation in a trial using the Montgomery-Asberg Depression Rating Scale (MADRS). Methods An AI program was trained to score and provide narrative rationales for each item of the MADRS assessed by an audiotaped MADRS interview that was scored by 18 prospective rater trainees for a depression study. AI-generated scores were compared to the group scores, an “expert” score, and the original “live” rater who had conducted the interview. Results Total MADRS scores were 27 by the AI program, 28 by the expert rater listening to the audio, 26.3 ± 3.0 (SD) by a group of 18 rater trainees, and 31 by the live rater. AI and expert scores were essentially equivalent and effective in identifying 3 ratings outliers. Although close, not a single rater matched all 10 item scores derived by the AI program. The live rater scored the subject as more symptomatic than either the AI program or raters listening to the audiotape. Discussion In this exploratory illustration of one audiotape, an AI program generated scores that were akin to audio reviewers, and identified rater outliers, but scored lower than the live rater conducting the interview. The findings from this small illustration suggest that AI may have utility to assist rater qualification and warrants further exploration in larger studies with multiple AI scores.