Evaluating an AI-Driven Computerized Adaptive Testing Platform for Psychological Assessment: A Randomized Controlled Trial

This randomized controlled trial evaluated the psychometric performance, efficiency, and clinical utility of an artificial intelligence (AI)–driven computerized adaptive testing (CAT) platform for mood and anxiety assessment, compared with traditional fixed-form measures. A total of 300 adults (aged 18–65) from urban community mental health clinics were randomized to complete either an AI-based adaptive battery incorporating a model-tree CAT and transformerbased natural language processing for open-ended responses (Tadesse et al., 2021) or a traditional fixed-form battery (Beck Depression Inventory–II, State-Trait Anxiety Inventory, NEO Five-Factor Inventory). Licensed clinicians, blinded to assignment, subsequently conducted SCID-5 interviews; half reviewed reports augmented with explainable AI (XAI) decision aids, and half reviewed reports without AI support. The AI platform demonstrated high internal consistency (Cronbach’s α = .88; McDonald’s ω = .86) and strong convergent validity with established self-report scores (r = .78– .84, p < .001). Administration time was reduced by 35% (M = 14.2 vs. 21.8 minutes; t(298) = 19.40, p < .001). Clinician diagnostic concordance with SCID-5 increased when using XAI aids (κ = .82) compared to no AI support (κ = .71; F(1,298) = 16.30, p < .001). These findings support the reliability, validity, and efficiency of AI-based adaptive assessment, and highlight the value of human-in-the-loop XAI frameworks for enhancing diagnostic accuracy. Future research should extend validation to diverse linguistic and clinical populations, assess longitudinal predictive validity using electronic health record data, and develop standardized XAI evaluation protocols to ensure equitable and transparent AI integration in mental health care.