Evaluating adolescent depression-related GAI agents on the basis of the FAITA-Mental Health Framework: A Comparative Case Series Study (Preprint)

BACKGROUND Adolescent depression is a pressing global health concern, with timely screening vital for early intervention. Barriers like limited access to professional resources, stigma, and adolescent reluctance often hinder early detection. Generative artificial intelligence (GAI) tools offer scalable, low-stigma support but remain exploratory for depression diagnosis, with limited evidence on reliability, usability, and alignment with established clinical standards—especially for adolescents, whose unique needs demand tailored assessments. OBJECTIVE This study aimed to evaluate the performance and consistency of a GAI-based robot named “Xiao-Yu” for adolescent depression screening, using the FAITA-Mental Health framework. It sought to assess the tool’s usability among adolescents with emotional disorders, compare its diagnostic accuracy against psychiatric professionals, and examine consistency via test-retest consistency after treatment. Additionally, it aimed to validate the FAITA-Mental Health framework’s applicability for evaluating GAI-driven mental health tools. METHODS Xiao-Yu was developed using the MINI-KID diagnostic scale, and GPT-4 was developed via prompt engineering. Its performance was compared with that of psychiatric professionals. Half of the participants underwent repeated testing with Xiao-Yu after two weeks of treatment. FAITA-Mental Health and supplemental metrics were used for evaluation. RESULTS Xiao-Yu achieved 100% task completion and a 72% F1 score in diagnosing 13 adolescents. The level of consistency with professionals was 77%. FAITA-Mental Health scores: trustworthiness, 6/6; user experience, 5/6; autonomy, 2/4; equity, 2/4; transparency, 1/2; crisis management, 2/2. User acceptance was 92%. CONCLUSIONS The GAI has moderate diagnostic consistency, reliability, and high acceptability for adolescent depression. FAITA-Mental Health is a viable evaluation framework but should incorporate clinical metrics (e.g., accuracy, reliability) for practical utility.