Development and Validation of the Artificial Intelligence in Mental Health Scale: Application for AI Mental Health Chatbots

Background/Objectives: Artificial intelligence (AI)-based chatbots present a viable approach to overcoming several challenges associated with conventional psychotherapy, such as high financial costs, limited access to mental health professionals, and geographical or logistical barriers. Thus, these chatbots are increasingly employed as complementary tools to traditional therapeutic practices in mental health care. Our aim was to develop and validate a scale to measure attitudes toward the use of AI-based chatbots for mental health support, i.e., the Artificial Intelligence in Mental Health Scale (AIMHS). Methods: A multidisciplinary panel of experts assessed the content validity. To confirm face validity, we carried out cognitive interviews and calculated the item-level face validity index. We applied factor analysis to verify the construct structure. We assessed measurement invariance across demographic subgroups. Concurrent validity was evaluated using three valid instruments. Reliability was tested through Cronbach’s alpha, Cohen’s kappa, and the intraclass correlation coefficient. Results: Factor analysis supported a two-factor five-item model. The two factors were technical and personal advantages, and explained 81.28% of the variance. The AIMHS demonstrated adequate concurrent validity, evidenced by statistically significant correlations with Artificial Intelligence Attitude Scale (r = 0.405, p-value < 0.001), Attitudes Towards Artificial Intelligence Scale (acceptance subscale; r = 0.401, p-value < 0.001, fear subscale; r = −0.151, p-value = 0.002), and Short Trust in Automation Scale (r = 0.450, p-value < 0.001). Configural, metric and scalar invariance were supported by our findings. Cronbach’s alpha was 0.798, and intraclass correlation coefficient was 0.938. Cohen’s kappa for the five items ranged from 0.760 to 0.848. Conclusions: The AIMHS is a five-item psychometrically sound and user-friendly instrument capturing two dimensions; technical and personal advantages. Future research should be undertaken to further evaluate the psychometric properties of the AIMHS across diverse populations and contexts.