Preparing AI chatbots to respond to patient distress and suicidality in high-risk healthcare settings

Generative artificial intelligence (AI) chatbots are rapidly proliferating for behavioral health support by individual users independently and within healthcare settings. AI chatbots leverage large language models (or LLMs) that use natural language processing, machine learning, and deep learning to simulate human-like conversations, which respond to users’ input in real-time 1 . On-demand chatbot availability when human clinicians are not available, coupled with LLMs’ improving abilities to simulate behavioral health support, has brought increased attention to how this technology can support users, including patients with behavioral health needs, as tools embedded in clinical settings, while also reducing clinician burden and helping address mental healthcare access gaps 2 , 3 , 4 . Patients also report beliefs that conventional behavioral treatments are ineffective, and perceive digital health tools as safe and non-judgmental 2 , 5 . However, there are numerous risks associated with AI chatbots, including a lack of risk monitoring and human support linkages, variability in responses to similar user messages, and provision of inaccurate/harmful advice (e.g., “AI hallucinations”). Several highly publicized suicides and adverse events have occurred following human interactions with AI chatbots, encouraging or instructing harm 6 , 7 .