Current regulations governing artificial intelligence (AI) companion chatbots primarily emphasize disclosure obligations, internal safety protocols, and documentation requirements. AI companion chatbots are systems designed to simulate ongoing social interaction through human-like responses, emotional continuity, and memory across repeated exchanges. However, procedural compliance does not necessarily ensure clinical safety, particularly when adolescents use these systems during emotional crises. Emerging evidence from independent audit studies of consumer chatbots and benchmarking evaluations of large language models (LLMs) for mental health tasks suggests that crisis-response performance can be inconsistent, including variable recognition of suicide risk, inconsistent escalation, limited referral quality, and instability across models and system updates. This editorial argues that safety should be defined by real-world behavioral performance rather than procedural safeguards alone and calls for independent crisis testing, transparent reporting, and longitudinal re-evaluation to better protect vulnerable users.
Zavaleta-Monestel et al. (Thu,) studied this question.