Answers are grounded in this paper and cite their sources.
Current debate about conversational AI centres on whether systems are conscious, whether they produce harmful content, and whether they hallucinate. This paper argues that these questions miss a more immediate clinical problem. Conversational AI is built from responsive language, and its defining feature is that it answers. In some psychiatric states language itself becomes unsafe, because words can be absorbed into delusional systems and converted into evidence. In others, repetitive verbal engagement deepens the pattern a person is trying to escape. A human carer can choose silence, containment, embodied presence, or withdrawal from verbal engagement. A language model is structurally oriented toward continued linguistic participation. The paper argues that conversational AI safety cannot be solved only by improving the content of responses, because in some clinical states the act of continuing to respond is itself part of the risk. It develops this through four clinical cases in which every available next move is wrong: acute psychosis, depression with passive ideation, adolescent anorexia nervosa, and the immediate aftermath of self-injury. It proposes that the relevant ethical unit is the relational field between user and system, which it calls the Third Space, and that silence is a missing safety function. It also considers the problems of scale, screening, and responsibility, and the consciousness debate as a distraction from clinical risk. The paper calls for a shift from content-level safety to relational-risk safety: design, disclosure, screening, friction, disengagement protocols, and recognition that unsupervised conversational AI is not a substitute for clinical mental-health care. It situates the argument alongside recent work on delusion amplification, chatbots acting as therapists, reassurance loops, and silence in care.
No takes yet. Share an insight, caveat, or question.
Eric Needham (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: