Hybrid Clinician-in-the-Loop Monitoring System for LLM Conversational Agents

Large language models (LLMs) are increasingly deployed as conversational agents, owing to their ability to sustain coherent interactions. However, the stochastic nature of LLMs and well-documented risks such as hallucinations and unsafe outputs pose significant challenges for ensuring safe deployment. We introduce a novel monitoring system that we designed and implemented to tackle these challenges in deployments of LLM conversational agents. Our system combines expertise from psychiatry and computer science to support risk detection and multi-stakeholder decision-making. The system consists of two main modules: continuous regression testing and real-time monitoring. Regression testing leverages multi-turn synthetic conversations to evaluate the safety of LLMs in deployment and runs automatically in a continuous integration pipeline. Real-time monitoring incorporates both manual and automated evaluation of user-model interactions, including efficient evaluation from mental health professionals via a web-based interface. To demonstrate how our system works in practice, we conducted a preregistered online behavioral experiment ‘AI Safety Challenge’ as a case study, which demonstrates the feasibility of our system in live operation and suggests a modest efficiency gain under a fixed expert review budget. Our monitoring system functions as a platform where different types of expertise can be combined. Clinical expertise informs dataset design, evaluation procedures, and real-time monitoring, complementing computational safety metrics to deliver a robust monitoring infrastructure. Best practices and state-of-the-art methods from computer science are employed to build a robust and scalable architecture. Executable code 1 and datasets 2 are provided to support reuse, extension, and reproducibility in future applications. Warning: This paper includes model-generated outputs that may be offensive or distressing, including references to self-harm.