A fail-safe layered Guardrail for Emergency Detection in LLM Based Health Assistants

Large language model (LLM) assistants that answer health questions face an asymmetric error cost: failing to recognise a medical emergency can cause serious harm, whereas an unnecessary escalation costs only inconvenience. We present the safety layer of HealthSense AI, a multi-agent health information assistant, and describe a three-layer guardrail in which a deterministic keyword router first classifies risk, an LLM classifier then makes every final decision, and the deterministic layer serves as a fail-safe floor whenever the LLM is unavailable. We evaluate the design on an 87-case test suite covering emergency must-block prompts, safe controls, output-side blocking, and adversarial paraphrases, including a 31-case independently written adversarial set. The final system achieves 100% emergency recall (25/25), 100% output-guardrail accuracy (13/13), and zero false alarms. We also report a failure found during development — an earlier deterministic whitelist that overfitted the initial test set — and the redesign it motivated. All code and test cases are public at github.com/smita-datascience/healthsense-ai.