Cognition, oversight, and the safety case for documentation AI

Large language models (LLMs) can now draft hospital discharge summaries. Reported hallucination rates vary by measurement context, from about 1%–2% in structured summarisation to over 60% under provocation testing. The common argument that LLMs outperform a tired resident treats a high‐stakes clinical and legal document as a text‐quality problem. We argue that a potentially underappreciated concern is the weakening of clinician oversight when reviewing fluent machine‐generated text, and that the joint clinician‐AI system is central to safety. We propose a Clinician‐AI Oversight Model with four safeguards: bounded autonomy, structured verification, error‐signal monitoring, and longitudinal evaluation of cognitive effects.