Designing Trustworthy AI in Healthcare: Experiences With Copilot Agents, Agentic Models, and Rag Integration

Healthcare systems need to reduce administrative burden and support decision-making for clinical practice. Artificial intelligence approaches have the potential to reduce documentation and support diagnosis. Copilot Agents are in-app assistants that enable users to ask questions, automate documentation tasks, and coordinate clinical work processes without interrupting their current tasks within electronic health record systems. Agentic AI is not limited to single-turn questions and responses but also includes goal-aware reasoning during multi-turn tasks. Examples of such tasks span from processing prior authorizations to transitioning care calls and quality measurement documentation. Further, the clinical review at several checkpoints in the architecture is important to the implementation. For the RAG to be factually correct, language model outputs are grounded in validated institutional knowledge bases and clinically accepted guidelines. Source attribution mechanisms enable clinicians to trace model outputs to their respective information sources or references. Critical to the architecture of the RAG are security, privacy, and interpretability constraints in medical practices. Governance frameworks created by ongoing monitoring, responding to incidents, and involving stakeholders are essential for successfully using AI solutions in a way that supports rather than replaces clinical decision-making