Real-world evaluation of a multilingual agentic retrieval-augmented clinical AI framework for evidence-grounded medical decision support

Large language models may support clinical information tasks, but hallucination, limited evidence traceability, and inconsistent workflow integration remain barriers to clinical deployment. We evaluated Dr.Copilot, a multilingual retrieval-augmented clinical AI framework combining hybrid evidence retrieval, specialized tools, and model-directed orchestration. During a 70-day deployment, 835 registered users generated 956 conversation threads; the analytics log contained 22,574 events and 4,226 tool-use events. In the original 64-prompt expert benchmark, Dr.Copilot and GPT-5.5 had comparable weighted scores (4.607 vs. 4.561; paired difference 0.046, 95% CI -0.064 to 0.140; p = 0.103). In a new independent 75-case, fixed-backbone external validation, Dr.Copilot scored 4.557 versus 4.217 for GPT-5.5 (difference 0.340, 95% CI 0.181 to 0.519; p = 0.00023), with fewer clinically important errors (5/75 vs. 16/75; p = 0.00098). A 30-case four-arm GPT-5.4 ablation found the highest mean score with standard RAG (4.813), while additional prespecified-tool and model-directed routing conditions did not yield uniform further gains. Targeted safety, pediatric, multilingual, citation, temporal, and abstention analyses identified both strengths and important failure modes. These findings support retrieval as the clearest incremental contributor in this evaluation, while underscoring that agentic tool orchestration requires direct validation and does not itself establish clinical benefit or patient-safety improvement.