This repository contains the complete reproducibility package supporting the manuscript “A Real-Time Artificial Intelligence Diagnostic Copilot in Simulated Primary Care Consultations: Randomized Simulation Study”. The study was a case-level randomized, adjudicator-blinded, simulation-based physician-performance study conducted in a secure web-based virtual primary care clinic. Thirteen board-certified family and community medicine physicians completed 260 voice-based simulated consultations based on 40 diagnostically challenging adult clinical vignettes. Each consultation was randomized to either real-time assistance from the Medsys AI diagnostic copilot or an unassisted simulation condition without external diagnostic aids. No real patients were enrolled, no clinical care was delivered or modified, and no identifiable patient health data are included. The repository brings together the study protocol, clinical case materials, blinded adjudication files, analysis datasets, inter-rater agreement tables, data dictionary, executable Jupyter notebook, and a study analytical transparency document. The Jupyter notebook contains the full statistical-processing code and the rendered outputs used to reproduce the manuscript results. The analytical transparency document provides an end-to-end explanation of the statistical workflow, maps each major code block to the corresponding manuscript methods and results, and documents how the outputs were computed from the locked adjudicator-derived dataset. The protocol is included as originally prepared. The final statistical analysis implemented in the notebook differs from the initial protocol by using cross-classified mixed-effects logistic regression models with random intercepts for physician and case, a structure better suited to the repeated physician-case design. These changes, together with all sensitivity, exploratory, consultation-time, automation-bias, and human-AI comparison analyses, are documented in the notebook and in the analytical transparency document to support full third-party verification. What is included: 1. Study protocol - PC-MEDSYS protocolV1. 1 Feb 2025. pdf Full protocol describing the original study design, eligibility criteria, randomization, masking, procedures, outcomes, adjudication process, sample-size assumptions, and initial statistical-analysis plan. 2. Case materials - Details on 40 cases used in the trial. pdf Description of the 40 adult clinical vignettes used by the virtual-patient simulator, including case identifiers, gold-standard diagnoses, demographics, original clinical content, and simulator prompts. 3. Adjudication package - Adjudicator 1-Fernando Cerro Zarabozo. pdf - Adjudicator 2-Sara Garcia de Francisco. pdf - Adjudicator 3-Fernanda Domínguez. pdf Row-level documentation of the independent blinded adjudication process for physician and AI diagnostic outputs. The diagnostic acceptance criteria are described in the protocol. 4. Data dictionary - Data Dictionary_ Medsys AI Pre-clinical Trial Dataset Interpretation. pdf Definition of dataset variables, coding conventions, adjudication fields, trial-arm variables, timing variables, AI-output fields, and derived analysis variables. 5. Primary dataset and agreement tables - Phase₃complete results₂3092025. csv: Main episode-level analysis dataset, with one row per simulated consultation. - Phase₃complete results₁2062025 - Concordancia evaluadores para diagnóstico médico. csv: Inter-rater agreement matrix for physician diagnoses. - Phase₃complete results₁2062025 - Concordancia evaluadores para diagnóstico Medsys. csv: Inter-rater agreement matrix for AI-generated diagnoses. 6. Analysis code and rendered outputs- Data processing script-In silico trial with Medsys AIᵤpdated₀4072026. ipynb. End-to-end executable notebook including data loading, variable derivation, inter-rater reliability, primary mixed-effects models, Monte Carlo marginal probabilities, leave-one-physician-out sensitivity analyses, Top-1 and Top-2 sensitivity analyses, consultation-time analyses, case-difficulty and physician-ability moderation analyses, workload/order analyses, automation-bias analyses, and exploratory physician-versus-standalone-AI comparisons. 7. Study analytical transparency document - StudyAnalyticalTransparencyDocument. pdf Detailed explanation of the statistical workflow and its correspondence with the manuscript. This document is intended to support transparent third-party reproduction of the analyses. This repository is intended to support peer review, independent statistical re-analysis, and transparent reuse of the study materials. Users can reproduce the main manuscript results by downloading the dataset files and running the Jupyter notebook in a Python 3. 11 environment with the package versions described in the manuscript and notebook.
Cusácovich et al. (Sun,) studied this question.