Answers are grounded in this paper and cite their sources.
OBJECTIVES: The objective of this work is to demonstrate the value of simulation testing for rapidly evaluating artificial intelligence (AI) products. MATERIALS AND METHODS: Researcher-physician teams simulated the use of 2 Ambient Digital Scribe (ADS) products by reading scripts of outpatient encounters while using both products, yielding a total of 44 draft notes. Time to edit, perceived amount of effort and editing, and errors in the AI-generated draft notes were analyzed. RESULTS: Ambient Digital Scribe Product A draft notes took significantly longer to edit, had fewer omissions, and more additions and irrelevant or misplaced text errors than ADS Product B. Ambient Digital Scribe Product A was rated as performing better for most encounters. DISCUSSION: Artificial intelligence-enabled products are being rapidly developed and implemented into practice, outpacing safety concerns. Simulation testing can efficiently identify safety issues. CONCLUSION: Simulation testing is a crucial first step to take when evaluating AI-enabled technologies.
No takes yet. Share an insight, caveat, or question.
Biro et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: