Human-in-the-loop principles promise to improve the performance and reliability of clinician-facing artificial intelligence (AI) by looping expert professionals into system development and evaluation. However, little is known about how such principles are realized in practice once AI systems are deployed in everyday clinical work. This paper presents findings from an ethnographic study of a long-standing automatic speech recognition system used for clinical documentation in a Danish hospital. We show how clinicians’ ongoing auditing of AI output becomes a necessary condition for system reliability while remaining ambiguous within existing clinical work organization. Our analysis identifies a set of three main tensions between clinical documentation and auditing work; auditing work and system alteration; as well as collaborative augmentation work and established professional accountabilities. To account for these dynamics, we characterize dual-purpose data work, capturing how clinical documentation simultaneously serves patient care and AI system development. We further describe the emergent form of overhead work to explain how human-in-the-loop is sustained through extensive, often invisible work required to make ongoing collaboration possible across organizational boundaries. Our findings advance understanding of post-deployment human-in-the-loop AI in healthcare and outline opportunities for computing, which makes auditing work meaningful, reduces overhead, and renders expert contributions transparent.
Vase et al. (Wed,) studied this question.