The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadvertently be utilized in ways that do not align with the intended purpose of the manufacturer, raising patient safety concerns and liability issues. We have identified a regulatory gap where LLM applications that could be repurposed, malfunction or inadvertently be misused for medical purposes fall outside the scope of the Medical Device Regulation (MDR), by lacking medical intended purpose, and meaningful AI Act safety obligations, by not qualifying as high-risk systems, yet present patient safety risks without adequate regulatory guidance. In this manuscript, we aim to address this regulatory gap for LLM applications in healthcare by identifying these ambiguous gray area LLM applications and providing four recommendations – validation, mitigation, compliance, and alignment – focused on prevention and risk mitigation rather than reclassification as medical devices, to advance the responsible implementation of LLM applications into healthcare while ensuring patient safety.
Workum et al. (Wed,) studied this question.