Navigating the regulatory boundary: a sandbox approach to validating scope expansion in LLM-based ambient AI scribes

Background AI as a medical device (AIaMD), particularly devices using large language models (LLMs), pose a regulatory challenge. Device qualification and classification depend partly on the manufacturer’s stated intended purpose. Unlike traditional devices with fixed outputs, LLMs can generate outputs beyond their intended purpose through hallucinations and other behaviours, creating uncertainty around device classification. Innovation and approach Within the Medicines and Healthcare products Regulatory Agency (MHRA) AI Airlock regulatory sandbox, we investigated intended purpose, medical device qualification and real-world performance of TORTUS, an LLM-based ambient AI scribe that generates clinical notes. Methods Virtual and real-world testing evaluated two safety mechanisms using a linear mixed-effects model. The Shell, reducing hallucinations and the Context Guardrail, mitigating adversarial prompt injection. Simulation workshops tested how classification judgements changed when participants assessed functionality, behaviour and user experience rather than intended purpose statements alone. Results In real-world testing, hallucinated content appeared in 2.9% of text segments analysed per clinical note, with device function falling outside of its intended purpose (scope expansion) in 1.98%. The Shell significantly reduced scope expansion across both testing environments. The Context Guardrail was effective in virtual testing but showed no significant real-world effect. Workshop participants found that qualification judgements depended more on product functionality and user interaction than on intended purpose alone. Conclusions LLM-based AIaMD intended purpose is most robust when supported by well-defined premarket validation and continuous postmarket monitoring of real-world behaviour. Quantitative scope-expansion measures offer a basis for ongoing monitoring. Regulatory frameworks should address monitoring infrastructure and how stated intended purpose can be confirmed in practice.