ABSTRACT There is mounting global interest in the revolutionary potential of AI tools. However, its use in healthcare carries certain risks. Some argue that opaque (‘black box’) AI systems in particular undermine patients' informed consent. While interpretable models offer an alternative, this approach may be impossible with generative AI and large language models (LLMs). Thus, we propose that AI tools should be evaluated for clinical use based on their implementation risk, rather than interpretability. We introduce a practical decision algorithm for the clinical implementation of black‐box AI by evaluating its risk of implementation. Applied to the case of an LLM for surgical informed consent, we assess a system's implementation risk by evaluating: (1) technical robustness, (2) implementation feasibility and (3) analysis of harms and benefits. Accordingly, the system is categorised as minimal‐risk (standard use), moderate‐risk (innovative use) or high‐risk (experimental use). Recommendations for implementation are proportional to risk, requiring more oversight for higher‐risk categories. The algorithm also considers the system's cost‐effectiveness and patients' informed consent.
Allen et al. (2025) studied this question.