Transparency fallacy: insights from healthcare professionals on the safe use of medical AI

As medical Artificial Intelligence (AI) becomes increasingly embedded in clinical care, transparency is widely promoted as a means to support safety, accountability, and trust. In this paper, AI 'transparency' refers to the provision of meaningful information about an AI system (a definition drawn from the International Organization for Standardization) and is conceptually distinct from AI ‘explainability’. We examine the policy and regulatory dimensions of transparency in medical AI, drawing on semi-structured interviews with twelve healthcare professionals in Australia and New Zealand, jurisdictions whose regulatory frameworks align with those of other International Medical Device Regulators Forum members. The study, first, contributes evidence about what information healthcare professionals require about medical AI and the limitations of AI transparency. Participants primarily supported clear disclosure to healthcare professionals of AI use and key instructions for safe operation of the technology. Yet the findings reveal that, in practice, healthcare professionals often overlook labels, face low AI literacy, and experience information overload, all of which undermine transparency’s effectiveness. This demonstrates the 'medical AI transparency fallacy': the incorrect assumption that providing more information about a medical AI system, by itself, ensures its safe and responsible use. Second, the article proposes a three-step, holistic policy approach to improve the effectiveness of transparency around medical AI. We recommend 1) improved information design, 2) a stronger information supply chain through medical education and clinical and professional guidance, and 3) greater regulatory disclosure to enable external scrutiny. This framework aims to guide regulators, policymakers, and industry in developing policies for the safe implementation of medical AI. Given the small sample, future research should test these findings at scale to assess broader generalisability.