Abstract Artificial intelligence (AI) is rapidly reshaping healthcare by supporting earlier diagnosis, assisting clinical decision-making, and improving operational efficiency. However, most systems remain deployed within human-in-the-loop workflows, limiting their impact and underscoring the need for greater automation grounded in trust and transparency. Despite widespread adoption of radiology AI, predictive analytics, and emerging large language models (LLMs), hospitals lack a standardized framework to evaluate fairness, reliability, accuracy, and real-world safety. Prior failures illustrate how ambiguous objectives and unvalidated proxy targets can produce inequitable outcomes and erode clinical trust. This paper argues for a unified evaluation framework capable of assessing both traditional classification models and generative LLMs via transparent reporting of performance markers, subgroup fairness analyses, and hallucination detection. Current FDA pathways provide a foundation, but they remain insufficient for governing continual-learning systems and generative models in clinical workflows. We propose that the FDA extend these mechanisms to require mandatory disclosure of training data provenance and standardized benchmarks for clinical safety and relevance. Establishing such a framework is crucial to ensure that AI advances deliver autonomously safe and trustworthy healthcare. Keywords: healthcare AI, evaluation framework, algorithmic bias, FDA regulation, large language models, trustworthy AI
Jay Vashisht (Sun,) studied this question.