A Survey on Medical Competence Evaluation Benchmarks for Large Language Models

ABSTRACT Large language models (LLMs) show considerable potential to revolutionize healthcare through their performance across diverse clinical applications. Given the inherent constraints of LLMs and the critical nature of medical practice, a rigorous and systematic evaluation of their medical competence is imperative. This study presents a comprehensive review of the established methodologies and benchmarks for evaluating the medical competence of LLMs, encompassing a thorough analysis of current assessment practices across medical knowledge, clinical practice competence, and ethical–safety considerations. By integrating clinician competency assessment frameworks into LLMs evaluation, we propose a structured tri‐dimensional framework that systematically organizes existing evaluation approaches according to medical theoretical knowledge, clinical practice ability, and ethical–safety considerations. Furthermore, this research provides critical insights into future developmental trajectories while establishing foundational frameworks and standardization protocols for the integration of LLMs into medical practice.