With great power comes great responsibility: How narrow conceptions of validity in high‐stakes testing undermine competence

Context As Medical Education recognises the 100th anniversary of Kelley's writing on validity, ongoing efforts to implement competency‐focused training make the centenary a critical time to reflect on the sufficiency of validity argumentation for high‐stakes testing. Shifting dynamics in assessment, after all, have never been more apparent as recent years have witnessed countries that traditionally place heavy emphasis on high‐stakes testing (e.g. the United States and Canada) respond to social pressures by decreasing their use while others (e.g. the UK) have moved the opposite direction. Focus The needs of institutions, administrators, students, communities and patients are evolving within a context of dynamic changes in society, education and technology. We, therefore, offer critical analysis of three myths that continue to surround high‐stakes testing as a means to encourage collective and deliberate reflection about how the field can best adapt. Discussion In the era of competency‐based education, it is not only insufficient but also counter‐productive to mount a validity argument without demonstrating how high‐stakes decisions account for (and influence) what happens before and after moments of high‐stakes testing. High‐stakes decisions intended to assure competence can paradoxically, yet predictably, threaten competence if the assessments they rely on prioritise short‐term hurdles over the skills and behaviours required for long‐term success.