A criterion-referenced cut-scoring method using cognitive diagnostic modelling for mastery determination

What was the educational challenge? Despite their prevalence, norm-referenced standards and fixed cut scores have inherent limitations. Criterion-referenced standards, while more desirable, are often deemed impractical and criticized for their reliance on “minimal competence.” What was the solution? This paper introduces a novel cut-scoring approach based on a modified contrasting groups method, informed by cognitive diagnostic modeling (CDM) analysis results. It aligns content and performance standards and supports criterion-based, mastery-oriented absolute standards without relying on the “minimal competence” concept. How was the solution implemented?Demonstration of all assessed attributes according to CDM analysis determined the gold standard for competence. A modified contrasting groups method was then applied, using the best Youden’s index for choosing the cut score. The method has been tentatively applied in no- to high-stakes assessments. What lessons were learned that are relevant to a wider global audience? The findings suggest that most cut scores are lower than they should be. The reliance on compensatory, single-score assessments may cause unfit medical students to be allowed to practice. Initiatives that aim to strategically use assessment to foster learning in a longitudinal manner, such as programmatic assessment, seem to be a moral imperative.What are the next steps?While the described method is currently limited to formative purposes due to regulatory norms, it has a potentially positive impact on patient safety. Therefore, we advocate for its exploration in high-stakes licensing examinations.