Leveraging Natural Language Processing: Toward Computer-Assisted Scoring of Patient Notes in the USMLE Step 2 Clinical Skills Exam

The United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam uses physician raters to evaluate patient notes written by examinees. In this Invited Commentary, the authors describe the ways in which the Step 2 CS exam could benefit from adopting a computer-assisted scoring approach that combines physician raters’ judgments with computer-generated scores based on natural language processing (NLP). Since 2003, the National Board of Medical Examiners has researched NLP technology to determine whether it offers the opportunity to mitigate challenges associated with human raters while continuing to capitalize on the judgment of physician experts. The authors discuss factors to consider before computer-assisted scoring is introduced into a high-stakes licensure exam context. They suggest that combining physician judgments and computer-assisted scoring can enhance and improve performance-based assessments in medical education and medical regulation.