Using the Angoff Method to Set Defensible Cutoff Scores for Standardized Patient Performance Evaluations in PA Education

Purpose: This study explored the reliability and credibility of a standardized patient (SP)-based performance exam in physician assistant (PA) education with passing standards set using the Angoff method. Methods: PA faculty were asked to serve as judges, using the Angoff method to set a passing cut-off score for a four-case SP exam. Fifty-eight clinical-year PA students were evaluated using the exam. Passing cut-off levels and passing rates were reported for the Angoff method, for a fixed percentage cut-off score of 70%, and for a norm-referenced standard. Reliability of test items was assessed using Cronbach's alpha. Judge agreement was evaluated using kappa statistics. Credibility of judge's ratings was assessed using Pearson correlation, comparing student performance on test items with item difficulty as defined by the Angoff method. Results: The passing score was 62% (100% pass rate) for the Angoff method, 70% (88% pass rate) for the fixed percentage, and 72% (81% pass rate) for the relative standard. Cronbach's alpha for the test items was 0.75. Judge agreement was substantial (kappa coefficient = 0.71). Pearson correlation between actual item difficulty and difficulty as predicted by the Angoff method showed significant positive correlations (+0.44, p < 0.05). Conclusions: The Angoff method proved to be a reliable and credible method for setting a passing cut-off score for the exam. This study demonstrates that different standard-setting methods yield different results and care should be taken to engage in a defensible process when making grading decisions for SP examinations.