Using Resampling to Estimate the Precision of an Empirical Standard-Setting Method

Several standard-setting methods can be used for authentic performance-based assessment using direct observation. Rational methods (e.g., the Angoff method) use judgments of test items to set a cutscore, whereas empirical methods (e.g., the borderline group or contrasting groups procedure) use judgments of examinees to obtain a cutscore. When comparing standard-setting methods, a critical concern is the precision of the obtained cutscore. With the Angoff method, cutscore precision can be estimated by applying generalizability analysis to the item judgment data. The empirical methods, however, lack a generally applicable procedure to estimate the precision. In this study a method is developed to estimate the cutscore precision for empirical methods by using resampling. The method was illustrated for a regression based standard-setting method applied to two actual datasets. The results show the applicability of the method: The precision estimate obtained with the resampling method was in reasonable agreement with the estimate obtained by using standard linear regression equations.