Methods of Improving Employment Test Critical Scores Derived by Judging Test Content: A Review and Critique

Setting critical scores on content valid tests may involve making judgments about test content. A conclusion which has been reached in the testing literature is that the Angoff method is preferred as this type of judgmental method of setting standards or critical scores. The current paper reflects this conclusion, but also discusses several limitations of the Angoff method, and outlines why less than optimal data may be obtained using this preferred method or similar methods. Several techniques are reviewed and critiqued that may be used in conjunction with the Angoff method or similar methods to attempt improvement of the psychometric quality of the critical score data. Components from two psychometric frameworks (generalizability theory and Cronbach's, 1955, accuracy scores) are integrated and applied in assessing the impact that each of the techniques may have on Angoff critical score data. The techniques reviewed can be roughly divided into: (a) procedural methods (involved in the judgment process), and (b) psychometric methods (involved in data analysis following judgment). The present review is intended as a resource to personnel testing specialists who may be interested in methods of bolstering the process of judgmentally deriving critical scores.