Does Choice of a Criterion Matter?

For reasons of objectivity and job relevance, a hands-on performance test (HOPT) has been advocated as the best measure to use as a validation criterion. However, the cost of HOPTs has inspired a search for surrogate measures such as job knowledge tests (JKTs). This study questions whether choice of criteria makes a difference for the purpose of developing an accurate selection composite. Hanser's (1990) research using Army Project A data found that use of surrogates in place of a HOPT made little difference for the selection accuracies of predictor composites of the Armed Services Vocational Aptitude Battery (ASVAB). This study uses Marine Corps Job Performance Measurement (JPM) data to replicate Hanser's results. It also questions whether a criterion is needed at all for developing an accurate selection composite. It demonstrates that predictor composites such as Ree and Earles's (1990a) psychometric g and unit-weighted regression coefficients can be as useful as those developed using an outcome criterion. It is concluded that use of surrogates can make no appreciable difference in the accuracies of selection composites and that equal or better selection composites (e.g., psychometric g, unit weighting) can be produced without reference to any criterion. It is cautioned that choice of a criterion will make a difference when used for purposes where the strength of the relationship between aptitude and criterion is critical. Setting selection standards is one such purpose.