Attempting to improve the validity of a well‐established assessment centre*

Various procedural changes were introduced to an assessment centre (AC) for the selection of naval officers in order to assist the assessors to use the evidence more appropriately in arriving at their overall assessment rating (OAR). Procedural changes included: the reduction in the number of dimensions assessed; assembling component predictor information under the relevant dimension on an evidence organizer; a visual indication of predictive validity of components on the evidence organizer; and focusing assessors' discussion on areas of disagreement or criticality. Data concerning component predictors, dimensions and the OAR were examined for applicant groups before and after the changes were introduced. Assessors seem to have used the numerical components correctly when assessing two of the three dimensions although the OAR still appeared to place too much weight on the group exercises (compared with written evidence). No improvements in OAR predictive validities against training performance criteria were obtained after the changes, although there was an improvement in the prediction of voluntary turnover. These results are discussed in terms of implications for the AC studied, subjective versus mechanical integration of AC data, and explanations of AC validity. It is argued that, at least in the selection context, the traditional explanation of OAR validity (in terms of ‘high’ assessment technology) is not tenable. An explanation in terms of the availability of a wide range of evidence, and the assessors' belief in behavioural consistency, is better supported by the available data.