Key points are not available for this paper at this time.
A meta-analysis of 111 interrater reliability coefficients and 49 coefficient alphas from selection interviews was conducted. Moderators of interrater reliability included study design, interviewer training, and 3 dimensions of interview structure ( standardization of questions, of response evaluation, and of combining multiple ratings). Interactions showed that standardizing questions had a stronger moderating effect on reliability when coefficients were from separate ( rather than panel) interviews, and multiple ratings were useful when combined mechanically (there was no evidence of usefulness when combined subjectively). Average correlations (derived from alphas) between ratings were moderated by standardization of questions and number of ratings made. Upper limits of validity were estimated to be.67 for highly structured interviews and.34 for unstructured interviews.
Conway et al. (Sun,) studied this question.