Dimensionality, internal consistency and interrater reliability of clinical performance ratings

A total of 6444 ratings of the financial performance of 424 third- and fourth-year medical students were made by house officers and attending teachers during 12 separate internal medicine rotations. Ratings were based on 13 behaviourally anchored rating scales. One rating was randomly selected per student per evaluator type (house officer and attending teacher) during each of the 12 rotation periods. Ratings were factor analysed separately within each rotation period. Two factors emerged consistently, and congruence coefficients across the 12 occasions were high (0.88 or greater). The factors were labelled 'problem-solving (10 items) and 'interpersonal skills' (three items) on the basis of their content. Internal consistency coefficients of the indices constructed from items in the two factors and the total of the 13 items were high (0.9 or greater) and did not differ substantially when computed separately on the ratings from house officers and attending teachers. Interrater reliabilities on the individual items ranged from 0.14 to 0.33.