Measuring the reliability of subject classification by men and machines

Procedures for measuring the consistency of document classification are described. Three subject specialists classified 997 abstracts of psychological reports into one of eleven categories. These abstracts were also mechanically classified by a computer program using a factor‐score computational procedure. Each abstract was scored for all categories and assigned to the one with the highest score. The three manual classifications were compared with each other and with the mechanical classifications, and a series of contingency coefficients was computed. The average reliability of manual classification procedures was equal to 870. The correlation between automatic and manual classification was .766.