Application of Analysis of Variance to the Estimation of the Reliability of Observations of Teachers’ Classroom Behavior

MANY EDUCATORS and psychologists have come to believe that the efficient way to esti mate the internal consistency of a measuring in strument is to divide it into halves, score each half separately, correlate the pairs of scores so obtained by means of the Pearson product-mo ment coefficient, and correct this value for the fact that it is based on halves instead of wholes. This procedure was developed independently by Spearman (13) and by Brown (2) for the solution of certain test construction problems. It is also widely believed that the efficient way to e s t i mate the stability of an instrument is to adminis ter equivalent forms to a sample of subjects and correlate the two sets of scores so obtained. It is the purpose of this paper to suggest the effi ciency of the analysis of variance technique for estimating the reliability of educational meas ures, and to illustrate use of the technique on ob servations of teachers' classroom behavior. The analysis of variance has been suggested as a method for estimating test reliability by Jackson (6,7), Hoyt (5) and Alexander (1). Its use for estimating the reliability of grades as signed to Compositions or essay examinations was described by Pilliner (11). Lindquist (8: 357-82) has recently presented a rather com plete discussion of the general use of the analy sis of variance technique for reliability estima tion in educational and psychological mea s?re ment. The method is particularly well adapt ed to observational data, as Lindquist remarks, but concrete examples of its proper use are not available in the literature. In connection with a longitudinal study of teacher education graduates of the New York City municipal colleges (14), the reliability of two observational techniques for assessing teachers' classroom behaviors was studied. The method of estimating reliability used in this longitudinal study will be described briefly, and its application to the classroom observation data will be illustrated. Methods of computa tion will not be given since they are readily acces sible in other sources; emphasis will be on the logic of the procedure and the interpretation of re sults.