The Reliability of Classroom Observations and Student Surveys in Non-Research Settings: Evidence From a Middle-Income Country

We present one of the first Generalizability studies of non-test measures of teaching effectiveness administered by practitioners in a middle-income country. The reliability of observations varies widely (from 0 to 0.75 on a 0–1 scale) and depends upon their context (whether they are conducted during training or on the job) and rater assignment configurations. The reliability of surveys varies substantially, coinciding with a change in the students sampled across occasions. Our estimates of reliabilty are comparable to those from research in high-income countries, but the variation within our estimates and between them and those from individual studies suggests that practitioners should conduct their own reliability analyses. We offer guidance on leveraging such analyses to improve the reliability of their measures.