Causal inference in educational effectiveness research: a comparison of three methods to investigate effects of homework on student achievement¹

In educational effectiveness research, it frequently has proven difficult to make credible inferences about cause and effect relations. The article first identifies the main categories of threats to valid causal inference from observational data, and discusses designs and analytic approaches which protect against them. With the use of data from 22 countries which participated both in the Trends in International Mathematics and Science Study (TIMSS) 2003 and TIMSS 2008 with samples of Grade 8 students, 3 different methods are then applied to investigate effects of amount of time spent on homework on mathematics achievement: (a) 2-level regression, which is applied to separate student-level relations from class-level relations; (b) instrumental variables regression, using teacher-reported homework time to instrument student-reported homework time; and (c) a difference-in-differences analysis investigating country-level change between 2003 and 2007. All 3 methods showed that there is a positive effect of homework time on student achievement.