Measuring Learning Effectiveness: A New Look at No-Significant-Difference Findings

Researchers, instructional designers and consumers of ALNs must be cautious when interpreting results of media comparison studies. Much of the literature purports to have found no significant difference in learning effectiveness between technology-based and conventional delivery media. This research, though, is largely flawed. In this paper, we first outline the philosophical positions of the opposing sides of an intense debate in the literature as to whether delivery media alone influence learning outcomes. We then select at random several representative media comparison studies to illustrate the inadequacy of their methodologies and conclusions. More important, we derive critical design considerations for those who evaluate or conduct media comparison research. ALN practitioners should not assume that students would learn better from technology delivery systems. Rather, ALN practitioners should adhere to time-tested instructional design strategies, regardless of the medium they choose. Learning effectiveness is a function of effective pedagogical practices. Accordingly, the question for ALN practitioners ought to be: "What combination of instructional strategies and delivery media will best produce the desired learning outcome for the intended audience?"