The Impact of Data-Analysis Methods on Cumulative Research Knowledge

The methods of data analysis used in research have a major effect on the development of cumulative knowledge. Traditional methods based on statistical significance testing have systematically retarded the growth of cumulative research by making it virtually impossible to discern the real meaning of research literatures. Meta-analysis makes it possible to demonstrate graphically the high price the research enterprise has paid for its reliance on significance testing. But in addition to these demonstrations, reform will require that researchers come to understand that the benefits they see as flowing from the use of significance tests are illusory. In research practice and in training of researchers, we must use and teach appropriate data analysis methods: point estimates of effect sizes and confidence intervals within individual studies, and meta-analysis in the integration of multiple studies to produce final conclusions. These reforms are essential to the progress of cumulative research knowledge.