Key points are not available for this paper at this time.
Background Systematic review refers to a growing set of research methodologies intended to critically appraise and synthesize research to inform policy and practice. Systematic review methods, widely used in fields such as medicine, psychology, and education, are being applied in engineering education. Analysis of current systematic review practices in engineering education can contribute to continuous improvement of an important set of synthesis methodologies. Purpose To promote continuous improvement in systematic reviews, researchers are currently developing criteria to evaluate the quality of systematic review methodologies. The purpose of this article is to identify prior systematic reviews in areas of interest to the engineering education community, use these criteria to critique these systematic reviews, and make recommendations for improving use and quality of systematic review methods in engineering education research. Scope/Method Using systematic review search methodologies, we identified 12 conference papers and 37 journal articles that are systematic reviews published since 1990. Two reviewers coded each by topic, education level, quantitative or qualitative sources and synthesis methods, and 18 quantitative measures of review quality. Interrater reliability was calculated using kappa as described by Landis and Koch, which ranged from 0.43 (moderate) to 0.80 (almost perfect) with overall agreement of 0.64 (substantial). Conclusions We found that the 49 identified systematic reviews in engineering education are particularly strong in stating their purposes, rationale, and inclusion criteria. They can be improved by explaining reasons for excluding studies, appraising quality of included studies, engaging multiple coders in evaluation procedures, stating their limitations, and, in the case of meta-analyses, describing statistical methods in greater detail.
Borrego et al. (Wed,) studied this question.