Systematic Reviews in Medical Education

In medicine, physicians are charged to use the best available evidence to make decisions about the benefits and harms of clinical interventions. To this end, physicians have come to rely on systematic reviews to provide evidence in a synthesized form.1 In medical education, practitioners are similarly encouraged to utilize systematic reviews to facilitate the practice of evidence-based education,2,3 such that systematic reviews are the most published form of knowledge synthesis in medical education, with over 300 published in the last 20 years.4The Institute of Medicine defines a systematic review as “a scientific investigation that focuses on a specific question and uses explicit, prespecified scientific methods to identify, select, assess, and summarize the findings of similar studies.”1 Systematic reviews can also include a meta-analysis to demonstrate evidence through statistical power by combining results from individual studies. Taking root in the positivist tradition (ie, an absolute truth can be determined from scientific experiments), systematic reviews have traditionally focused on answering questions about effectiveness (eg, whether an intervention works) and have relied heavily on synthesizing findings of randomized control trials.5,6 However, over time, the scope of systematic reviews has expanded to include multiple study designs and to answer a variety of research question types. For example, in the context of medical education, the following are illustrative questions that could be answered by a systematic review:In Box 1, The Case of Dr Smith, a program director considers the use of a systematic review to answer an important question for the program. Each article in this JGME special series on conducting and reading reviews will consider this same question through a different review methodology and philosophical underpinning.7Like all knowledge syntheses, systematic reviews have strengths and weaknesses that should be considered. A major strength of systematic reviews is that, through a rigorous and clearly defined process, they aim to identify, critically appraise, and distill all of the individual studies on a topic to provide readers with recommendations to inform their practice.8 Another strength of a systematic review is that it aims to answer a focused, specific question, with a clearly defined population, intervention/exposure, comparison, and outcomes. This focus on a well-defined question concentrates all aspects of the conduct of the review and helps readers readily determine whether the review applies to their situation and context.9Clear guidance exists for conducting and reporting systematic reviews, which can facilitate the systematic review process, decrease bias, increase research transparency, and enable reproducibility.10 For example, researchers can access the freely available Cochrane Handbook11 or the JBI Manual for Evidence Synthesis,12 which provides detailed instructions and considerations for conducting reviews. Specific to the context of medical education, there are resources, such as a Best Evidence Medical Education (BEME) guide on conducting systematic reviews,13 an article by Cook and West that describes a stepwise approach,14 and 2 “Twelve Tips” articles.15,16 For transparency in the reporting of systematic reviews, researchers are referred to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)17 and its recent extension specific to searches.18 Those conducting systematic reviews in medical education would also benefit from reviewing the STructured apprOach to the Reporting In healthcare education of Evidence Synthesis (STORIES; see Box 2).19While a systematic review's targeted question can be considered a strength overall, this narrow focus may not allow for the comprehensive overview of a topic sought by most educators. For example, a reader may be interested in learning more about feedback, but find that a systematic review on the topic focuses narrowly on specific types of feedback or a very focused population. In these cases readers may do well to search for a narrative review or scoping review. Readers should also recognize that systematic reviews do not answer questions about why and how an intervention does or does not work, which is more the domain of a realist review.9To conduct a systematic review, authors need to analyze primary studies. In some cases topics are discussed in the literature, but their coverage is limited to commentaries, perspectives, or a few preliminary studies. Medical education populations and settings are usually diverse and difficult to tightly define. Interventions may also include subtle or obvious differences among studies, and “control” groups are actually difficult to control in the medical education realm. Thus, systematic review searches with well-defined criteria may find few papers to synthesize. Therefore, it is not feasible to conduct a systematic review until there is a significant body of evidence (ie, number of relevant articles) available, which can mean that a systematic review might not be published until the topic is quite established in the literature. In these cases, researchers might consider an alternate knowledge synthesis type, such as a scoping review, to provide an overview of the topic and a map of the current literature (see Box 3).20Lastly, like other knowledge synthesis types, systematic reviews are a major undertaking and are highly resource- and time-intensive.11 Researchers must accept that systematic reviews take time to conduct: researchers estimate it takes from 6 months to 2 years,21 in addition to the time for the publication process.22 This can affect the timeliness of access to distilled evidence and has implications for researchers with promotion or graduation deadlines.Systematic reviews should exhibit multiple hallmarks of quality. As a baseline, we encourage readers to consider the elements put forward by PRISMA (eg, including a research question, reporting a complete search strategy, using multiple coders).17 In addition to PRISMA, we recommend several hallmarks of quality specific to systematic reviews conducted in medical education.As health professions education cuts across multiple fields (eg, education, psychology, sociology), a hallmark of a high-quality systematic review is that it includes a search of multidisciplinary and education-focused databases (eg, Web of Science, ERIC [Education Resources Information Center], PsycINFO), which helps to ensure the inclusion of relevant studies published outside of journals focused on biomedicine. Additionally, when appropriate for the topic, the review should provide details of teaching strategies, discussions of theory use included in the studies, and information about the educational context of the intervention.18 The systematic review should also contain an appraisal and description of the quality of the included studies. In medical education, quality indices for quantitative studies include the Medical Education Research Study Quality Instrument (MERSQI) and the Newcastle-Ottawa Scale–Education (NOS-E).23Below we provide a brief overview of key steps in conducting a systematic review; however, we strongly advise that researchers consult other resources, such as the Cochrane Handbook,11 for further details and a full rationale for each step. Additionally, we encourage researchers to familiarize themselves early with the relevant reporting guidelines (eg, PRISMA or STORIES) to facilitate robust reporting and to ensure that they meet reporting requirements for their targeted journal.Researchers may consider approaches such as PICO (Population, Intervention, Comparison, Outcome)24 or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type)25 to guide question design. For example, using SPIDER, a research question could be: What are residents' (sample) experiences (evaluation) of burnout (phenomenon of interest) as explored through qualitative (research type) interviews (design)? Dedicating time upfront to formulating a strong research question will pay dividends, as the research question informs almost all aspects of the review.Strategically assemble a team with the necessary expertise to answer the research question. Recruitment may include subject matter experts, information professionals, and statisticians, if conducting a meta-analysis. We also suggest considering the inclusion of stakeholders (eg, learners, faculty). Discuss upfront the time commitments necessary to fully participate in a systematic review to ensure team members are comfortable with the overall research plan and time line. Systematic reviews conducted by a solo researcher are strongly discouraged.The team should register a protocol of the review with a systematic review register (eg, PROSPERO) to identify similar efforts26 and to satisfy journal editors who increasingly require registration.We encourage researchers to collaborate with an information professional to design a comprehensive search strategy that includes multiple databases and combines hand searching of key journals and the grey literature. Keep meticulous records of all searches and note the search dates. Keep in mind that it may be necessary to update searches if there is a significant lag time between the initial search and submission to a journal.Using inclusion criteria formulated in alignment with the research question, at least 2 team members should independently screen all titles and abstracts of citations to determine if they meet the inclusion criteria. For those meriting full-text review, 2 team members should also independently screen the full-text articles for inclusion. In both screenings, the presence of 2 independent raters helps to minimize bias. The availability of a third researcher to serve as a tiebreaker can also be useful.To extract data from the primary studies, the researchers should create and pilot a data extraction tool to be applied to the full text of included studies by a minimum of 2 team members. Depending on the review's study question, extracted data can be analyzed qualitatively or quantitatively, including through meta-analyses. Meta-analysis of quantitative studies is possible only if the data from the individual studies are homogenous; if data are too heterogeneous, as is often found in medical education, it is not possible to accurately perform a meta-analysis. In this step, the authors should also critically appraise the included studies using such tools as the MERSQI and NOS-E.23All authors should participate in the drafting and revision of the review manuscript. When writing, ensure that the manuscript meets the reporting guidelines (eg, PRISMA, STORIES) of the target journal. Also, to increase the usability of the review, ensure that readers are provided a cohesive discussion of the findings in relation to the existing literature. To facilitate this step, keep in mind what will be valuable to stakeholders attempting to integrate the findings into their educational practices.Undertaking a systematic review is an intensive process designed to produce a rigorous product that provides readers a synthesis of the evidence to answer a specific research question. While systematic reviews have both strengths and weaknesses, they are commonly conducted in the context of medical education and broadly accepted by journals in the field. Researchers considering conducting systematic reviews are advised to consult the variety of guidance available before beginning their research.