Evidence and gap maps: a comparison of different approaches

Evidence maps are a relatively new approach to systematically identify and report the range of research activity in broad topic areas or policy domains. Although they go under various names, they have the common feature of adopting a systematic approach to mapping begins with a broad question, theme or issue which defines the scope of the evidence map. For reasons discussed below, we adopt the term Evidence and Gap Map to refer to the specific approach being proposed for Campbell, and evidence map to refer collectively to the range of approaches which have been adopted. As we describe in Part 2, evidence mapping of various forms has been around for at least 15 years. A range of different approaches to evidence mapping and synthesis have been developed to support evidence informed policy making. In 2016, (Miake-Lye, Hempel, Shanman, (Oakley, Gough, Oliver, (Salina Bates, 2006); (Peersman, 1996) evidence maps (Liu, Parker, Hetrick, (Bragge, et al., 2011); Scientific uncertainties (Jacobson, 2014), and Evidence Gap Maps (Snilstveit, Vojtkova, Bhavsar, see Table 1. The same year the Evidence for Policy and Practice Information and Coordinating Centre (EPPI-Center) published a review of personal development planning for improving student learning, distinguishing between the systematic map of the research which been undertaken and the systematic synthesis of what the evidence says (Gough, Kiwan, Sutcliffe, Simpson, & Houghton, 2003). Two years later the first visual map was published as the Evidence Based Policing Matrix (Rosenberg & Knox, 2005). In 2005 EPPI-Centre published a mapping methodology which was adapted the following year by the Social Care Institute for Excellence (SCIE) in response to a lack of empirical data to answer specific questions using systematic reviews, and a perceived need for a methodology to describe the literature in a broader field of interest than that typically covered in a single review. In 2010, 3ie produced its first evidence map, labelled the Evidence Gap Map. The organisation's experience in preparing maps for a range of topics in international development is reviewed by Snilstveit (Snilstveit, Vojtkova, Bhavsar, & Gaarder, 2013). Several organizations in the international development sphere have since worked with 3ie, or independently, to develop their own evidence maps which typically have variations in design. These include International Rescue Committee (IRC, 2016), USAID (Vinogradova & Heaner, 2016), Sightsavers (Virendrakumar, Jolley, Gordon, Bascaran, & Schmidt, 2016), BEAM Exchange on Market Research. Each of the agencies producing maps has a different approach, and each defines evidence map differently. A review of the definitions (see Annex 1 for full definitions) shows the following eight components contained in the definitions (see Table 2)1: What is perhaps missing in the definitions is a more explicit statement that mapping commonly tells us what evidence there is, but not what this evidence says - though some maps do include information on effects. This point is often misunderstood by those commissioning evidence maps and so should be made clear up front where it is the case. From these definitions we suggest the following comprehensive definition. Elements in square brackets … may be considered optional. ‘An evidence and gap map is a systematic visual presentation of the availability of relevant evidence of effects for a particular policy domain. The evidence is identified by a search following a pre-specified, published search protocol. The map may be accompanied by a descriptive report to summarize the evidence for stakeholders such as researchers, research commissioners, policy makers, and practitioners Evidence maps summarize what evidence there is, not what the evidence says’. Evidence and gap maps have two sets of relatives. The first relatives are other evidence synthesis research outputs. The second are different evidence platforms. Figure 1 shows different evidence synthesis products, and the inverse relationship between scope and content. Systematic reviews are narrowest in scope but deep on content, coding, analyzing and reporting effect sizes, intervention characteristics and contextual factors. A review of reviews may be broader in scope but may be more restricted in the depth of analysis. Evidence and gap maps have a still broader scope but report a far more limited range of information about the reviews and primary studies they include. A mega-map is a map with a very broad scope. It shows other maps and reviews, but not primary studies. And a map of maps only reports other EGMs in that policy space (Phillips, et al., 2017). Evidence maps are a ‘break’ on the content spectrum, as reviews and reviews of reviews report what the evidence says whereas many maps report what evidence there is but not what it says. Scope and content of different evidence synthesis products Evidence maps may be a way of curating (or brokering) a body of evidence to make it accessible to users. We distinguish three main evidence platforms with successively higher levels of curation or brokering: Beyond evidence portals there are evidence-based guidelines and checklists which contain recommendations and directions. These last three are different from databases and maps as databases and maps take users back to the research studies, whereas portals etc. present the evidence to inform decision-making without the user needing to consult the research directly. This paper is focused on evidence maps. A companion paper will discuss evidence portals. For nearly all the agencies reviewed, evidence maps generally have a broad thematic scope covering a range of interventions and outcomes. For example, 3ie maps generally cover a sector or broad sub-sector such as forest conservation (Puri, Nath, Bhatia, & Glew, 2016), youth and transferable skills (Rankin, et al., 2015), peacebuilding (Cameron, et al., 2015), and primary and secondary education. SBU has mapped gaps (‘uncertainties’) for topics such as attention deficit hyperactivity disorder (ADHD), home care, and evidence-based nursing for treatment of people with schizophrenia. Department of Veteran Affairs has topics such as ‘Women veteran health research literature’(Elisheva, et al., 2017) and ‘Evidence map on mindfulness’ (Hempel, et al., 2014). Since 2007, the Global Evidence Mapping Initiative has mapped 1,644 neurotrauma publications, covering fifty-three high priority questions (Bragge, et al., 2011). An example of evidence map by Global Evidence Mapping Initiative is ‘Social and communication disorder following brain injury’. Social Care Institute for Excellence maps cover areas such as the extent and impact of parental mental health problems on families, and the acceptability Salina Bates, 2006), accessibility and effectiveness of interventions, second examines the recovery approach in day services in adult mental health care (Sharif, Brown, & Rutter, 2008). As of January 2018, the Epistemonikos database contains over 115,000 documents, of which 22,649 are systematic reviews, 76,834 are primary studies and 17,144 are structured summaries of evidence. The database is available in nine different languages: Arabic, German, English, Spanish, Italian, Dutch, Portuguese, Chinese, and French. It contains 7,266 official translations, 642 collaborative translations and 899,022 automated translations (Rada, Pérez, & Capurro, 2012). The database supports maps of the evidence found by database searches – these maps show which primary studies are reported in which systematic reviews found from a particular search. IZA's World of Labour presents reviews on issues related to labour market policy and related economic issues such as ‘Do immigrant workers depress the wages of native workers?’(Giovanni, 2014). The reviews include an actual geographic map of where the evidence comes from. The scope of Sightsaver's evidence map is limited to eye health. They have published five evidence maps so far, on cataract, diabetic retinopathy, glaucoma, trachoma and refractive error. The Beam Exchange evidence map is focused on market-based approaches to promote economic development, improve access to services and reduce poverty (Robinson & Rust-Smith, 2017). The IRC has published five evidence maps covering the organization's five main outcomes areas: health, education, economic wellbeing, safety, and power. In addition, there are three cross-cutting maps on cash transfer interventions, service delivery interventions, and interventions in humanitarian emergencies. The scope of the evidence maps of the Collaboration for Environmental Evidence is typically broader. Examples include woodfuel value chain (Sola, et al., 2017) and farm land abandonment at high altitude (Haddaway, Styles, & Pullin, 2014). An exception to this general broad scope of maps is the Evidence Based Policing Matrix which has just one outcome (reducing crime and disorder) rather than a set of outcomes. ODI (Marcus & Cunningham, 2016) summarized evidence concerning initiatives and outcomes in four evidence maps: one providing a visual overview of all the evidence in the database, one focused on initiatives with young people acting as agents, one on young people as advocates, and one summarising evidence on initiatives that involved both agents and advocates activity. UNICEF-Innocenti have published an evidence gap map (EGM) of the evidence base for adolescent interventions in low- and middle-income countries (LMICs). The thematic scope broadly corresponds with the UNICEF adolescent well-being outcome domains of protection, participation and livelihoods. All the agencies reviewed state pre-specified inclusion criteria for their evidence maps. Inclusion criteria for maps use the PICOS approach of population, intervention, relevant comparison groups (effectiveness studies only, though specifying comparisons is usually not included in map inclusion criteria), outcomes, and eligible study designs. In practice, there will usually be some iteration between the studies identified and finalizing the intervention and outcome categories. Most agencies produce maps including both systematic reviews and primary studies. Exceptions are the Evidence-Based Policing Matrix which only includes primary studies, and both SBU and Sightsavers maps include only systematic reviews. 3ie EGMs are maps of effectiveness studies, and so the primary studies are only studies of effects, that is impact evaluations. Unlike systematic reviews, evidence maps may include on-going studies. This is especially the case for on-going reviews since registering review protocols is common practice, which remains less the case for primary studies. 3ie staff have argued that the term EGM should be restricted to this sort of map (Snilstveit, Vojtkova, Bhavsar, & Gaarder, 2013). compare Evidence Gap Maps, as supported by the International Initiative for Impact Evaluation (3ie), with other evidence mapping synthesis products, such as scoping reviews, systematic review and rapid evidence assessment. They propose that gap maps focus on studies assessing intervention effectiveness, as well as systematic reviews of such studies, while other mapping/ scoping approaches may include a broader range of evidence to address questions other than those of intervention effectiveness. Epistemonikos uses systematic reviews as the fundamental piece of the database architecture and includes articles linked to an included systematic review i.e. overviews of reviews including evidence-based policy briefs and guidelines, primary studies included in systematic reviews, and structured summaries of that evidence. Some agencies restrict their included studies in specific ways. UNICEF-Innocenti exclude transferable skills and youth employment-related interventions and outcomes as other EGMs address these. Beam exchange evidence gap includes empirical data with studies not earlier than 2000 as this when original framework document for making markets work better for the poor (M4P) was developed. Critical appraisal for the quality of studies indicates the confidence in the study findings. For example, in 3ie maps systematic reviews are colour coded green, orange and red corresponding to high, medium and low confidence in study findings. 3ie does not conduct critical appraisal of primary studies on the grounds that it would be too time consuming given the time frame in which maps are usually produced (3-6 months). In 2017, BEAM Exchange updated the original evidence inclusion protocol to include an additional evidence quality grading system, which assesses all evidence documents as either ‘high confidence’ (represented by purple ‘bubbles’) or ‘low confidence’ (represented by orange ‘bubbles’). Critical appraisal is applied using a pre-specific critical appraisal tool or chechklist. 3ie, UNICEF-Innocenti and Sightsavers use and adapted version of the SURE checklist, and in SBU and ODI apply the AMSTAR checklist. Epistemonikos presents critical appraisal of articles included in structured summaries. For CEE, critical appraisal (the formalised assessment of reliability and risk of bias in individual studies) may be performed to some extent, but this is typically restricted to an assessment of study internal validity (quality or susceptibility to bias), since external validity (generalisability) cannot typically be assessed for the broad topics often investigated with systematic maps. The Evidence Based Policing Matrix reports primary studies only, for which critical appraisal is performed using Scientific Methods Scale (SMS). Critical appraisal is not presented by SCIE and GEMI. Our recommendation that critical appraisal of all included studies is desirable, and should be performed unless resources do not allow it. Search is comprehensive and systematic, similar to systematic review search strategy and is sometimes more purposive for Impact Evaluations. The choice of source varies among organisations and may include: There is considerable overlap in the search strategy adopted by these organizations for evidence maps and there is uniform adoption of a comprehensive and systematic search and involves search across all the sources found relevant for the research question. However in ‘3ie’ search for IE may be more purposive. Key difference in the source for search is that, ‘3ie’ does not involve hand searching of journals or literatures and ‘Evidence Based Policing Matrix’ specifically involves searching website of professional and government organization. In Epistemonikos, articles coming from different databases are de-duplicated using software and a manual revision of borderline cases. Coding captures key features of included studies allowing analysis of the map. The more data are captured then the more dimensions the map can have, or filters be applied to restrict the map to a subset of the evidence by user-determined criteria. The following domains have been used by different organisations for classifying included studies (Table 3 provides an overview of which organizations use which study characteristics): In addition, study publication details are of course coded so that users can locate the evidence underlying the map. How these data are represented is discussed in the next section on reporting / presentation. Typically, Intervention categories, Outcome and study are used for in all the methodologies of evidence mapping. of Outcome in 3ie gap maps and Evidence Based Policing Matrix in Evidence Based Policing key intervention dimensions of of of intervention, of and outcome and of effect are used for data while in 3ie, studies are coded to relevant intervention and outcome included in the study characteristics are included in maps and use from the International of and The is an exception in being the only map to report what the evidence says and of In Epistemonikos, the document is and document such as primary studies, systematic reviews, overviews and the approaches have been used for presentation of evidence maps by the different the a a as map and descriptive report In the two additional dimensions may be use of different and to studies in the map. Each of these approaches is in The is used by Evidence Based The shows five dimensions to the and show the effect and of the literature Each three there is a of each intervention, with the of the the of reviews for that intervention and the of the the intervention Figure shows an example of Evidence map on by evidence synthesis Evidence Based A is used by ‘Evidence Based Policing Matrix’ to single studies in three intervention on the by the of the intervention from general to and its of from to Each study is by a or coded to show study i.e. are as a in the map. This approach shows of studies if they are for a common intervention, are to as of These of effectiveness are to provide into the and of and can be used by agencies to guide various of their this approach is not a evidence and a approach to the evidence. The matrix mapped with intervention studies Evidence Based Policing Matrix 3ie present their evidence gap maps in a framework that provides to summaries of included studies. is used to the quality of the systematic review evidence in the gap map so that users can the state of the evidence in the field use The same assessment is not made for primary studies. The of each the of studies. and health EGM 3ie and health evidence gap Evidence Gap Map on adolescent well-being in low- and middle-income protection, and and well-being UNICEF EGMs are on a matrix framework to evidence on specific interventions or focus The focus areas for which reviews were identified are found the and the of evidence is the The in the matrix the of a systematic or literature review the relevant focus on a will up a The of evidence presented in the reviews is gap map The BEAM Evidence Map and database and presents evidence on the impact and effectiveness of market approaches. The the of intervention which to the in the the of the evidence which the document contains Beam Exchange evidence map Beam Exchange A matrix of evidence is a all the systematic reviews a question, and all of the studies included in these reviews. The matrix the systematic reviews in the and the studies included in those reviews in the All the systematic reviews in the address the of interest for or a broader that includes the of interest for a Matrix of Evidence Epistemonikos IRC evidence maps are not available as tool but are available in map IRC The are on the people as agents and advocates of development ODI World of map indicates for which countries empirical evidence on a specific topic The of key and additional in the provides the basis for this map. empirical evidence for one particular that will be in a different from to different of The colour of the it is to the used in The Global by the World for Global and A that is not in the will be This feature can be as users the of given the study as captured by its The the indicates many relevant studies address this policy question. immigrant workers depress the wages of native Evidence map An additional feature is that maps may be available with the that the user can on a map to access either a study i.e. publication details and a of the These summaries typically include information Some agencies do not provide but present the of the mapping in a descriptive A descriptive report the of evidence and its such as and which study have been The reports identify gaps and SBU present in the of a descriptive report and a to show where there is evidence as in the SBU a systematic review with high or quality the effectiveness of treatment of following SBU Global Evidence Mapping Initiative presents maps with descriptive and study Prevention and of in the of Global Evidence Mapping Initiative Collaboration for Environmental Evidence use and to the quality if of evidence across a of they present study as a a information That may be a map the and of included studies reports methodologies are used by Evidence Mapping ‘Social Care Institute for for Environmental and ‘Evidence for Policy and Practice Information and Coordinating Centre Some such as 3ie and the which provide provide a descriptive There are variations in the evidence can be the map. The of 3ie maps, for example, the user can access a database for each included which in has a to the study though that may be a to of and is with Beam ODI and Evidence map have on the of Evidence maps are used to inform research for commissioning and the use of evidence to inform decision making by Global Evidence Mapping ‘Social Care Institute for for Environmental ‘Evidence and ‘Evidence Based Policing ‘Evidence for Policy and Practice Information and use evidence map as a of the in systematic review or to help make a decision about or not to a systematic review. Evidence maps have been used to identify evidence gaps by Social Care Institute for for Environmental ‘Evidence For example, used as to gaps for evidence for as the basis for a to identify the research et been used for and by For and of Social The Global Evidence Mapping for Environmental and ‘3ie’ evidence map and ‘Evidence Based Policing Matrix’ are used to inform policy decision and as tool for adoption of evidence research and particularly to visual of evidence and The search for an evidence map is undertaken up to a particular point of The map of as new evidence Maps more than two to three years are less as a guide to research though this is if the map has captured on-going studies. as with systematic reviews, to relevant maps need to be updated or the agencies reviewed, the Evidence Based Policing Matrix and Epistemonikos are updated on a SBU the evidence gap in their database but not the evidence maps. Some Global Evidence Mapping Collaboration for Environmental Evidence has a policy that maps should be but there is no for that this Based on the above Table the and between evidence and gap maps and systematic reviews. There are many of evidence and gap maps are systematic, so need an protocol with an explicit search strategy with clear inclusion and criteria. The main difference is the more limited of studies for a map, since it only reports what studies there are not what it says. Based on the we propose the following principles for producing evidence maps: updated three years of last if scope is broad or field Evidence and gap map definitions the definition used at a evidence gap when Evidence Maps, an of the Gap Map approach developed by the International Initiative for Impact Evaluation Evidence Maps and present the available research evidence at of the of to what interventions do and do not work in each outcome EGM is a with outcomes of interest the and interventions of interest the cash