Systematic Review of Quantitative Measures of Stakeholder Engagement

Stakeholder engagement in research has received increasing attention in recent years.1, 2 The term “stakeholder engagement” refers to the process of meaningful involvement of those who are engaged in making decisions about programs.3 Engaging members of the target population is often key to improving the relevance of the issues studied, the procedures used for study, and the interpretation of outcomes of research studies, health promotion activities, and disease prevention initiatives.4-6 The utility of stakeholder engagement has been well established in the literature,7-9 but there are few examples of measurement and evaluation of the degree to which stakeholders are engaged in these activities and the impact of engagement on positive outcomes. These types of evaluations have been limited in scope, and largely focused on qualitative approaches.10-14 Qualitative methods cannot be easily compared across programs or institutions.15 Necessary reliability and validity information describing self-reported levels of stakeholder engagement are also lacking, and is essential to identifying the impact of engagement on the scientific process and scientific discovery. We present the results of a systematic review of the existing quantitative measures of stakeholder engagement in published research and programs. We used the methods of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA16) to review the literature on measures of stakeholder engagement. We defined stakeholder engagement as the involvement of people who may be affected by a research finding or program. We included any type of stakeholder that was available in the literature, and included all settings and situations. We did not limit the search to research or intervention projects, because we reasoned that the field was relatively new, and therefore we might find good measures in multiple areas from which to draw our search. All authors contributed substantially to multiple aspects of the article: (1) the concept and design or analysis and interpretation of data, (2) the drafting or revision of the article, and (3) approval of the final version. We searched the peer-reviewed literature using two electronic bibliographic databases: PubMed (web-based) and the Web of Science (web-based). These database searches for all years until 2013 were conducted between July and September 2014. The 2014 search was conducted in January 2016. With assistance from a reference librarian, we generated a master list of search terms to use with both databases. The following Medical Subject Headings (MeSH) terms were selected: stakeholder engagement, community engagement, community engaged research. These terms were then entered into the two chosen databases using quotations to ensure that search terms were verbatim and separated by “OR” to ensure inclusion if any of the search terms were present. A master list of articles was created using Endnote X8 citation management software from these search results. All published reports from 1973 to 2014 were identified and retrieved. Two authors independently reviewed each title and abstract using predetermined inclusion and exclusion criteria and resolved disagreement through rereview and discussion until they reached consensus. Included articles appear in English, in a peer-reviewed journal, report original research, and appear to use a quantitative measure for at least one construct reported to measure stakeholder engagement. The full texts of articles whose abstracts met our inclusion criteria were retrieved and then examined for further inclusion/exclusion criteria. Our primary exclusion criterion was an article that does not contain stakeholder engagement, then subsequently, lack of quantitative measures. To ensure consistency in data abstraction, we created a standardized codebook for use by all authors, based on a prior measurement review. All authors reviewed the same five studies using the codebook and identified and resolved disagreements in coding. The codebook was thereafter revised. Included studies were distributed among seven coders for data abstraction. For all studies included in the review, two authors independently coded and compared their results. Coding discrepancies within pairs were resolved through discussion among authors. We opted for this group consensus method because of the enormous variability in terminology used to describe engagement characteristics and constructs across studies. The variables extracted included constructs measured, names of measures, stakeholder group engaged, type of engagement project, whether or not the measurement was part of a training exercise, and relationship of construct to study outcome (if engagement was assessed to be related to the dependent variable in the study). Data were entered into Excel files and evidence tables were constructed, organized by article first author, and stratified by type of construct. We divided the abstracted measures into (1) measures that were collected using participant-reported methods, no matter how the self-report was collected, such as online, paper and pencil methods (n = 53), and (2) observational methods, defined as a variable that asked researchers to observe and quantify behaviors relevant to engagement, such as attendance at a public event (n = 51). Figure 1 contains the data on article eligibility and coding patterns. The search identified 3,576 (PubMed n = 1,112, Web of Science n = 2,464) articles using our key words. Six articles were excluded for being non-English language (one Chinese, two German, two French, and two Spanish language articles). The abstracts of these articles were translated and determined to not impact the outcome of this review. From the remaining articles, 741 were excluded for being duplicate articles found because the databases used had some degree of overlap. After initial screening, we identified 2,829 non-duplicate English language titles (see Figure 1). Of these, 2,582 were excluded during the abstract and title screening phase of the review, leaving a total of 247 articles. We excluded 179 articles at the final coding stage because they ultimately did not meet our eligibility criteria when the full-length article was coded and discussed. A total of 68 articles contained quantified measures of stakeholder engagement and met our final criteria.17-84 Table 1 contains the 53 participant-reported measurements of stakeholder engagement found in 38 studies identified in our search. In each of these studies, participants indicated the extent of their engagement in the project using responses to one or more scales. The types of projects included here were broad: research projects, community input projects, and interventions. As seen from this table, none of the articles use the same stakeholder measure. Only a small number of reviewed articles (5 out of 53; 13%) reported psychometric data about the measure. Some of the measures (5 out of 53; 13%) had reliability calculated in the form of alpha coefficients, yet none of the scales presented any information on content validity or on other types of validity (i.e., criterion, construct). The stakeholder populations targeted in these studies widely varied, from members of a defined general public to participants in community groups and members of advisory boards. Only 25 of 53 (47%) measures reported assessment and testing of the significance of the relationship of participant-reported engagement measure to outcome. Of those that assessed the relationship of measure to outcome, 100% of the studies indicated a significant relationship between engagement measure and at least one of the outcomes. A total of 50 separate observational measures were used to assess stakeholder engagement by counting or recording behaviors, from a total of 34 articles identified in the search (Table 2). The observational measures of engagement included “counts of referrals” and “attendance at events.” No reliability data were presented along with the identification of each measure in this table. No attempts were made in any of the studies to support the validity of the observational measures of engagement through measuring a hypothetically related construct alongside the observational measures. Many of the measures were related to outcomes as part of the study but the outcome testing used a variety of measures. Representation Process Information Outcomes What planned and actual procedures were used by Healthy HotSpot staff to recruit community institutions What planned and actual procedures were used by community partners to recruit stores Feel like they belong in their community; Know their way around their community; Know the rules; Feel accepted; Can be independent in their community; Like where they live, Feel close to others; Know people well enough to say hello; Have fun things to do; and 10. Have productive activities to do in their community. We systematically reviewed the literature on existing measures of stakeholder engagement. We found a variety of measurements, with differing qualities and development trajectories. Some were simply counts of event attendance, while others were theoretically based and developed with sound psychometric principles and analyses. The variability in the process of identifying these measures listed in the articles found for this review was considerable, making grouped analysis difficult. Many of these measures were participant head counts by researchers or participants themselves. As such, these do not really measure any sort of engagement directly. Therefore, we cannot tell if these measures actually assess engagement or some other cluster of factors that motivate people to attend events and activities. On the one hand, these counted types of measures are relatively easy to obtain and can be gathered from documents that already exist, such as meeting minutes or counts of attendees at events. Thus, they are an easy measure to gather and use, and are often presented without much extra work. But likely we need more data before considering that high attendance equates to high engagement in a process, particularly in light of potential confounders such as incentives for attendance. These types of data would be relatively easy to collect and produce, and this systematic review points to the need for these corroborating data. Also, there is no consensus in the literature on defining engagement, what kind of engagement is desired, and how involved a community member must be to be considered “engaged.” These are all likely context-dependent, and therefore expectations of consensus are not relevant. The participant-reported measures of engagement are varied and diverse in their names, definitions, and purposes. Data on the psychometric properties of scales were generally lacking, although some investigators provided some limited data on scale performance, presented in Table 1. The scales measured a broad range of concepts, including motivations for participating, strength of relationship between researcher and community, comfort with community activities, and familiarity with community members. This breadth indicates that we need clearer definitions of the construct(s) involved in engagement before new development occurs. In most of the publications, the development process was not detailed and no plans were proposed to identify scale psychometric properties. This absence could pose a problem when comparing across multiple communities or when attempting to obtain a point estimate of engagement to better understand a research process or community or stakeholder's actual involvement, especially in large-scale research efforts. Weiner and others have identified a similar lack in the implementation science literature as well,85 pointing to a potential challenge for new efforts, such as Patient-Centered Outcomes Research Institute and the Precision Medicine Initiative. There were several choices that we made that limit the generalizability of the findings. First, we did not formally rate the quality of each measurement study. We found that our initial attempts to rate measurement quality resulted in low ratings, due to the absence of psychometric data. This is likely due to the early nature of this field, but future work will hopefully find improvements in the methodological development of measures for engagement. We did not review related constructs that could be used to assess some component of, or were related to, engagement, given the lack of definitional clarity and term usage across literatures. Therefore, we likely did not include measures that could have provided some information on this topic. Our focus on English-language articles is a limitation of the data as presented; however, our assessment of the non-English articles we were able to translate indicates that this did not seem to bias the review. What is the right kind of engagement measure to use? First, we clearly need to develop better scales that (1) use a theory or model, (2) provide psychometric data, (3) can be short and used with large-scale projects, and (4) pick up the key elements of engagement that are critically important for involvement in health-related projects. These are high standards indeed, but feasible. Some scales exist, but because relationships and purposes of engagement vary so widely, selecting the right scale for the situation will depend on context, goals, and relationship. It is possible that no single scale will likely meet every need. This ultimately is an empirical question. Given the increasing interest among researchers and funders in improving health outcomes through engagement of diverse stakeholders (communities, groups, individuals) in health-related research and programs, we recommend greater attention to developing good quality (validated, yet flexible) measures that can help us assess the level of engagement throughout an ongoing process, specific to varying types of engagement, and health outcomes related to the engagement process. Better measures would be particularly helpful in assessing engagement of diverse groups of stakeholders in a research project. When stakeholders can communicate their values in a research team, those values often influence the research process and can be an important source of new research questions, as well as a source of more accurate interpretations of the research findings.86 Some of the most interesting conceptual work is going on in qualitative projects. Qualitative data can accompany or inform quantitative approaches in mixed methods projects. For example, Schulz and colleagues10, 12 applied both qualitative and quantitative approaches to evaluate process and group dynamics in community-based participatory research (CBPR) projects. Their evaluation addressed perceptions of openness, trust, and ownership, and informed an annual quantitative survey in which partnership members rated these aspects of group functioning. Questions such as one's “sense of ownership/belonging to the group” may be construed as measuring a deep level of engagement. More qualitative research can improve the field's understanding of how various stakeholders describe their expectations for levels of engagement and inform quantitative scales representing engagement in the words and thoughts of the original stakeholders. The volume of research in this area has increased over the last few years, so we are hopeful that we will see the development of high-quality measures in these large studies. This work was supported by grants from the National Human Genome Research Institute P50 HG 3374.and K99HG007076 No author reports having a conflict of interest with this work.