Evaluation of a stakeholder advisory board for an adolescent mental health randomized clinical trial

INTRODUCTION: Community engagement in research is widely accepted as best practice, despite gaps in existing frameworks to evaluate its process, context, and impact on research. The Screening in High Schools to Identify, Evaluate, and Lower Depression (SHIELD) study evaluated the use of a school-based major depressive disorder screening tool in the identification of symptoms and treatment initiation among adolescents, and was developed, implemented, and disseminated in partnership with a Stakeholder Advisory Board (SAB). We summarize outcomes of the evaluation strategy applied through our partnership with the SAB and explore gaps in the available engagement evaluation tools for mixed stakeholder populations including youth. METHODS: SHIELD study SAB members (n = 13; adolescents, parents, mental health and primary care providers, and professionals from education and mental health organizations) advised on study design, implementation, and dissemination over a three-year period. Both SAB members and study team members (i.e., clinician researchers, project managers) were invited to quantitatively and qualitatively evaluate stakeholder engagement after each project year. At the conclusion of the study, SAB members and study team members were asked to evaluate the application of engagement principles in overall stakeholder engagement across the study period, using portions of the Research Engagement Survey Tool (REST). RESULTS: SAB members and study team members responded similarly when evaluating engagement process (i.e., valued on team, voice represented); means ranged from 3.9 to 4.8 out of 5 points across all three project years. Reported engagement within study-specific engagement activities (i.e., meetings, study newsletter) varied from year to year, with some discrepancy between SAB member and study team evaluations. Using REST, SAB members reported the alignment of their experience with key engagement principles the same or higher than study team members. Qualitative feedback at the conclusion of the study generally matched quantitative measures; adolescent SAB members, however, reported disengagement from stakeholder activities that was not accurately or effectively captured in evaluation strategies employed across the study period. CONCLUSIONS: Challenges exist in effectively engaging stakeholders and evaluating their engagement, particularly among heterogenous groups that include youth. Evaluation gaps should be addressed through the development of validated instruments that quantify the process, context, and impact of stakeholder engagement on study outcomes. Consideration should be given to collecting parallel feedback from stakeholders and study team members to fully understand the application and execution of engagement strategy.