Evaluation of learning in emergency medicine: An umbrella IRB protocol for education outcomes research

Medical educators have a responsibility to evaluate teaching innovations, analyze curricular changes, and assess learning. This context provides opportunities for scholarly productivity and dissemination of new educational innovations. However, educators may stop short of formally studying their interventions or publishing their innovations in part due to the administrative hurdles presented by the institutional review board (IRB) process. Lost opportunities for scholarly productivity have implications for individuals and for the specialty. The ACGME requires residents and core faculty to engage in scholarly activity1, 2 and while basic science and clinical research are encouraged, education outcomes research is an attractive option. Boyer and Glassick advocated to increase the visibility and recognition of education-based scholarly work.3, 4 The application of medical education research can shape educational policy and practice both institutionally and nationally.5 Significant educational effort that goes unshared and unrecognized keeps medical educators working in silos at their own institutions, potentially doing redundant work. This innovation addresses one barrier to educational scholarship, research, and dissemination: the hurdle of completing the IRB approval process. IRBs are responsible to ensure that research complies with applicable regulations, meets commonly accepted ethical standards, follows institutional policies, and adequately protects research participants. IRBs may view educational research through several different lenses. Some research may be deemed “quality improvement” or “not human subjects research” and not require formal IRB review. Many education studies will require IRB reviews and may be deemed IRB-exempt due to “legitimate educational interest” or due to the use of deidentified data. A different approach is to consider overarching research frameworks, known as “umbrella protocols.” These have been created to expedite IRB approval of related studies within a clinical data registry,6 which serves as a repository of clinical data about a group of individuals (e.g., trauma registry). While commonly used in clinical medicine there is little literature on the application of this concept to education-based research.7, 8 This innovation report provides guidance to faculty seeking to streamline their educational scholarship through an umbrella IRB protocol. Umbrella IRB protocols operate through the foundational understanding that the corpus of trainee data can be considered an educational registry. Creation of an educational data registry with IRB oversight allows educators to perform program level analysis of curricula, assessments, and interventions. Instead of submitting separate IRB protocols for each study, the processes of protocol development and data collection are expedited and streamlined through a single infrastructure, removing a key barrier to scholarship for clinician educators. This innovation shares the steps and lessons learned through protocol creation, approval, and implementation at our institution. The basis of the educational umbrella protocol is an educational registry from which data are pulled to answer research questions. Creating an educational data registry allows investigators to ask a variety of study questions and facilitate the export of deidentified multisource data for analysis. Given the above aims and assumptions, we prespecified a comprehensive list of data sources to guide the creation of an educational data registry (see Figure S1). We anticipated data sources as much as possible during the initial phase of protocol creation, knowing that adding data sources later would require an addendum to the original IRB protocol. Within this registry, learner data can be combined with clinical data in the aggregate, without patient identifiers, to gauge whether a curricular intervention or assessment tool translates to improvement in real-world patient outcomes. Validity studies under the umbrella framework are possible by combining data streams from other instruments for analysis across the learner continuum from UME to GME. Other educational data streams could be used in conjunction with surveys of learner satisfaction that are routinely collected to assess the learning experience. The Kirkpatrick framework of education outcomes research is commonly used for curricular evaluation (Table 1). Study our learners’ changes in their knowledge, skills, and attitudes using pre- and post-intervention assessments Study learner performance using existing assessment data that are already created for the purposes of formative and summative feedback Do faculty that engage in a structured, small group workshop with a TEE simulator learn the required views better than a faculty who practices independently? Use real-world clinical metrics to illustrate that our educational methods and assessment tools correlate with improvement in patient care in our ED Use aggregate deidentified clinical data on efficiency (patients per hour, resource utilization) and safety (quality assessment triggers and scores) to answer whether our curricula translate into a real-life environment and whether our assessment tools can accurately predict high and low clinical performers We developed an internal process for proposal review and approval to ensure that all substudies meet the inclusion and exclusion criteria of the umbrella IRB protocol. A steering committee composed of education researchers familiar with the umbrella protocol intake proposed studies and perform an initial review. Prospective study protocols are then reviewed in depth by two steering committee members who are not co-investigators to mitigate conflict of interest. All study oversight is conducted within our department's education research group, not the university's IRB, with tracking of proposed, rejected, and approved studies through RedCap. We adopted the approach of a virtual registry, meaning that we abstract and link learner data on demand following substudy approval. We do not routinely or continually abstract and input educational data and thus do not have a database containing all potential data sources waiting for use by investigators. This was a practical decision, as we did not want our education program managers using valuable time abstracting data streams that may go unused. Our research framework was deemed IRB-exempt at our institution in October 2022. From that point on, studies falling within its strict inclusion criteria upon internal review could be deemed IRB-exempt by extension under the umbrella and allowed to proceed. Creation of the data registry presented an early logistical challenge. Previously published education data registries7 have used rolling data entry into a large database. While comprehensive and ideal, this approach is challenging without significant data management support. With rolling data entry, considerable time and effort goes into abstracting and entering educational data points that may go unused. Given the expected silos between UME and GME, our data streams came from a broad range of disconnected sources. We opted for a virtual registry, abstracting and entering only the educational data needed for a particular study. By creating a data registry on demand, multisource data can be linked using a unique identifier for each learner or analyzed in aggregate form, depending on the investigators’ needs. This can be done by administrative personnel, such as clerkship and program coordinators, or by faculty that already have access to the identified data as part of their educational role (provided that these personnel are not investigators in the proposed substudy). While all data are deidentified, approved studies are monitored routinely to limit the exposure of educational data. Only the data needed to answer a substudy question are released and access is revoked once each substudy is complete. Since the IRB exemption of the educational umbrella protocol in October 2022, four projects have been approved through its internal review process. Most approved projects (Table 2) explore the effectiveness of curricular changes or the implementation of new curricular innovations. Four other projects have entered into the review process. Of these, two of these projects would have met the inclusion criteria for the IRB-approved protocol but were abandoned due to logistical reasons separate from the umbrella substudy approval process. A third project targeted a learner group (faculty) not part of the original umbrella protocol inclusion criteria and has been placed on hold pending an amendment of the protocol. A fourth project was determined to be better suited for a separate existing IRB-approved protocol. Ultrasound peer review scores Timing and number of ultrasounds performed Level 2 Level 4 Level 1 Level 2 Level 1 Level 2 While IRBs operate under a common set of research ethics and governing laws, their receptiveness to an overarching protocol like ours will be variable. Some IRBs will not approve an “umbrella” concept, citing concerns over expedited study approval from a subgroup with a vested interest in productivity and lack of adherence to the protocol over time. In this case, a protocol submission for an educational data registry alone may be a better option. Learner protection is paramount, and there are different legal protections in place in UME and GME. While the Family Educational Rights and Privacy Act (FERPA) does not apply to GME learners, we obtained a FERPA waiver permitting the use of medical student data without learner consent, as all data streams within the data registry are already collected as part of the student educational experience. Despite all substudies being minimal to no risk of harm, institutions may require informed consent of all learners to ensure transparency and psychological safety. If consent is required, there must also be a process in place to opt out of the data registry at any time. It is best to create a comprehensive list of potential data streams up front during initial IRB submission of an umbrella framework. Individual studies must use data streams that are predefined by the umbrella protocol. Each time a data stream is added, an addendum must be submitted to the IRB. For example, faculty learners were not included in our original umbrella protocol IRB submission and we have submitted an amendment. Individual protocols and data streams should be reviewed periodically to ensure adherence to the core aims of the umbrella protocol. By removing the IRB submission and approval process as a significant barrier, residents and faculty who occasionally engage in research or are new to education research entirely are more likely to engage in scholarly work. Experienced education researchers can further increase their academic output by no longer having to submit new protocols or modify existing protocols to launch new studies. With diminished regulatory burden, our principal investigators can fully embrace a mentoring role in collaborative research groups. In our department, we have noticed increased cross-disciplinary collaboration with researchers that typically work outside of the education domain, evaluating the interaction between their clinical and operational research, learner behavior, and patient outcomes. Our early experience with an educational umbrella protocol can serve as a blueprint for other departments to do the same. With an upfront investment in creating overarching protocols and research processes for education outcomes research, our entire field stands to gain. Publishing curricula and assessment tools with greater frequency can help medical educators share information across institutions. If multiple institutions invest in the creation of similar protocols, multisite comparisons of longitudinal educational outcomes become possible. Publication and scholarly expertise remain the primary currency for academic promotion, and an umbrella protocol can help educators earn academic credit for the work they already do. The authors declare no conflicts of interest. Figure S1. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.