The Education Data Warehouse: A Transformative Tool for Health Education Research

The science of health professions education research is growing increasingly sophisticated across the spectrum of training.1 Outcomes by which learners and curricula are assessed are moving away from test performance to more nuanced measures of clinical reasoning, behavioral changes, and even clinical patient outcomes.2 There are national calls for standardizing these outcomes in support of individualized learning paths and a longitudinal view of a physician's progression through his or her education.3,4 Rigorous comparative studies designed to answer these new questions require larger numbers of participants, often more than a single school or training program can provide, making collaborative educational research necessary. This emphasis on improved educational outcomes tracking also requires new methods and tools, some of which are being adopted from epidemiologic and clinical research.5 In this issue of the Journal of Graduate Medical Education, Carney and colleagues present the P4 project, a collaborative effort of 14 residency training programs in a unique practice-based research network designed to compare the relative impact of educational variations in their curricula and training programs.6 In this framework, each residency program can implement locally designed and tailored curricular changes, but overarching and standardized instruments are used to collect consistent data across all participants on a longitudinal basis. These outcome data are gathered both by directly surveying participants and also by aggregating electronically available national data sources such as the Family Medicine Board scores, the Accreditation Council for Graduate Medical Education, and the National Residency Match Program. The authors describe the process by which multiple points of data were collected for each participant to measure the trajectory of the participant's competency development. Data from 330 residents and their 14 respective programs were collected; these data will produce far more powerful results than any one program would have achieved. This project successfully aggregated and normalized the data across the disparate training programs and integrated information from national electronic databases. This “data warehouse” approach enabled the authors to evaluate rigorous and complete data on both the context of local educational interventions and potential sources of variation in learner outcomes; each training program could evaluate itself over time (intraprogram), and against cross-institution measures of curricular design variations (interprogram). The P4 example has the further benefit of allowing prospective comparisons of more than one curriculum without requiring individual programs to implement both “control” and “intervention” curricula, a task that is especially difficult in a busy residency program. The authors propose that the P4 project could serve as a model for Centers of Excellence in Educational Research. Such centers would be made up of institutions that pool data in longitudinal databases for the purpose of educational epidemiology, essentially forming a type of educational Framingham study.7 A task force from the 2007 Millennium Conference previously defined this type of education research database as “an information repository containing measurements prospectively collected over time on individuals from multiple educational/training programs and professional settings with the intent of conducting analyses to answer specific research questions relevant to medical education.”8 We propose the term education data warehouse (EDW) as one that captures this essence and conveys the parallels that these systems have with their clinical and research equivalents (Figure). FIGURE 1 Standards Enable Integration of Data Across Programs and Institutions The large-scale development of EDWs will balance the great potential rewards against considerable challenges and complexities. Encouragingly, emerging standards for health professions' education data, such as those created by the MedBiquitous group,9 are lowering the barriers for cross-institutional collaborations. Standards such as the Healthcare Learning Object Metadata and Medical Education Metrics will allow institutions not only to efficiently aggregate learner data from different sources, but also provide a mechanism for continuity of data as learners progress from medical school through residency, and into continuing medical education. The American Association of Medical Colleges is undertaking an effort to create a comprehensive Curriculum Inventory Portal,10 a repository of medical school curricula that includes teaching methods, assessment methods, and topics taught. These types of detailed data provide a much richer measure of the “educational dose” that each learner has received. When performing studies like those that Carney et al describe here, understanding learner outcomes in the context of detailed information about teaching modality, duration, assessment strategy, and instructional technique is critical to drawing robust conclusions. One example of a successful EDW is the Jefferson Longitudinal Study of Medical Education. Using its data on 8000 students tracked during the course of 40 years, the group recently published a rigorous study of MCAT examination performance as a predictor of medical school and residency performance.11 Perhaps the most powerful, and the most challenging, advance from the EDW technique would be the ability to automatically link a learner's education and training with patient-level clinical outcomes.7,12,13 Combining these data on a large scale will provide unprecedented insight into how the education of students, residents, and practicing physicians translates into improved patient health. To be meaningful, this type of evaluation requires long observation periods of prospective data as participants cross institutional, clinical, and educational boundaries. A key enabler of this effort will be the growing number of health information exchanges, regional or state-level collaborations in which health care provider organizations share clinical data with training institutions. To achieve broad use of EDW-enabled health education research, health education will need new capabilities and resources. Faculty members with skills in large-scale databases, educational informatics, and the issues associated with integrating disparate data sources will need to be integral members of the research team. These faculty members will work alongside education and hospital administrators cognizant of the value of longitudinal education research in their respective missions. As mentioned above, standards and vocabularies are needed to represent not only health concepts like clinical diagnoses, but also educational concepts, such as competencies and educational objectives. Federal funding for the creation of school-based research networks, similar to practice-based research networks, would foster collaboration between institutions and training programs. Our institutions need to work with policy makers to overcome legal or organizational barriers to linking education and clinical data. In addition, policy makers need to recognize that sharing these data is of value, and new techniques of de-identification of electronic outcome data can efficiently minimize risk. Perhaps most importantly, we as a medical education research community need successful demonstrations of the power of this approach, such as the one described in the highlighted article. Although clearly important for academic research, comprehensive longitudinal, multi-institutional EDW data can drive strategic decisions that result in curricular improvements on a national scale. By influencing, for example, accreditation standards and national standardized examinations, collection of these heretofore unavailable data will enable a shift in focus from the concerns of individual medical schools to a still greater mission, that of aligning medical schools with coordinated efforts to improve the overall health care system.