Getting Real(ist) With Program Evaluation

Evaluating educational initiatives can be frustrating and difficult. Medical educators often attempt to use experimental approaches, such as exposure vs control groups or pretesting vs post-testing, to evaluate educational programs or curricula. Such approaches frequently presume clear cause-effect pathways (ie, this curriculum led to that behavior change) and quantifiable outcomes. However, education research often aims to measure social constructs such as competence, medical knowledge, or in the case of the article by O’Hara et al, culture. Such constructs may not lend well to quantitation or linear causality, requiring the use of instruments (eg, surveys) that depend on layers of inferences linking observable indicators (eg, verbal responses) to abstract phenomena (eg, competence).1 These inferential steps are often insufficiently articulated or theoretically justified. Educational interventions are also often multifaceted (ie, multiple components), complex (ie, more than one causal pathway), situated (ie, influenced by context), and emergent (ie, unexpected, novel phenomena develop).2 In this commentary, we describe how theory-informed program evaluation frameworks can be leveraged to better understand, explain, and strengthen complex educational initiatives.In this issue of Hospital Pediatrics, O’Hara et al describe the evaluation of the Teaching Excellence Among Medical Providers (TEAM) program.3 TEAM is an educational intervention in which trained faculty observers assess Pediatric Hospital Medicine faculty and fellows in the clinical environment, providing structured feedback on educational methods, clinical skills, feedback delivery, and written evaluations. The authors developed 6 themes, interpreted through 3 cultural lenses: organization, identity, and practice. Overall, it was concluded that the TEAM program had a positive impact on the local clinical learning environment and educational culture.Program evaluation, which we define here as efforts to understand the changes that occur in response to an educational initiative,4 is an integral part of innovation. Table 1 summarizes three different approaches to program evaluation, though myriad others also exist. Many educators have used an evaluation approach called the Kirkpatrick model,5 which focuses on 4 levels of learner-related outcomes: satisfaction, learning (ie, knowledge or skills gained), behavior change, and system or context impacts. Although relatively easy to understand and implement, the Kirkpatrick model examines only outcomes and does not explore the underlying theories and mechanisms that drive educational change.6 Such program evaluation can lead to oversimplification and assume linear causality within inherently complex systems. For example, only looking at behavior change (Kirkpatrick level 3) might presume that TEAM feedback was the driver of changes in faculty behavior. However, simply being observed (ie, Hawthorne effect7) may truly be the mechanism of change.Notably, O’Hara et al employed a logic model,8 which articulates a more complete program theory than Kirkpatrick’s hierarchy. Logic models can be quite useful in both planning and evaluating curricula by outlining a program’s inputs, activities, outputs, and outcomes. However, although logic models can be constructed with considerable sophistication, they are often applied in ways that emphasize linear cause-effect linkages, underappreciate the impact of context, and do not fully articulate the mechanisms that drive change (ie, why or how change occurred). Without explicit attention to mechanisms and context, logic models risk becoming descriptive maps rather than explanatory theories. For example, in this article, the authors report that TEAM fosters professional identity formation and growth mindset but do not report the mechanisms or social theory by which this occurs. They also do not explore how the local context promotes or inhibits this impact. A more complete understanding of mechanisms and context can enhance program transferability to new settings and allow others to adapt the intervention more effectively across diverse contexts.Although the findings in the article by O’Hara et al are educationally useful, we advocate for program evaluators to explore not only “what happened” but “what worked, for whom, under which conditions, and why.” Realist evaluation is one approach that aims to answer such questions.9 Rather than focusing solely on outcomes, realist evaluation seeks to identify the mechanisms through which an intervention produces change and how those mechanisms are shaped by context. Mechanisms are defined as “underlying entities, processes, or structures which operate in particular contexts to generate outcomes of interest.”10 In practice, realist evaluators develop context-mechanism-outcome (CMO) configurations, which describe how specific features of a setting (context) activate particular processes (mechanisms) to produce observable results (outcomes).11 Repeating CMO patterns, called demi-regularities, can be identified to describe context-dependent mechanisms that drive observed changes. Theories of change (what works, for whom, under which conditions, and why) can then be iteratively developed and refined as ongoing implementation and evaluation occur. Realist evaluation has been used in multiple health professions education program evaluations, including those related to clinical competency committee entrustment decisions,12 mentorship programs,13 faculty development,14 and equity initiatives.15 This approach can be particularly useful in the complex, contextual world of education programs.In TEAM, one possible CMO configuration could be that in a psychologically safe and culturally supportive environment (context), structured observation and feedback (mechanism) can prompt reflective self-assessment (outcome). In contrast, in an environment where observation is linked to summative evaluation (context), the same intervention may be perceived as high stakes (mechanism) and trigger impression management or resistance (outcome). A logic model might describe what happened and in what order (eg, observation and feedback led to improved faculty teaching) while minimizing the role of context or overlooking the underlying mechanisms driving change. Logic models may represent a pathway as stable and predictable, whereas realist evaluation highlights that same intervention works differently depending on context because it activates different psychological and social mechanisms.Realist evaluation recognizes the dynamic changes that occur as a program continues. For example, in TEAM, the developers, faculty physicians, trainees, patients, and situational contexts will change in response to the program, which will in turn lead to new or refined explanations about how and why the program works. There may also be new results or outcomes that arise from these evolving mechanisms and contextual interactions. These emergent processes and outcomes differ from unintended consequences (which O’Hara et al do report); emergence represents qualitatively new explanatory pathways that develop as the program interacts dynamically with its context over time. Through a realist approach, program evaluation becomes an “iterative, explanation-building process”16 rather than an exercise in measuring outcomes at a single point in time.Haji et al provide useful recommendations on how to take an exploratory and explanatory approach to program evaluation.16 Evaluators can articulate an initial program theory by clarifying a set of hypotheses about how and why an intervention is expected to produce its intended outcomes. This begins at the point of program conception and can help designers specify the mechanisms through which the program is expected to operate. For example, before implementation, TEAM developers may hypothesize that structured peer observation fosters reflective practice, which in turn improves teaching behaviors, particularly in environments with strong cultures of feedback. Such a hypothesis can be bolstered by known educational, psychological, and sociological theories such as mindset theory,17 impression management theory,18 and experiential learning theory.19 Outlining these assumptions and supporting theories allows evaluators to move beyond implicit beliefs and instead generate explicit hypotheses that can be evaluated and refined over time.Once a program is under way, evaluators can engage in iterative inquiry, examining whether the program is achieving its intended outcomes (“Is it working?”), how those outcomes are being produced (“How is it working?”), what additional or unanticipated processes are emerging (“How else is it happening?”), and what unintended or emergent outcomes may be occurring (“How else is it working?”). For TEAM, continued exploration of what factors influenced if and how TEAM fostered reflection and behavior change, as well as surprising or emergent phenomena and their impact on the program, can be used to refine the overarching program theory.Program evaluators must also understand and articulate the larger context in which the program operates, which both modifies and is modified by the program. As an example, O’Hara et al describe that successful implementation depended on an institutional culture that values longitudinal relationships, learner-centered education, and psychological safety, as well as strong faculty buy-in and commitment to teaching. Without these cultural and relational foundations, even well-designed structural innovations may struggle to achieve their intended impact. Evaluators can explicitly state and incorporate these contexts into their CMOs. As the TEAM program evolves, the processes of observation and feedback may themselves reshape the educational culture in positive or negative ways, influencing team dynamics and reinforcing or undermining the program’s intended effects. Such continuous reflection allows evaluators to move beyond static outcome assessment and instead toward a dynamic understanding of program impact within complex educational systems.O’Hara et al thoughtfully leveraged a logic model to guide the design and evaluation of the TEAM program, representing a meaningful advancement beyond the Kirkpatrick model that is commonly seen in health professions education. Their work provides a strong foundation for theory-informed program development and assessment. We encourage program evaluators to build on this momentum by further articulating program theory, centering mechanisms and context, and incorporating iterative inquiry to deepen explanatory insight. Such approaches shift program evaluation from outcome accounting toward explanation, enabling more thoughtful adaptation and transfer of programs across diverse contexts.