Assessing the assessment of interventions: we're not there yet

In any attempt to improve education, there is always a trade-off between effort and potential gain. Sometimes the problem is clear, the solution obvious and the expected results unequivocal. At other times, the drive for change is an urge to improve current affairs and make them even better than they already are. In all cases, instigating an educational intervention involves making a leap into the dark. You never know exactly what the effects will be. In their paper on cost analysis in this issue of Medical Education, Walsh et al.1 point out the importance of assessing whether money spent on medical education represents value relative to cost. They address the need for clarity in the nomenclature of cost analysis by defining and illustrating methods and terms that might be used. Moreover, they indicate several other challenges in the evaluation of educational interventions; it is hard, for example, to define what constitutes effectiveness, let alone find measures for effects and prove causality of interventions within the complex systems of medical education. Walsh et al.'s publication1 provides an opportunity to reflect on what we aim to achieve by our interventions in medical education, what their actual effects are and how we could or should decide upon their usefulness. In any attempt to improve education, there is always a trade-off between effort and potential gain Change in medical education is often impelled by a problem that needs a solution. Thus, the objective of change is self-evident: we wish to solve the problem. This ‘problem’ may take any of many sorts of forms and shapes: it may be very practical and involve a shortage in training rooms, or it may be more theoretical and refer to, for instance, a lack of integration of different types of knowledge. From an evaluative standpoint, knowing what the objective is makes it much easier to establish the outcomes on which the success of an intervention should be assessed.2 Nowadays, given the abundance of innovations and smart e-solutions available, a novelty per se can seem attractive.3 However, without a defined problem, it is difficult to decide on an objective, and therefore even harder to plan an evaluation to define the intervention's utility. The motive for change should be to seek a fitting solution to a particular problem. Only then is evaluation possible. Knowing what the objective is makes it easier to establish the outcomes on which an intervention should be assessed The interventions to which we subject medical education are meant to improve it. Ideally, we would evaluate all of these interventions to confirm that they contribute to that goal. For evaluative purposes, however, it is probably more realistic to differentiate between two types of change. On the one hand, there are large-scale interventions that can be evaluated on their own merits. Examples of these include the introduction of a new assessment programme or simulation-based education. For instance, Ma et al.4 evaluated the use of simulation-based education in central venous catheterisation on the measures of learner and patient outcomes. They found benefits for learners' knowledge and performance on simulators and for some clinical outcomes, such as the number of needle passes. Then there are the many smaller adjustments to education that can be made, such as by decreasing the size of tutorial groups, introducing a different form with which to register feedback, or changing from a paper-based textbook to an online version. It is practically impossible to regard all of these smaller adjustments as interventions to be evaluated separately. Moreover, this would not acknowledge the fact that most of them are part of an ongoing cycle intended to sustain quality, which should be subject to monitoring in its entirety. In fields other than medical education, such as corporate business, monitoring and adjusting the quality cycle are considered to be part of regular organisational management.5 However, in the field of health care, this kind of assessment appears to be quite challenging as a result of factors such as social complexity and lack of expertise and training in this type of evaluation.6 People involved with medical educational interventions might lack awareness of the processes of which they are part.7 As long as this is the case, it will impede their contribution to quality improvement. In fields other than medical education, monitoring the quality cycle is considered to be part of regular organisational management The evaluation of effects is yet another challenge. As Walsh et al.1 put it: ‘Medical education interventions typically result in many different effects at the same time.’ With respect to interventions in our field, we are intuitively concerned with the extent of intended effects in the educational domain. However, an innovation in a social system may be expected to lead to both desirable and undesirable effects.8 Although this is usually overlooked in evaluation reports about medical education, it is evident that changes in more complex social systems, such as hospital departments, cannot easily be anticipated and are bound to lead to unintended effects. Research on the effects of workplace-based assessment (WBA) as a novel method in residency training illustrates that not only teaching and learning are affected. For instance, WBA may influence workload, which may be felt to be increased by the imposition of the extra task, but which may also be decreased in trainees who are relieved by the insight into their performance offered by the method. Other effects include an increase in attention to training in general, which may lead to comparisons among and discussion of various departments' handling of training.9 It is the interaction of these various effects that determines an intervention's actual impact on practice. Assessments of medical education innovations have been mainly directed at their intended effects, or at both intended and unintended effects within the educational domain.10, 11 This has created important knowledge about what type and amount of educational effects might be expected. However, evaluation confined to this area will not help us to build a complete picture of the full impact of an innovation. Specific knowledge about any other effects that might be expected of an educational intervention is therefore necessary, but still in its infancy.9 Such knowledge is indispensable if we are to determine the end to which investments have been made. Changes in complex social systems, like hospital departments, cannot easily be anticipated and are bound to lead to unintended effects In conclusion, educational change often comprises complex social interventions that result in various effects that will probably be both intended and unintended, desirable and undesirable. There are many sorts of effect, such as those on the learning process of individuals, the entire training system, willingness to keep improving in the future and the efficiency with which people and money are employed. We argue that those who plan an intervention should be clear not only about their aims, but also about how they will monitor the effects of their intervention. The decision on whether or not something is worthwhile will refer to a context-bound assessment of which aspects are more important than others, but should be based on clear insights into what the intervention brought about. We join Walsh et al.1 in striving for clarity in the discussion of this topic and emphasise the need to establish common concepts and accurate terminology on interventions, and their costs and effects.