Broadening the debate about quality in medical education research

As regular as the tide it would appear that the world of medical education research finds itself in an endless oscillation between the flood of promoting the empirically-grounded approach that has evolved over the past many decades and the ebb of criticising the strength of the findings that have accumulated. The most recent challenge I have seen put forward to the community occurred in the past 4 months at two international conferences in which the keynote speakers, both fantastic scholars whose opinions I respect a great deal, coaxed the field towards raising its standards after pointing out that both the rigour and relevance of research in our field is lacking. Both used Todres, et al.'s1 2007 paper to point out that a full two thirds of medical education research studies were observational in nature while less than three per cent utilised randomised controlled trial (RCT) designs. In a letter published in this issue Jones, et al.2 clarify that they did not intend to advocate for RCTs (or quantitative methods more generally) in that 2007 article, but rather, that their message was that whatever research design is used it should be used to the highest standards of practice for that design. This is an important clarification. One could easily be forgiven for inferring that claims of stagnation within medical education research accompanied by the observation that the majority of studies in the field are observational implies that observational studies (and the qualitative research encompassed in that broad label) are less desirable than other methods, as indicated by Dornan, et al.,3 in their reply. Many other scholars have weighed in on this debate.4-6 That conversation, in and of itself, is good for the state of medical education research even if it sometimes elicits feelings of déjà vu. This editorial is not intended to simply rehash that discussion, but rather, is aimed at prompting reflection on how we might move the discussion forward by changing the terms of the debate. As a member of the audience in both of the presentations mentioned above I found myself searching for novel ways to argue that the standards of the biomedical research community and that community's desire to place RCTs (and systematic reviews of RCTs) on a pedestal are inappropriate standards through which to judge the quality of research in medical education. It was then that I realized, however, that it is one thing to say that the biomedical community's hierarchy of evidence is insufficient in educational contexts, but quite another to offer an alternative model that provides guidance regarding how to judge whether or not the research being performed is enabling progress to be made. If we think our work in the field needs to be held to a different standard, the onus falls upon us to determine what that standard should be. A fundamental problem in this regard is diversity. A generally held value in the field is that many disciplines have insights to offer the educational community and that the variable ways of knowing that accompany each discipline, therefore, must be represented within our research efforts. One should not paint too Utopian a view in this regard as it is undoubtedly true, as Brian Hodges has so eloquently taught us, that different discourses (ways of seeing the world) can conflict with one another, either explicitly or implicitly, with potentially negative repercussions for both the field and individual students.7 On the whole, however, my experience has led me to believe that most people working within the field of medical education research recognize that the assumptions, methods, and worldviews that any of us hold as individuals are insufficient to address every issue of relevance to educators in the health professions. The challenge this creates in terms of defining universal standards for rigour are substantial. Even within the relatively narrow field of medicine the now omnipresent 'users' guides,' developed to assist clinicians and researchers in critically appraising the literature, count 25 in number, the last in the series effectively amounting to a users' guide on how to use the users' guides.8 How then can we put forward a set of criteria that will enable the next set of authors who choose to conduct a review of the quality of research in medical education to better represent the goals of the field and the progress that we think is being made? It is a critically important question because, quite frankly, it would not bother me in the least if the Todres, et al. study were repeated 10 years from now only to bemoan that the distribution of research methods was unchanged. I ask the question as a way of promoting discussion rather than because I claim to know the answer. Having said that, I would propose that the answer lies somewhere in the recognition that scientific progress has little to do with method. Isaac Asimov once said 'The most exciting phrase to hear in science, the one that heralds discoveries, is not "Eureka!" (I found it!) but "That's funny…"'. As researchers our goal should be to identify phenomena of interest (and practical relevance) and to let curiosity about those phenomena drive an exploration of existing empirical findings and theories as well as driving the generation of research methods tailored to the purpose of better understanding that particular phenomenon. Any one method is likely to be insufficient if the phenomenon is complex (as is typically the case if the phenomenon is related to education). In fact, Richard Boyd and other philosophers of science considered realists go so far as to argue that methods in science are produced by scientific theory just as much as theory is driven by method. The realists believe that every new observation brings us closer to understanding reality even while current beliefs are only ever considered approximations of reality.9 One need not get into esoteric debates about positivism versus constructivism or whether or not 'reality' exists to appreciate the value in this realist view. Throughout this editorial I have deliberately avoided using the word 'evidence' when discussing research in medical education. That is because the word has been hijacked to narrowly come to mean proof that something works rather than the broader definition of an available body of information.10 In complex domains like medical education research I think we are better served by focusing on the broader definition, striving for progress through the accumulation of empirical (i.e., based on verifiable observations) information that is relevant to our practical aims than we are by getting locked into placing different types of information into a static and inevitably narrowly defined hierarchy of evidence. The very best RCT may prove that a specific educational intervention was effective, but if it does not advance understanding of a more general phenomenon than that particular course or workshop then it will not be of substantial use to the broader community.11 Scholarship is about determining how to best adapt the empirical findings present in the literature to the needs and constraints of one's local context (and engaging in continuous quality improvement exercises to ensure that one's adaptations are meeting their mark). It is not about simply ordering the most effective educational therapy off the formulary. That pill doesn't exist. What this means in terms of judging the quality of research being performed in medical education is that any 'objective' checklist, such as those advocated by the users' guides, will be insufficient. Any simple quantification of the frequency with which certain methodological standards are met will be insufficient. Any simple tabulation of the types of research designs employed will be insufficient. Determining the value of any given study requires judgment guided by reference to the empirical findings and theories that came before it. Determining whether or not progress is being made in an area of study requires judging whether or not empirically unsupported ideas are being discarded, whether or not the conversations stimulated by the research efforts have changed, and whether or not the focus of our research efforts continue to evolve. The field of medical education research continues to grow in terms of the amount of research published, the number of conferences available, and the number of people who are carving out careers for themselves as medical educators and medical education researchers. This all provides indirect evidence that a large body of people are finding value in the research that is being done. That said, we do need to heed Todres, et al.'s fundamental message that to continue our progress we need to find better ways to sell what we do to various funding agencies and government organizations. In doing so, however, my sincere hope is that we will define the quality of medical education research based on judgments of whether or not progress is being made with regard to how sophisticated our understanding of the problems becomes, rather than on whether or not a particular research methodology has been adopted. Yes, we need to look for fatal flaws in research methodology that might invalidate the conclusions being drawn in any given study and, yes we want to ensure that research in the field maintains a focus on helping educators solve the practical dilemmas they face on a day-to-day basis. Still, the extent to which we define our efforts purely based on any particular methodological criteria is the extent to which we miss a great opportunity to truly stake claims to progression in our thinking and our educational innovations.