Answers are grounded in this paper and cite their sources.
It is interesting to see how some developments in assessment in the USA and UK seem to have gone in opposite directions. The USA has a longstanding tradition of high-quality national licensing examinations, whereas, in the UK, each medical school has traditionally been responsible for its own examination programmes. Although UK medical schools use external examiners, the country has no national licensing examinations. Strikingly, in the USA, methods to ensure the quality of medical students and graduates other than those of standardised testing are developed. By contrast, the idea of introducing a national licensing system is currently under investigation in the UK. Apparently it is feared that differences between medical schools in assessment in general and in pass/fail scores in particular may produce graduates of different levels of quality. A recent paper by Kathy Boursicot et al. demonstrates that such differences do exist and discusses its results in terms of the support they lend to the introduction of a national licensing examination.1 The debate between the systems − a fully centralised national licensing system versus a decentralised in-school assessment system − needs to incorporate a myriad of arguments rather than only comparability between schools. I would like to highlight here some of those I consider most important. At present, the notion of competencies is highly popular; many curricula have redefined their outcomes in the form of competencies. I think that an element of continuous growth is inherent to the notion of competencies: it is strongly linked to continuous learning. Yet most licensing approaches apply a ‘single-shot’ technique with 1, 2 or 3 tests, each of which has to be passed. A candidate who passes such tests is considered to have reached an adequate standard in terms of the subject matter tested and will not be tested again. In terms of competences, this constitutes a misalignment between assessment and education. By contrast, there are a substantial number of elements involved in becoming a medical expert that can and probably should be tested definitively at certain time-points. For example, taking a patient's blood pressure, knowing the anatomy of the liver and being able to calculate a t-test are elements that should be mastered at particular time-points and in which further improvement is not central to development as a medical professional. These aspects can be tested in a single-shot approach with licensing examinations. However, generalisability is problematic in single-shot approaches. No matter how carefully designed the single-shot assessment is, it always represents a limited sample in time. It is very difficult to say how the relative performances of students from different medical schools would have been if the test had been administered on a different occasion. Progress test results in the Netherlands, where 4 medical schools jointly produce and administer progress tests, show large variations between the relative positions of each medical school depending on the test moment, the year groups compared and the cohorts of students.2 This illustrates the caution with which single-shot comparisons should be interpreted, both on the student and the school level. I should note, however, that consistent differences on progress test results from 1 school have had a considerable impact on improvements in its curriculum, illustrating how powerful a national test can be in this respect. By necessity, all national licensing assessment approaches I know of are highly structured. They mainly use multiple-choice questions, extended-matching questions and objective structured clinical examinations (OSCEs). Moreover, they are rather inflexible in that the same test is given to all candidates. This is much along the lines of the way medicine treats screening, as a highly structured, objective and patient-independent assessment with the sole purpose of discriminating between people (those who are sick and those who are healthy; those at high risk versus those at low risk). In-school examinations, however, need to provide additional information. Firstly, it is important to determine which educational intervention is needed to improve optimally a student's competence (we might call this a ‘therapeutic’ decision). The second decision involves which type of additional information or assessment is needed for an individual student to determine his or her progression to a next phase (a sort of ‘diagnostic’ decision). In an ideal situation, the results on national tests would represent a starting point at which a school could evaluate a student's performance on a more in-depth basis. I use these terms here because they describe so succinctly what is meant, but I think the terms ‘low’ and ‘high’ are inadequate. I disagree with their implication that one is more important than the other. If, for example, you want to learn how to do multiplication by hand, memorising multiplication tables might be of ‘low’ level and understanding the simple algorithm of long multiplication might be of ‘high’ level importance. This does not make sense, as both are necessary requirements for successful multiplication. Moreover, each requires a different educational approach (can you picture a PBL approach to multiplication tables?) and a different assessment approach. National tests typically focus on ‘low’ level skills, whereas currently there is growing interest in higher order skills and competences, such as ‘professionalism’, ‘scholarship’, the ability to think as a ‘scientist’, etc. Regardless of how well designed it is, any highly structured examination (even a national OSCE) is of limited value in this respect. A system with national licensing requires a well equipped licensing body. This is often an organisation in which many leading experts in the field are employed. The impact of such groups of experts on research and development in assessment is huge. Consider the enormous contributions of, for example, the National Board of Medical Examiners or the American Board of Internal Medicine to our knowledge and understanding of assessment. In a decentralised situation this expertise is, at best, dispersed through the country or not available at all. Many medical schools do not even employ medical educationalists. In my country, which has only 8 medical schools and an open, collaborative culture, strengths and weaknesses in this area are largely overcome by a vivid exchange in knowledge and experience, but in a country with many more medical schools this may be almost impossible. No matter what combination of systems is used, for me, the central issue concerns ensuring that any high-stakes decision, whether it be about an individual student or about a school, should be made only on the basis of extensive information. The essential discussion for me is then how to combine the best of both worlds. How do we ensure comparability of educational outcomes in the absence of national licensing? How do we translate the results of national licensing into improvements in the curriculum? Most importantly, how do we combine the results of both in quality management? These questions are high on my research agenda. To stimulate this discussion, I would like to suggest the notion that the results of licensing examinations are more useful for comparing curricula than for comparing the results of individual students, and that comparisons on grade point averages from in-school assessments only are useless in ensuring and controlling the quality of medical graduates in a country.
No takes yet. Share an insight, caveat, or question.
Lambert Schuwirth (2007) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: