Objective Structured Clinical Examination—Are They Truly Objective?

To the Editor Recently, Anesthesia & Analgesia published the results and analysis of the first Objective Structured Clinical Examination (OSCE) component in the certification process of the American Board of Anesthesiology (ABA).1 As the authors mention, in 2018, the ABA became the first and only member of the 24 member boards of the American Board of Medical Specialties (ABMS, Chicago, IL) to introduce the OSCE as part of its applied examination. In part, this was to satisfy the 2015 Accreditation Council for Graduate Medical Education (ACGME) safety and quality improvement requirements for all residency programs with particular reference to professionalism and communication. However, the ACGME did not specify that the OSCE was the appropriate or the only approach to test these requirements. In fact, there was never a requirement that these skills need to be objectively evaluated by the ABMS as part of their certification process. To pass these skills, the ABA expects most candidates to perform these “communication tests” by abiding to definite criteria and “check” certain boxes that are detailed in their published OSCE scoring guidance. Yet, it is common knowledge that we all individually tailor our approach to patients and modify our body language, demeanor, tone, and degree and depth of explanation based on their anxiety levels, knowledge, and understanding. It takes many years, maybe decades, for us to perfect this art and not all of us can achieve the same level of competency. In fact, it is counterproductive to patient care if we approach all patients in a standardized fashion. The ABA states that it was encouraged by the use of OSCE as part of testing in the United Kingdom and the Israeli National Board Examination in Anesthesiology.2 However, it should be noted that in the United Kingdom, it is part of the primary Fellow of the Royal College of Anaesthetists (FRCA) examination (similar to ABA basic examination) and has no bearing on the certification that is issued by the General Medical Council (similar to various state boards) based on the training. In all 3 countries, there is no published evidence to show that the introduction of OSCE has improved patient outcomes and satisfaction or decreased disciplinary proceedings against the anesthesiologists. In fact, there is little evidence to support the notion that the certification itself has any effect on such measures. As an indicator of its certification cogency, ABA used the license disciplinary action as a pointer of written and oral specialty certification examination effectiveness.3 Although negligence or incompetence are the most common causes (yet, only 34% of the total in this study), factors such as alcohol and substance abuse, inappropriate prescribing practices, inappropriate contact with patients, and fraud are responsible for the majority of such actions and these cannot be foreseen in the OSCE or Structured Oral Examination (SOE).4 The authors rely heavily on the Rasch model that essentially analyzes categorical data (such as the grading system used for these examinations: “consistently,” “often,” “occasionally,” or “rarely”) and compares the results from such a grading system as a function of examination difficulty. However, this is confusing, since they mention how those who failed the OSCE portion of the examination did best on what they considered the easiest portions of the examination (they mention how failed candidates had the lowest score in Ethical Issues and Application of Ultrasonography, both of which are ranked very low in the difficulty figure; Table 2). Furthermore, the Rasch model construction itself is misleading, because the authors do not explain how the pass/fail cutoff was determined for the OSCE examination. Given that the authors found a very weak Pearson correlation between the 2, I wonder why the passing standard was equal between the 2 exams (as seen in Figure 1). The authors seem to rely heavily on correlation across tests to convey how the OSCE portion will test unique skills overlooked by the previous examination method. Nevertheless, the following seems to a bold assertion, “When candidate score pairs are graphically depicted for any 2 examinations (Figures 2 and 3), the observed variations from a linear relationship suggest that the examinations measure different abilities, that there is measurement error associated with each examination (eg, individual scores do not perfectly reflect the ability of a given candidate), or a combination of both factors.” First, the relationship in Figures 2 and 3 is more of a linear one; even if they do deviate from a linear relationship slightly, this does not necessarily mean the tests measure different abilities. It could just be that the OSCE examination is easier than the SOE, since the figures show how more people failed the SOE portion. Perhaps if the authors included a difficulty chart for the attributes tested on the other portions (the basic, advanced, and SOE), one could really quantify how much the relationship deviates from a linear model. Some of deviation is explained by the Hawthorne Effect that occurs when individuals adjust their behavior as a result of being watched or observed.5 The accompanying editorial briefly alludes to this.6 However, the OSCE, as administered by the ABA, seems to be very subjective and opens doors for discrimination based on factors other than performance alone. To increase the confidence in their examination system, it is important for the ABA to disclose any evidence they have to the contrary. I am certain that the ABA has considered all these factors before the introduction of the OSCE. In a country that is increasingly polarized on issues such as race, gender, skin color, political affiliation, nationality, accent, and presumed immigration status, it is paramount for the ABA to look at such data that should be readily available in the context of SOE and OSCE and publish it. In the context of OSCE, it should be noted that, recently, the Federation of State Medical Boards and National Board of Medical Examiners (NBME), cosponsors of the United States Medical Licensing Examination, announced the discontinuation of work to relaunch a modified step 2 clinical skills examination (an OSCE equivalent).7 Basavana Goudra, MD, FRCA, FCARCSIDepartment of Anesthesiology and Critical Care MedicineFDA member for Anesthetic and Analgesic Drug Products Advisory CommitteePerelman School of MedicineHospital of the University of PennsylvaniaPhiladelphia, Pennsylvania[email protected]