A Comparative Analysis of Objective Structured Clinical Examination (OSCE) Observed Scores and Global Rating Scores using a Novel Approach

Background: Objective Structured Clinical Examinations (OSCEs) are a cornerstone of medical education. Despite their widespread use, the relationship between observed scores and global rating scores in OSCEs remains a topic of debate. This study aimed to identify potential scoring discrepancies between the observed scores and global rating scores of OSCEs. Method: This retrospective observational study analyzed anonymized OSCE data from 1,571 undergraduate medical students in the 4 MB program at a single medical school over nine years. The data collected from randomly selected OSCE sessions included observed scores calculated as percentages for individual stations and global rating scores (GRS) assigned holistically at the station level. A key change made midway through the study refined the GRS, splitting the “Borderline” category into “Borderline Pass” and “Borderline Fail.” The Data were analyzed using raincloud plots, ordinal regression modelling, and tree-based approaches to identify and visualize discrepancies between the two assessment measures. Results: The analysis identified discrepancies between observed scores and global rating scores, revealing that a single observed score often corresponded to multiple global rating categories. Ordinal regression and tree-based models highlighted substantial variability, particularly within mid-range categories (GRS bands 2, 3, and 4), making evaluations of these ranges more subjective and uncertain. The conditional inference tree further illustrated that the mid-range observed scores lacked clear alignment with specific global rating categories, underscoring the inconsistency and variability in examiner assessments. Discussion: This study emphasizes the need for consistent and comprehensive assessment tools. The findings align with previous research, highlighting the challenges in aligning the observed scores and global rating scores in OSCEs. The identified discrepancies emphasize the necessity of adopting a feedback system that integrates both qualitative and quantitative aspects. Conclusion: This research highlights the importance of structured feedback in bridging the gaps between the two scoring methodologies and in enhancing student learning, professional development, and faculty advancement.