Innovating OSCEs: a multiphase study evaluating the opportunities and challenges of digitally enabled clinical assessments

Background and aims The COVID-19 pandemic prompted an Australian osteopathy program to develop a hybrid face-to-face and digitally enabled objective structured clinical examination (OSCE) using videoconferencing for remote assessment. This study aimed to explore the opportunities and challenges associated with the hybrid OSCE.Methods and participants This pragmatic multiphase study evaluated reliability of asynchronous grading and explored participant perceptions using a survey. Ninety-one osteopathy learners undertook a summative assessment where synchronous grading (T1) occurred. Asynchronous grading (T2) was conducted 5 months later, total OSCE scores and individual criteria scores were compared. Analysis of correlation, agreement and test-retest reliability was performed to determine reliability. Survey data from 17 learners and three examiners were gathered using a modified Student Perceptions of e-Assessment Questionnaire (SPEAQ).Results Moderate agreement (κw = 0.45, p < .001) and strong correlation (r = 0.69, p < .001) between total scores was achieved. Statistically significant correlation was achieved on only five of the 16 station criteria (ρ = 0.41–0.44, p < .001) and both agreement and test-retest reliability were poor. This indicates total OSCE scores showed a consistent pattern between T1 and T2, although this consistency did not extend reliably to individual scoring criteria. Survey mean scores ranged from 3.75 to 4.84 (SD = 0.36–1.29) with the highest mean ratings being for affective, practicality and pedagogy domains.Conclusions Results indicate asynchronous video-based grading may be feasible, but reliability was not consistently demonstrated within the constraints of this study. Positive perceptions were reported by learners and examiners indicating the digitally enabled OSCE was acceptable and offered pedagogical benefits that were unique to the hybrid format.