Longitudinal interrelationships of OSCE station level analyses, quality improvement and overall reliability

Objective Structured Clinical Examinations (OSCEs) are a key component within many healthcare assessment programmes. Quality assurance is designed to ensure rigour and credibility in decision making for both candidates and institutions, and most commonly expressed by a single measure of reliability. How overall reliability interrelates with OSCE station level analyses is less well established, especially with respect to the impact of quality improvements. This work examined longitudinal interrelationships of reliability and station level metrics alongside interventions to improve the OSCE, revealing an anticipated relationship between poor reliability and poor station level analyses. However, longitudinal profiling revealed that overall reliability proved relatively unresponsive to continued improvements across stations - highlighting the importance of station level analyses as a routine part of any assessment quality assurance.