Answers are grounded in this paper and cite their sources.
Background As competency-based education has expanded among health professionals, greater emphasis has been placed on teaching and assessing clinical skills as a domain distinct from knowledge. Given that clinical skills assessments involve multiple facets, establishing their validity and reliability is crucial. This study evaluated the validity and reliability of a summative clinical skill assessment (S-CSA) at a single college over three consecutive years (2023–2025). Methods The S-CSA consists of clinical performance examinations (CPX) using patient-encounter cases and objective structured clinical examinations (OSCE) using clinical skill cases. Data from three years were analyzed, with 47, 46, and 56 examinees in 2023, 2024, and 2025, respectively. The analyses included descriptive statistics, correlations among CPX domain scores, CPX total scores, and OSCE total scores, and estimation of generalizability coefficients using generalizability theory (G-theory). Results Descriptive analyses showed a consistent tendency for CPX scores to be lower than OSCE scores. In particular, the history-taking and physical examination domains tended to show lower achievement levels across the years. Correlations among CPX domains were modest to moderate, whereas correlations between OSCE and CPX total scores remained low, suggesting that the CPX and OSCE assessed distinct constructs. The G-coefficients ranged from 0.534 to 0.850. Reliability estimates varied across cohorts but became more stable over time, with fewer poorly functioning stations identified in later years. Sensitivity analyses demonstrated that the exclusion of underperforming stations substantially improved reliability estimates in earlier cohorts. Conclusions Over three years, repeated analyses of the reliability of the clinical skills assessment showed that the examination ultimately reached a level generally considered acceptable for high-stakes assessment. This improvement may have been supported by the identification of stations that undermined the overall reliability and by the increasing number of CPX stations emphasizing clinical reasoning. Continued case development and further refinement are required. Future studies should include item-level reviews and analyses with larger sample sizes.
No takes yet. Share an insight, caveat, or question.
Lee et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: