This study provides a toolbox of 23 assessment tools for surgical skills but highlights that few utilize contemporary unitary concepts of validity, indicating a need for updated validation frameworks in surgical education.
In Brief Objective: The purpose of this study was to create a technical skills assessment toolbox for 35 basic and advanced skills/procedures that comprise the American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) surgical skills curriculum and to provide a critical appraisal of the included tools, using contemporary framework of validity. Background: Competency-based training has become the predominant model in surgical education and assessment of performance is an essential component. Assessment methods must produce valid results to accurately determine the level of competency. Methods: A search was performed, using PubMed and Google Scholar, to identify tools that have been developed for assessment of the targeted technical skills. Results: A total of 23 assessment tools for the 35 ACS/APDS skills modules were identified. Some tools, such as Operative Performance Rating System (OSATS) and Objective Structured Assessment of Technical Skill (OPRS), have been tested for more than 1 procedure. Therefore, 30 modules had at least 1 assessment tool, with some common surgical procedures being addressed by several tools. Five modules had none. Only 3 studies used Messick's framework to design their validity studies. The remaining studies used an outdated framework on the basis of "types of validity." When analyzed using the contemporary framework, few of these studies demonstrated validity for content, internal structure, and relationship to other variables. Conclusions: This study provides an assessment toolbox for common surgical skills/procedures. Our review shows that few authors have used the contemporary unitary concept of validity for development of their assessment tools. As we progress toward competency-based training, future studies should provide evidence for various sources of validity using the contemporary framework. We report a technical skills assessment toolbox on the basis of the American College of Surgeons/Association of Program Directors in Surgery surgical skills curriculum and provide a critical appraisal of assessment tools. This study provides an overview of unitary framework of validity and an in-depth discussion about misunderstandings about validity and underutilization of this framework in surgical literature.
Ghaderi et al. (Tue,) studied this question.