Is Technical Expertise Necessary to Assess Technical Expertise? Let's Ask the Kids

Background: Simulation has an increasing role in surgical training; however, using validated tools such as the Objective Structured Assessment of Technical Skills (OSATS) is time-consuming, which may be a potential barrier to simulation-based training. This study tests the hypothesis that assessors with technical expertise are necessary to objectively score a technical task. Methods: Three tasks, ring transfer, needle pass, and atresia cut, were performed using a synthetic thoracoscopic simulator for esophageal atresia/tracheoesophageal fistula. Three pediatric surgeons, 6 novice adults, and 3 children aged 9–13 years scored each attempt using the overall global OSATS rating from 1 to 5 (1 “repeated or awkward movements” to 5 “fluid movement. No awkwardness”). Results: For the ring transfer, all assessors scored novice participants significantly less than expert and intermediate scores (surgeon P = .0004, nonsurgeon adults P = .0009 and children P = .0003). For the needle pass, all assessors gave significantly different scores between novices and experts (surgeon P = .0007, nonsurgeon adult P = .0008, and children P = .0040). For the atresia cut, surgeon assessors gave significantly higher scores for experts and intermediate and novice ( P = .0004). Nonsurgeon assessors gave experts and intermediates significantly higher scores than novices ( P = .0001). Surgeon assessors achieved good reliability for ring transfer (0.8252) and needle pass (0.7769) compared with nonsurgeon assessors who showed poor reliability for the ring transfer (0.3959) and moderate for the needle pass (0.6551). Conclusions: Expertise in performing these procedures is not a prerequisite for an assessor to evaluate the technical skill, hence assessors of skill acquisition can be nonexpert, a nonsurgeon, or even a child. The variability in all groups suggests that reliability overall is increased with multiple assessors. Although nonsurgeon assessors may be appropriate for formative assessments, they lack the reliability to provide assessment of competence for high stakes complex tasks. Summative assessment will likely require at least 1 surgeon/expert assessor to provide reliability.