The SICNESS tool demonstrated strong inter-rater reliability for identifying non-technical errors in surgical mortality cases (κ 0.89; 95% CI 0.84-0.93).
Observational (n=844)
Yes
The SICNESS tool demonstrates strong to near-perfect inter-rater reliability for retrospectively identifying and categorizing non-technical errors in surgical mortality cases.
Effect estimate: κ 0.89 (95% CI 0.84-0.93)
BACKGROUND: Up to half of all surgical adverse events are due to non-technical errors, making non-technical skill assessment and improvement a priority. No specific tools are available to retrospectively identify non-technical errors that have occurred in surgical patient care. This original study aimed to develop and provide evidence of validity and inter-rater reliability for the System for Identification and Categorization of Non-technical Error in Surgical Settings (SICNESS). METHODS: A literature review, modified Delphi process, and two pilot phases were used to develop and test the SICNESS tool. For each pilot, 12 months of surgical mortality data from the Australian and New Zealand Audit of Surgical Mortality were assessed by two independent reviewers using the SICNESS tool. Main outcomes included tool validation through modified Delphi consensus, and inter-rater reliability for: non-technical error identification and non-technical error categorization using Cohen's κ coefficient, and overall agreement using Fleiss' κ coefficient. RESULTS: Version 1 of the SICNESS was used for pilot 1, including 412 mortality cases, and identified and categorized non-technical errors with strong-moderate inter-rater reliability. Non-technical error exemplars were created and validated through Delphi consensus, and a novel mental model was developed. Pilot 2 included an additional 432 mortality cases. Inter-rater reliability was near perfect for leadership (κ 0.92, 95% c.i. 0.82 to 1.00); strong for non-technical error identification (κ 0.89, 0.84 to 0.93), communication and teamwork (κ 0.89, 0.79 to 0.99), and decision-making (κ 0.85, 0.79 to 0.92); and moderate for situational awareness (κ 0.79, 0.71 to 0.87) and overall agreement (κ 0.69, 0.66 to 0.73). CONCLUSION: The SICNESS is a reliable and valid tool, enabling retrospective identification and categorization of non-technical errors associated with death, occurring in real surgical patient interactions.
Ey et al. (Tue,) conducted a observational in Surgical mortality (n=844). System for Identification and Categorization of Non-technical Error in Surgical Settings (SICNESS) tool was evaluated on Inter-rater reliability for non-technical error identification (κ 0.89, 95% CI 0.84-0.93). The SICNESS tool demonstrated strong inter-rater reliability for identifying non-technical errors in surgical mortality cases (κ 0.89; 95% CI 0.84-0.93).