Assessing Content Validity and Inter- And Intraobserver Reliability of the Paediatric Surgical Adverse Events Severity System (Pedsaves)

Retrospective adverse event (AE) collection through administrative codes and systems such as the National Surgical Quality Improvement Program (NSQIP) have been shown to be incomplete (1). Prospective AE collection tools, such as the Spine AdVerse Events Severity System (SAVES) and OrthoSAVES improve AE capture (2–4). Thus, a novel adaptation of these systems was developed for the Paediatric Orthopaedic population, named the Paediatric Surgical AdVerse Events Severity System (PedSAVES). This study aims to evaluate the content validity and assess the inter- and intra-observer reliability of PedSAVES. To assess the inter-observer reliability, surgeons, fellows, residents, and research members used PedSAVES to grade AEs deemed present in a survey containing 20 Paediatric Orthopaedic case vignettes. One month later, the order of the vignettes was shuffled and the survey was completed again to test the intra-observer reliability. Using a RedCAP survey that replicates the experience of using PedSAVES, adverse events are recorded prospectively from chart reviews. The AE collection using PedSAVES will be compared with data from traditional AE data collection (NSQIP, PSLS, etc) to assess the content validity of PedSAVES. Reliability testing revealed substantial agreement in diagnostic and severity assessments. Gwet's AC1 values of 0.65 to 0.79 demonstrated strong inter-observer reliability in diagnostics. The consistently high ICC values (0.87 to 0.96) for average raters indicate excellent reliability in severity scoring. The Orthopaedic attending surgeons showed slightly lower reliability. Validity testing is ongoing therefore, we have no results to present at this time. We anticipate it will be completed, including statistical analysis, in time for COA. PedSAVES has demonstrated to be a reliable system. Validity testing of PedSAVES is ongoing, but we predict the system is effective, based on the assessment of SAVES (3). Therefore, implementation of PedSAVES is likely to improve AE documentation at BCCH which may help decrease complications, improving our quality of care.