Introduction: The demand for pediatric sedation procedures is growing with the services now often provided by PICU staff. Assessing the most significant complications, airway related, in a structured manner can help evaluate the safety of the provided sedation. To support quality assurance efforts, our team has developed an airway scoring system assigning numeric values based on intervention complexity. The score ranges from 1 (most severe) to 10 (most favorable) with an overall score based on the most advanced intervention used. However, due to exclusion of high-risk patients from sedation, our clinical cases lack advanced airway events. This limits testing of the full scoring spectrum. The aim of our study is to assess the quality of our airway scoring system by testing it in both our sedation cases but also by introducing sim cases with advanced airway interventions such that the full spectrum of airway interventions are accounted for in our datasets. Methods: Ten SIM video cases illustrating advanced airway interventions were created and scored by 8 dental residents using our airway scoring system. Each case included events categorized from D to H, with H indicating highest severity. Additionally, we analyzed 150 previously collected real-life sedation cases. Statistical methods evaluated: (a) inter-rater reliability for overall and component scores, (b) resident vs. computer score agreement, (c) component-level inter-rater correlations, (d) correlation of the 7 components with overall score, and (e) the effect of SIM case inclusion. Results: In the 150 real-life cases, airway scores ranged from 3–10, lacking representation of the most severe (scores 1–2) events. Inclusion of SIM cases enabled coverage of the full scoring spectrum. Cohen’s Kappa was calculated for all seven airway components only after SIM data were added. These analyses confirmed high inter-rater reliability and completion of missing score categories. Conclusions: Prior to SIM case development, our scoring system could not be fully validated due to absence of severe events. After SIM case inclusion, all scoring components were represented, and Cohen’s Kappa confirmed strong inter-rater agreement. The airway scoring system now demonstrates validity across its full range and is appropriate for clinical use and quality assurance tracking.
Piryani et al. (Sun,) studied this question.