Ensuring flexible optical intubation skills using a portable virtual reality simulator – a validation study

Anesthesiologists must be able to perform flexible optical intubation (FOI) in challenging cases, but the skill is difficult to master and cannot be adequately taught using the traditional apprenticeship model. Mastery learning simulation-based training ensures that all trainees acquire basic competence in a safe environment before progressing to supervised performance on patients. However, mastery learning training programs require a simulation-based test with strong validity evidence and a defensible pass/fail standard. We developed a FOI test on a portable virtual reality (VR) flexible bronchoscopy simulator, consisting of 12 challenging cases, and administered the test to 15 novices and 15 experienced anesthesiologists. Performance metrics were automatically recorded by the simulator, and all five sources in Messick’s contemporary validity evidence framework were explored. Internal consistency reliability was Cronbach’s α = 0.89. Generalizability theory analyses found that six cases were required for formative assessment (reliability > 0.7) and 10 cases were required for summative assessment (reliability > 0.8). An independent samples t-test showed that the test could significantly discriminate between novices and experienced anesthesiologists, p < 0.001. A credible pass/fail standard of 34.4% was established using the contrasting groups’ standard setting method. This standard resulted in two false positives (i.e., novices that passed the test) and one false negative (i.e., experienced that failed the test). The simulation-based FOI test based on 12 challenging adult airway cases on a VR portable flexible bronchoscopy simulator demonstrated strong validity evidence. The test can be used for both formative (i.e., feedback) and summative (i.e., certification) assessment and thereby support a mastery learning simulation-based training program for FOI.