A pilot study of the use of artificial intelligence with high‐fidelity simulations in assessing endovascular procedural competence independent of a human examiner

BACKGROUND: With increased need for vascular surgery trainees to gain endovascular surgery proficiency, current models of case-numbers and subjective visual assessment are inadequate in capturing the skills required in endovascular surgery. We explored the use of high-fidelity simulators in (1) assessing endovascular surgical competence; (2) clinical decision making; and (3) the reliability of an artificial intelligence (AI) assessor. METHODS: Registrars, fellows and consultants from vascular surgery, interventional radiology and general surgery performed identical procedures on a high-fidelity simulator. Performance was independently assessed using a modified Reznick scale. Scores were compared to raw metric data extracted from the simulator, objective scores extracted from the recordings and analysed by AI. RESULTS: 22 participants were enrolled from vascular surgery (n = 6, 27.3%), interventional radiology (n = 10, 45.5%) and general surgery (n = 6, 27.3%). There were 12 trainees, 2 fellows and 8 consultants. Significant correlations between raw metric data and all categories of the modified Reznick scale except 'respect for tissue' were found. An AI demonstrated positive reliability in all categories, with some predictions being moderately correlated. CONCLUSION: The use of high-fidelity simulators to assess endovascular surgical competence has comparable correlations to the traditional assessment methods with global rating scales, which can be used in formative assessment. AI demonstrates an ability to support assessment but requires further research.