A machine learning model combining automated simulator metrics and biometric stress data predicted proficiency in simulated endovascular aortic repair with an accuracy of 80% and an AUC of 0.84.
Cohort (n=24)
Yes
Does a machine learning model combining simulator and biometric data improve automated proficiency assessment in simulated EVAR procedures compared to standard cut-off methods?
Machine learning models combining simulator-generated metrics and biometric stress data can accurately predict proficiency in simulated EVAR procedures, potentially reducing the need for expert instructor assessment.
Absolute Event Rate: 0.84% vs 0.65%
Endovascular skills can be learned through simulation-based education (SBE), and objective skills assessment is essential to ensure trainees' proficiency before proceeding to clinical procedures. Assessment requires trained instructors, which is a barrier to broader SBE implementation. This study aimed to evaluate if proficiency assessments in endovascular aortic repair (EVAR) can be automated using simulator-generated and biometric data combined with machine-learning. International EVAR novices followed a standardised SBE EVAR programme. Participants were assessed four times by experienced instructors using a validated assessment tool. Simulator performance metrics and biometric stress data were extracted from these cases to build prediction models based on contrasting groups' cut-off values and machine learning principles. During the program, 96 EVAR-cases were performed by 24 participants. Fifty (52%) cases were assessed as failed, and 46 (48%) as passed. The machine learning model outperformed the contrasting groups model with an accuracy of 80% compared to 65%. The area under the curve (AUC) of the ROC-curve for the machine learning model was 0.84 (95%CI: 0.75-0.94) compared to an AUC of 0.56 (95%CI: 0.48-0.64) in the contrasting groups model. The most predictive variables were total procedure time, activated clotting time, trainees' stress level, time to complete graft deployment, and heparin administration. Machine learning models based on simulator and biometric data can help determine proficiency in simulated EVAR procedures. This is a step toward minimizing the need for an instructor in basic EVAR SBE, but better simulator-generated metrics are needed to improve automated feedback.
Skov et al. (Fri,) conducted a cohort in Endovascular aortic repair (EVAR) simulation training (n=24). Machine learning model combining simulator metrics and biometric stress data vs. Contrasting groups method using simulator metrics alone was evaluated on Prediction of EVARATE pass/fail proficiency status (Area Under the Curve) (95% CI 0.75-0.94). A machine learning model combining automated simulator metrics and biometric stress data predicted proficiency in simulated endovascular aortic repair with an accuracy of 80% and an AUC of 0.84.