Predicting assessment of intraoperative competence in general surgery residents: evaluating accuracy and fairness of statistical learning methods

Predicting individual intraoperative competence can support progress monitoring and timely remediation for surgeons in training. This study aimed to develop and evaluate predictive models of intraoperative competence using multiple classification algorithms. A retrospective analysis of SIMPL OR data from general surgery trainees between September 2015 and May 2024 was conducted. Eight models were trained and validated using 10-fold group cross-validation: logistic regression, linear discriminant analysis, random forest, k-nearest neighbors, gradient boosting machine, support vector machine, generalized linear mixed model, and generalized neural network mixed model. Model performance was evaluated using standard metrics emphasizing both accuracy and fairness across gender. The dataset included 123,894 evaluations from 3,793 trainees and 2,755 faculty. Postgraduate year was the most influential determinant of competence across models. The generalized neural network mixed model achieved the highest Area Under the Receiver Operating Characteristic Curve (0.787) and accuracy (0.722), while gradient boosting and k-nearest neighbors demonstrated strong and balanced performance. Fairness analyses showed that demographic parity values were consistently lower for female trainees, indicating that female trainees were classified as competent less frequently than male trainees. This pattern was observed across all modeling approaches. Machine learning models, particularly generalized neural network mixed models, show promise for predicting intraoperative competence and supporting equitable, data-informed entrustment decisions in surgical training. Logistic regression performed comparably in many cases and model selection should consider interpretability, feasibility, and intended use in addition to predictive performance.