Vision-based recognition of surgical knot states in a virtual reality laparoscopic simulator: a five-class multi-architecture deep learning approach

Surgical knot tying is a critical psychomotor skill in laparoscopic suturing whose objective, automated assessment within virtual reality (VR) training environments remains largely unsolved. Existing methods rely on simulator-internal geometric representations that are tightly coupled to specific physical solvers and do not necessarily reflect the visual criteria applied by expert surgeons. This study presents a rigorously validated, multi-architecture deep learning framework for real-time visual recognition of five clinically meaningful knot states in a VR simulator, explicitly incorporating knot-quality discrimination and a transitional slip-knot category. A dataset of 11,940 RGB images across five classes ( no_knot , first_loop , square_knot , granny_knot , slip_knot ) was generated by 10 board-certified laparoscopic surgeons across 50 independent VR sessions (5 sessions per surgeon), each covering the complete knot-tying sequence from no-knot through the double-wrapped first throw and the square-knot, granny-knot, and slip-knot states. A strict session-level partition (70/15/15) was enforced to prevent temporal data leakage. Inter-rater agreement between the two annotating surgeons reached Cohen’s \(κ = 0.9958\) (almost perfect). Four convolutional architectures—EfficientNet-B0, ResNet-50, MobileNetV3-Large, and ConvNeXt-Tiny—were trained with a two-stage fine-tuning strategy and evaluated with per-class and macro-averaged accuracy, precision, recall, F1-score, and AUC. Gradient-weighted class activation maps (Grad-CAM) were computed for interpretability analysis. The best model was exported to ONNX (opset 11) for Unity/Barracuda deployment, with numerical equivalence formally verified and inference latency profiled on standardised CPU hardware. EfficientNet-B0 achieved the best accuracy-efficiency trade-off: overall accuracy \(=0.9967\) , macro-F1 \(=0.9965\) , macro-AUC \(=1.0000\) , and Cohen’s \(κ =0.9958\) (almost perfect model-expert agreement). Mean CPU inference latency was \(34.24 ± 6.65\) ms on a desktop workstation; operational deployment on the Meta Quest 3 headset sustains an average of 71 FPS via the dedicated NPU and asynchronous Barracuda scheduling pipeline, remaining effectively unchanged relative to the no-inference baseline (71.46 FPS). Grad-CAM analysis confirmed that activations concentrate on thread crossings and loop structures relevant to each knot category. ONNX conversion introduced negligible numerical deviation ( \(<10⁻⁵\) ). Treating knot assessment as a visual perception problem decoupled from simulator-internal physics yields robust, interpretable, and deployable models across a clinically grounded five-class taxonomy. A traditional HOG+SVM baseline achieves macro-F1 \(=0.8513\) ( \(κ =0.8155\) ), quantifying the advantage of deep representation learning ( \(Δ \) macro-F1 \(=+14.5\) pp). Sliding-window temporal smoothing ( \(W=5\) frames) raises macro-F1 to 0.9994 without retraining, and \(W=7\) achieves perfect accuracy on the held-out test set. The framework demonstrates the complete integration of a deep learning-based visual inference pipeline within a portable standalone VR simulator, enabling automatic recognition of clinically meaningful surgical knot configurations without affecting the rendering cycle performance.