Overcoming color domain shift in endoscopic spine surgery: robust instrument segmentation via aggressive photometric augmentation

The generalizability of AI models for spinal endoscopy remains poorly understood despite inherent domain shifts. This study evaluates these impacts on instrument segmentation. A public dataset comprising 10,662 instrument segmentations was partitioned into internal and external cohorts. Inter-institutional color disparities were quantified within the CIELAB color space. Fifteen lightweight segmentation models were trained using two distinct protocols: a baseline protocol with spatial augmentation and a proposed protocol incorporating extensive photometric augmentation. Performance was evaluated through Dice similarity coefficient (DSC) and intersection over union (IoU) on both test sets. Furthermore, robustness was assessed via color sensitivity tests involving test-time augmentation, and the maximum mean discrepancy (MMD) of bottleneck features was analyzed to investigate shortcut learning and feature space alignment. Significant inter-institutional color shifts were observed, primarily within the L- and b*-channels ( p <0.001). Baseline models exhibited severe performance degradation, with the maximum DSC decreasing from 0.962 (95% CI: 0.959, 0.965) on the internal test set to 0.784 (95% CI: 0.776, 0.792) on the external test set. These models also failed the color sensitivity test, indicating shortcut learning associated with superficial color profiles. Conversely, the proposed protocol maintained robust performance, achieving a peak DSC of 0.975 (95% CI: 0.973, 0.977) on the external test set. Furthermore, the inter-to-intra-institutional MMD ratio significantly decreased across all architectures ( p <0.001), demonstrating the successful acquisition of domain-invariant feature representations. We conclude that strategic photometric augmentation facilitates robust morphological learning, establishing an essential technical foundation for developing reliable, cross-institutional surgical assistance systems.