Validation of expected objective performance indicators in surgical data analysis

Objective Performance Indicators (OPIs) quantify surgical activity by measuring critical aspects of a case (e.g., surgeon movements, interaction forces, etc.) at clinically meaningful steps. They are particularly useful in robotic-assisted surgery, where the information required to compute them can be harvested automatically, and early studies have found that they help quantify surgeon skill. However, since surgical procedures can be highly variable and the boundaries between surgical steps often subject to interpretation, OPIs become similarly variable. Yet, OPIs should be insensitive to this variability to remain informative and unambiguous. Previously we proposed a method for computing expected OPI values in a probabilistic framework that has theoretical support for being robust to this variability. Here we validate this method and compare it against the conventional approach on a large clinical dataset. Using 5408 annotated clinical steps from 1,016 clinical cases across seven procedure types, we performed an extensive comparison of this new method alongside the conventional approach. A series of analyses compared the resulting OPI values focusing on potential biases and the variability of their values in the presence of annotator noise. We found that the probabilistic method resulted in OPI values that were less often biased than their conventional OPI counterparts (49.37%, and 56.63% respectively), and the bias magnitudes were smaller. Finally, we found that the probabilistic method resulted in significantly less variable OPI values in 40.56% of comparison, whereas 8.52% showed the opposite. Our analysis suggests that the expected OPIs are more robust to annotator variability, with less bias and smaller variance. We suggest that this probabilistic approach holds promise for the computational analysis, and perhaps the conceptual interpretation, of surgical data.