Video‐Based Assessment of Operative Competency in Laparoscopic Cholecystectomy: A Prospective Observational Study to Inform Rubric Design for AI ‐Assisted Evaluation

Objective This pilot study aimed to assess the interrater reliability of quantitative measures derived from video‐based evaluation of laparoscopic cholecystectomy performance, with the future aim of developing an objective assessment tool and the use of artificial intelligence for this procedure. Methods This study was conducted at a single metropolitan training hospital in Australia. Quantitative metrics based on 30 randomly selected videos were evaluated by three surgeon assessors. Interrater reliability was measured using percent agreement (PA) for categorical items and intraclass correlation coefficient (ICC) models for numerical items. Results Interrater agreement varied. For numerical items, moderate agreement was found for Hartmann's pouch grasping, gallbladder rotation, and incorrect clipping; good for perforation; and excellent for time to critical view of safety. Key categorical procedure‐specific items (e.g., 360° cystic duct dissection, two‐window view) achieved at least moderate agreement. Case difficulty assessment showed good to excellent agreement. Registrars demonstrated prolonged time to critical view of safety, more frequent Hartmann's pouch grasping, more gallbladder rotations, and increased instrument insertions. Competency ratings showed 45.8% of registrars as ‘developing’, while consultants were rated predominantly ‘good’ (40%) or ‘excellent’ (20%). Conclusions There is potential to develop an objective video‐based evaluation tool for laparoscopic cholecystectomy based on more quantitative assessment items, as evidenced by the interrater reliability results in this pilot study. Further research on artificial intelligence and computer vision will be beneficial in establishing a reliable and valid scoring tool.