Accuracy of Artificial Intelligence in CPT Coding in Gynecologic Surgery

INTRODUCTION: Accurate assignment of Current Procedural Terminology (CPT) codes is essential for billing integrity, compliance, and quality improvement. Manual coding, however, is resource-intensive and prone to human error. Advances in artificial intelligence (AI) and natural language processing (NLP) provide opportunities to automate CPT code assignment, yet their application in gynecologic surgery remains limited. While operative reports may vary among surgeons, for accurate coding, operative reports for procedures such as hysterectomy, myomectomy, and endometriosis surgery should include key elements and specific language as it relates to surgical approach, specimen weight, and adnexal. Leveraging large language models (LLMs) with tailored linguistic rules may improve coding accuracy and more standardized operative reports. OBJECTIVE: To evaluate the accuracy of GPT-based LLM in assigning CPT codes from gynecologic operative reports compared with gold-standard billing code assignment. The secondary outcome was to determine the effect of tailored operative note language on precision, recall, and overall accuracy of AI-based CPT coding for gynecologic operative reports. METHODS: 500 operative reports from a large not-for-profit academic health system were analyzed from January 1, 2025, to March 31, 2025. Inclusion criteria were operative notes documenting hysterectomy, myomectomy, or endometriosis surgeries (laparoscopic, robotic, abdominal, or hysteroscopic) with corresponding CPT codes assigned during the billing process. Records missing CPT codes or those related to gynecologic malignancies were excluded. Two models were tested: Model A (baseline model): Unconstrained AI-generated CPT codes were compared with reference-standard codes assigned by certified coders. Model B (with structured language): GPT model prompted with a structured linguistic framework. Rules emphasized: (1) identifying only procedures explicitly documented, (2) coding based on final operative approach, (3) applying uterine weight thresholds for hysterectomy/myomectomy, (4) distinguishing adnexal procedures when bundled versus stand-alone, (5) defining criteria for superficial versus deep endometriosis excision, and (6) excluding codes for procedures not performed. Performance was assessed using macro- and micro-averaged precision, recall, F1 score, and Jaccard index. McNemar’s test compared exact-match accuracy between models. RESULTS: Model A demonstrated moderate accuracy (macro-F1 = 0.53, micro-F1 = 0.54, Jaccard = 0.37). Incorporating linguistic rules (Model B) improved accuracy across all metrics (macro-F1 = 0.61, micro-F1 = 0.61, Jaccard = 0.44). Precision (correct codes when a code was given) increased from 0.61 to 0.70 and recall (ability to capture the correct code) from 0.51 to 0.59 (p=0.00042). The greatest improvement in accuracy was observed in cases where operative notes explicitly documented uterine weight, adnexal status, or extent of endometriosis. Ambiguity in documentation remained a key source of error. CONCLUSIONS: The baseline AI model showed only moderate accuracy, but providing specific operative details (e.g., uterine weight, surgical approach, procedures performed) significantly improved coding performance. Although still less accurate than human coders, these findings highlight the importance of precise operative documentation to make AI tools more accurate. While AI has potential to reduce coding burden and improve billing, further refinement and training are needed before it can be relied upon clinically.