Answers are grounded in this paper and cite their sources.
Background Accurate assignment of Current Procedural Terminology (CPT) codes is essential for reimbursement, compliance, and quality reporting in pediatric orthopedic surgery. Manual coding remains resource-intensive and prone to error and variability. Large language models (LLMs) represent a potential solution for automating CPT assignment directly from operative documentation. This study evaluated the performance of ChatGPT-4o and ChatGPT-5 in generating accurate CPT codes from de-identified pediatric orthopedic operative notes. Methods A retrospective observational study was performed using 20 de-identified pediatric orthopedic operative notes collected between January and April 2025. Notes were drawn from cases performed by 7 attending pediatric orthopedic surgeons, representing a diverse academic case mix to expose the models to variability in operative dictation style and documentation standards. Each case was categorized as single-code (1 CPT) or multi-code (≥2 CPT). Each operative note was entered into ChatGPT-4o and ChatGPT-5 using a standardized zero-shot prompt without priming or fine-tuning, and model-generated CPT codes were recorded. CPT codes assigned by the institutional billing department served as the gold-standard reference. Model outputs were adjudicated as true positives, false positives, or false negatives. Performance metrics included F1 score, precision, recall, exact-match accuracy, and rates of over- and under-coding. Results ChatGPT-4o achieved an overall F1 score of 0.48 (recall 56%, precision 45%), with higher accuracy in single-code cases (F1 0.53) compared with multi-code cases (F1 0.41). Over-coding occurred in 70% of cases, under-coding in 60%, and exact-match accuracy was 30%. ChatGPT-5 outperformed ChatGPT-4o across all measures, achieving an overall F1 score of 0.69 (recall 74%, precision 67%), with higher accuracy in single-code cases (F1 0.76) than multi-code cases (F1 0.61). Over-coding occurred in 25% of cases, under-coding in 30%, and exact-match accuracy improved to 60%. Conclusions ChatGPT-5 demonstrated markedly improved performance over ChatGPT-4o in CPT code prediction for pediatric orthopedic operative notes, particularly in single-code procedures. While accuracy gains were substantial, error rates remain above acceptable thresholds for unsupervised deployment. These findings suggest that LLMs may serve as augmentative tools to streamline coding workflows and reduce administrative burden, with potential earlier applicability in single-code pediatric orthopedic cases. Larger-scale validation and integration studies are warranted to support broader clinical implementation. Level of Evidence Level III - Retrospective comparative study Key Concepts 1. ChatGPT-5 substantially outperformed ChatGPT-4o in CPT code prediction from pediatric orthopedic operative notes, achieving an F1 score of 0.69 versus 0.48 and an exact-match accuracy of 60% versus 30%. 2. Both models performed better on single-code procedures than multi-code procedures, with ChatGPT-5 achieving an F1 score of 0.76 in single-code cases compared to 0.61 in multi-code cases. 3. Over-coding was the dominant error pattern for ChatGPT-4o (70% of cases), while ChatGPT-5 substantially reduced both over-coding (25%) and under-coding (30%), suggesting improved contextual bundling recognition. 4. Current error rates remain above acceptable thresholds for unsupervised clinical deployment; however, supervised integration of LLMs as a first-pass coding assistant may reduce administrative burden and variability in routine pediatric orthopedic cases. 5. Larger multi-institutional validation studies with domain-specific fine-tuning and EHR integration will be necessary before LLM-assisted CPT coding can be responsibly implemented in clinical billing workflows.
No takes yet. Share an insight, caveat, or question.
Manem et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: