The ClinicalBigBird NLP model outperformed board-certified anesthesiologists in ASA-PS classification, achieving a higher specificity (0.901 vs. 0.897), precision (0.732 vs. 0.715), and F1-score (0.716 vs. 0.713).
Observational (n=717,389)
No
Do natural language processing models improve the accuracy of ASA-PS classification compared to human physicians in surgical patients?
An NLP model (ClinicalBigBird) can automatically and objectively classify ASA-PS scores from pre-anesthesia evaluation summaries with performance comparable to or better than board-certified anesthesiologists.
Absolute Event Rate: 0.716% vs 0.713%
p-value: p=<0.01
The American Society of Anesthesiologist's Physical Status (ASA-PS) classification system assesses comorbidities before sedation and analgesia, but inconsistencies among raters have hindered its objective use. This study aimed to develop natural language processing (NLP) models to classify ASA-PS using pre-anesthesia evaluation summaries, comparing their performance to human physicians. Data from 717,389 surgical cases in a tertiary hospital (October 2004-May 2023) was split into training, tuning, and test datasets. Board-certified anesthesiologists created reference labels for tuning and test datasets. The NLP models, including ClinicalBigBird, BioClinicalBERT, and Generative Pretrained Transformer 4, were validated against anesthesiologists. The ClinicalBigBird model achieved an area under the receiver operating characteristic curve of 0.915. It outperformed board-certified anesthesiologists with a specificity of 0.901 vs. 0.897, precision of 0.732 vs. 0.715, and F1-score of 0.716 vs. 0.713 (all p <0.01). This approach will facilitate automatic and objective ASA-PS classification, thereby streamlining the clinical workflow.
Yoon et al. (Sat,) conducted a observational in Pre-anesthesia evaluation for surgery (ASA-PS classification) (n=717,389). ClinicalBigBird NLP model vs. Board-certified anesthesiologists was evaluated on F1-score for ASA-PS classification (95% CI 0.701-0.730, p=<0.01). The ClinicalBigBird NLP model outperformed board-certified anesthesiologists in ASA-PS classification, achieving a higher specificity (0.901 vs. 0.897), precision (0.732 vs. 0.715), and F1-score (0.716 vs. 0.713).