Improving ASA-PS classification accuracy using private-deployment-compatible large language models: a multilingual evaluation

Background ASA Physical Status (ASA-PS) classification is widely used in preoperative risk assessment but has substantial inter-rater variability. Large language models (LLMs) may improve classification consistency. Methods We assessed five early-generation and six advanced reasoning LLMs on published vignettes using an optimized prompt. Set A (Hurwitz 2017, n=10) served for prompt development; Set B (Owens/Mak/Riley, n=10) and Set C (De Cassai 2019, n=8, ASA I–V) provided independent validation. DeepSeek R1 was also tested on 150 Cheng et al. (2024) vignettes for aggregate distributional concordance. Set A was evaluated in English and Italian. Results On Set A, advanced LLMs reached 97.5% accuracy (117/120; bootstrap CI 95.0%–100.0%) versus 77.0% for early-generation models (77/100; p=0.0008) and the human baseline. On Set B, the four selected advanced LLMs converged for 9/10 cases. On held-out Set C, DeepSeek R1 achieved 100% accuracy (8/8) with a seven-call self-consistency ensemble. On Cheng et al., DeepSeek R1’s aggregate distribution showed no significant difference from experts (p=0.927; mean 2.30 vs. 2.32), but underrepresented ASA IV cases (2.7% vs. 8.0%). Conclusions Advanced reasoning LLMs performed strongly on standardized ASA-PS benchmarks, and DeepSeek R1 classified held-out ASA I–V cases accurately while showing aggregate-level concordance with experts. Self-consistency agreement scores flagged borderline cases. ASA IV underrepresentation supports pairing any deployed classifier with an independent high-risk escalation rule. Findings are limited to synthetic vignettes and one language pair; prospective validation on real-world electronic health records is required.