Evaluating large language models for patient consultations on osteoporotic vertebral compression fractures: a two-phase comparative study

To evaluate the quality, readability, error patterns, and expert-rated response safety of large language model–generated information for patients with osteoporotic vertebral compression fractures (OVCFs), and to explore differences between a selected model and a spine surgeon in a simulated written consultation setting. This two-phase sequential benchmarking study evaluated ChatGPT-5.1, Gemini 2.5 Pro, DeepSeek-V3.2, and Qwen3-Max. In Phase 1, the four models answered 26 expert-developed OVCF questions. Three senior spine surgeons independently rated accuracy, comprehensiveness, and expert-rated response safety using predefined 5-point criteria. Response length, readability, aggregate model-ranking concordance, and predefined error categories were also assessed. In Phase 2, ChatGPT-5.1 and one spine surgeon independently answered 17 de-identified patient-derived questions using the same standardized instruction. Accuracy, comprehensiveness, and expert-rated response safety were evaluated by two senior spine surgeons through consensus, whereas empathy and usability were rated by one researcher. Friedman and paired Wilcoxon signed-rank tests were used, with Benjamini–Hochberg correction for multiple comparisons. In Phase 1, the four models differed in accuracy (Friedman χ 2 (3) = 12.64, P = 0.00548, Kendall’s W = 0.162), comprehensiveness (χ 2 (3) = 38.68, P = 2.03 × 10 −8 , W = 0.496), and expert-rated response safety (χ 2 (3) = 19.86, P = 0.000182, W = 0.255). After multiplicity correction, ChatGPT-5.1 and DeepSeek-V3.2 generally received higher ratings than Gemini 2.5 Pro, although several comparisons with Qwen3-Max were not statistically significant. Omission errors occurred across all four models. In Phase 2, no statistically significant difference was detected between ChatGPT-5.1 and the physician comparator in accuracy (paired mean difference, 0.24; 95% CI − 0.12 to 0.59; adjusted P = 0.2482) or expert-rated response safety (difference, 0.53; 95% CI − 0.06 to 1.12; adjusted P = 0.1469). ChatGPT-5.1 received higher ratings for comprehensiveness, researcher-rated empathy, and usability (adjusted P = 0.0012, 0.0011, and 0.0012, respectively) and had lower estimated reading-grade requirements. ChatGPT-5.1 and DeepSeek-V3.2 generated structured OVCF patient-education content with favorable expert ratings, but clinically relevant omissions remained. The exploratory Phase 2 comparison involved only 17 questions and one physician comparator and did not establish equivalence, noninferiority, or real-world clinical safety. LLMs may support clinician-supervised patient education and communication but should not replace diagnosis, imaging interpretation, individualized risk assessment, procedural decision-making, or complication management.