Comparative cross-sectional evaluation of large language model performance in answering common patient questions on chronic kidney disease-mineral and bone disorder: Grok-4, DeepSeek-V3.2, ChatGPT-5.4, and Gemini-3-Pro

Chronic kidney disease–mineral and bone disorder (CKD-MBD) is a prevalent and serious complication among patients with advanced chronic kidney disease and those undergoing dialysis, with long-term management heavily dependent on patients’ self-management capabilities. As large language models (LLMs) become increasingly utilized in health information domains, evaluating their performance in answering patient questions within this complex subspecialty holds significant clinical relevance. This study aimed to compare the response performance of four mainstream LLMs (Grok-4, DeepSeek-V3.2, ChatGPT-5.4, and Gemini-3-Pro) to common CKD-MBD patient inquiries, focusing on information quality, readability, and transparency rather than serving as a formal verification of content validity or guideline concordance. This cross-sectional comparative evaluation study synthesized data from Google Trends, DXY (Ding Xiang Yuan), clinical practice guidelines, and relevant literature to construct a standardized question set comprising 26 frequently encountered CKD-MBD patient consultation items. The response performance of Grok-4, DeepSeek-V3.2, ChatGPT-5.4, and Gemini-3-Pro was simultaneously assessed. A multi-dimensional evaluation framework incorporating DISCERN, EQIP, JAMA benchmarks, and GQS was employed to blindly assess information quality, transparency, and overall utility. Six readability metrics (ARI, FRES, GFI, FKGL, CL, and SMOG) were applied to quantify text accessibility. Standardized safety downgrade rules were established for clinical risk screening. Inter-rater reliability was examined using intraclass correlation coefficients. Because the same 26 questions were answered by all four models, between-model comparisons were performed using question-level paired analyses, including Friedman tests for overall comparisons and paired Wilcoxon signed-rank tests for pairwise comparisons. Effect sizes and 95% confidence intervals were reported. A total of 104 model-generated responses were included. Question-level paired analyses showed significant between-model differences in the main quality indicators (DISCERN: P < 0.001, Kendall’s W = 0.248; EQIP: P < 0.001, Kendall’s W = 0.283; GQS: P = 0.007, Kendall’s W = 0.156) and in all readability metrics (all P < 0.001). Grok-4 achieved higher scores on patient-facing information quality indicators (DISCERN, 62.50 ± 6.62; EQIP, 73.08 ± 9.91%; GQS, 4.23 ± 0.59), followed by DeepSeek-V3.2, with ChatGPT-5.4 at an intermediate level. Gemini-3-Pro achieved the highest readability scores (FKGL, 10.71 ± 1.65; FRES, 50.62 ± 6.81) but the lowest information quality scores (DISCERN, 53.12 ± 12.37; GQS, 3.62 ± 0.85), whereas Grok-4 exhibited the poorest readability (FRES, 26.69 ± 10.60). All models performed poorly on JAMA transparency benchmarks, with mean total scores of 0.08 ± 0.39 for ChatGPT-5.4, 0.08 ± 0.27 for DeepSeek-V3.2, 0.00 ± 0.00 for Gemini-3-Pro, and 0.50 ± 0.95 for Grok-4 (on a 4-point scale), indicating that none met adequate disclosure requirements, despite Grok-4 showing a relatively higher mean score (0.50/4, 12.5%). Inter-rater agreement varied across indicators, with single-measure ICCs ranging from 0.277 to 0.852 and average-measure ICCs ranging from 0.535 to 0.945. Detailed ICC estimates with 95% CIs are provided in Supplementary Table S1 . For high-risk clinical topics such as dietary restrictions, phosphorus-lowering therapy, dialysis management, and parathyroidectomy, model responses frequently omitted CKD staging or dialysis status information, inadequately emphasized biochemical monitoring, and contained potentially misleading statements. Mainstream large language models may serve as supportive tools for CKD-MBD patient health education, though performance varies significantly. Grok-4 and DeepSeek-V3.2 achieved higher patient-facing information quality and completeness scores, while Gemini-3-Pro offers greater textual accessibility. However, given the absence of external factual reference standards and the generally limited transparency across all models, current LLM outputs cannot substitute for clinical guidance and should be applied cautiously under physician supervision. Future efforts should establish CKD-MBD-specific safety evaluation standards to enhance clinical adaptability and application safety.