When Treatment Gets More Complicated: AI Readability Declines in Breast Cancer Patient Education

Background The proliferation of Large Language Models necessitates evaluating their performance in communication accessibility. This study compares the linguistic architecture, readability, and automated psycholinguistic text properties of AI-generated breast cancer materials across varying clinical complexities. Methods Twenty standardized questions across diagnosis and treatment categories were queried through three independent iterations. Readability was analyzed using Flesch-Kincaid Grade Level, SMOG, and Gunning Fog indices. Linguistic dimensions were evaluated via LIWC-22 software as computational style metrics. Statistical analyses utilized Wilcoxon, Mann-Whitney U, Spearman’s correlation, and with False Discovery Rate (FDR) corrections. Cross-iteration stability was assessed with the intraclass correlation coefficient (ICC(2,1), two-way random-effects, and absolute agreement). Results Gemini had better readability than ChatGPT (FKGL 9.23 vs 10.07, q = 0.022); both exceeded the 6th-grade threshold. Treatment queries increased linguistic complexity for both (ChatGPT FKGL q = 0.012; Gemini FKGL q = 0.047). Gemini scored higher on Analytic (q = 0.001) and Authentic (q = 0.021) dimensions; ChatGPT scored higher on I-words (q = 0.001) and Cognitive Processes (q < 0.001). Treatment-specific shifts in Gemini Positive Tone (raw P = .041, q = 0.367) and ChatGPT Authentic (raw P = .025, q = 0.114) lost significance after FDR adjustment. ChatGPT showed a negative correlation between readability and Authentic score (ρ = −0.63, q = 0.026). Cross-iteration ICC values were poor-to-moderate for both models (mean ICC: ChatGPT = 0.58, Gemini = 0.59). Conclusions Both LLMs create practical information gaps for patients with limited health literacy, particularly when addressing complex treatment pathways. Finer-grained psycholinguistic claims did not withstand statistical correction, and cross-iteration stability was modest. These automated analyses highlight structural limitations of current LLMs, reinforcing the necessity of clinician-guided curation before deploying them for patient education.