Prompt Engineering and Follow-Up Questioning Improves the Readability of Spine Surgery Questions in Large Language Models

The field of spine surgery is complex, and patient education material within the field are often written at a reading level that exceeds the recommended standard. Large language models (LLMs), such as ChatGPT, have shown potential for generating educational content, but require further investigation regarding readability and adaptability in responding to follow-up questions. The purpose of this study is to evaluate which model and prompting strategies improve the readability the most, especially in newer models. ChatGPT-4o and 5 were prompted with 45 standardized spine surgery questions across five common procedures. Each question underwent five prompting phases: baseline (Phase 1), follow-up clarification (Phase 1.5), sixth-grade level request (Phase 2), rule-based prompting (Phase 3), and direct readability targeting (Phase 4). Readability was measured using SMOG, FRE, FKGL, GFI, and CLI scoring. Results were standardized and analyzed using paired tests, ANOVA, and interaction models. A resource for spine surgeons with the most readable answers was created. ChatGPT-4o generated significantly more readable responses than ChatGPT-5 across all phases except Phase 1 (p<0.001). However, ChatGPT-5 produced more reliable citations (p<0.05). Phase 2 yielded the most readable responses, with 51.11% meeting the sixth-grade level. Follow-up clarification questions and simplified prompt engineering were more effective than complex rule-based prompts. Prompt engineering and follow-up questioning significantly enhanced the readability of LLM-generated responses to spine surgery questions at a level appropriate for patients. Further investigation needs to assess whether these responses are more readable in real-world clinical settings instead of relying on objective readability scoring methods.