Evaluating ChatGPT’s Responses to Chiropractic Patient Questions: A Comparative Study

Background Large language models (LLMs) are used at an increased frequency by patients to answer health related questions. Prior studies have evaluated LLMs and their performance with musculoskeletal (MSK) and spine domains, but have not been used to evaluate chiropractic-relevant content. Objective To evaluate ChatGPT responses to commonly asked chiropractic patient questions for concordance with chiropractic clinical practice guidelines. Methods Thirty questions addressing low back pain, neck pain, and chronic musculoskeletal pain were developed based on current U.S. chiropractic clinical practice guidelines (CPGs). ChatGPT (GPT-5.2) generated responses in isolated sessions. Responses were assessed for readability using established indices, then five licensed chiropractors independently rated each response using a 5-point Likert scale across four domains: relevance, clarity, completeness, and accuracy. Inter-rater reliability was evaluated using Krippendorff’s alpha (ordinal). Results Readability measures demonstrated that ChatGPT responses were written at approximately a college-level reading grade. Expert ratings across all domain means (standard deviation) were predominately clustered at the upper end of rating scale (1-5): Relevance 4.72 ± 0.56; Clarity 4.76 ± 0.50; Completeness 4.23 ± 0.81; Accuracy 4.00 ± 1.02. The Krippendorff’s alpha (ordinal) ranged from -0.023-0.214 across the domains, all reflective of poor inter-rater reliability. Conclusion ChatGPT responses were broadly consistent with chiropractic CPGs, though readability levels exceeded recommended patient education standards. While the expert panel frequently assigned similar scores, minor scoring differences and restricted score distributions produced low chance-corrected reliability. These findings suggest that ChatGPT can produce guideline concordant responses but readability and interpretive variability should be considered when integrating LLM generated information into patient education.