Original Research: Evaluation of an Artificial Intelligence–Driven Language Model for Age- Specific Patient Education in Pediatric Orthopaedics

Introduction Describing complex orthopedic surgical procedures and diagnoses to pediatric patients across a large age range and at varying levels of health literacy can be quite challenging. Generative Artificial Intelligence (GenAI) offer a potential solution by providing real-time age specific explanations for pediatric patients. The use of GenAI large language models (LLMs) could potentially allow providers to more effectively communicate complex medical problems in an age-appropriate and comprehensible manner. Objective This study aims to validate the readability, accuracy, and comprehensiveness of responses produced by the ChatGPT-4 LLM when prompted to explain a specific pediatric orthopedic procedure indicated for a given diagnosis to patients of three different age groups. Methods ChatGPT-4 explanations were analyzed for forty-three pediatric orthopedic procedures and tests across prompts for three different age groups: 6 years old, 10 years old, and 14 years old. Responses were assessed for readability using six quantitative readability indices. The accuracy and comprehensiveness of the responses were rated by seven board-certified pediatric orthopedic surgeons, who also compared them to their own explanations used in clinical practice. Results Each of the six indices showed a significant difference in readability between the 6-, 10-, and 14-year-old age group prompts (p<0.001). The 6-year-old prompts were more readable than the 10- and 14-year-old prompts (p<0.001), and the 10-year-old prompts were more readable than the 14-year-old prompts (p<0.001) (Figure 1). Out of a 5-point scale, the mean (SD) attending-rated accuracy for the AI responses was 4.15 (0.81) with no significant differences in rated accuracy between procedures (p=0.051) or age groups (p=0.27). The mean attending-rated comfort was 3.95, with interrater reliability coefficients for accuracy and comfort of 0.92 and 0.94, respectively. Conclusions GenAI LLMs can generate age-specific explanations for common surgical procedures, treatments, and diagnostic markers in pediatric orthopedics. Physicians validated the accuracy of its responses and demonstrated a high level of comfort using the GenAI explanations in clinical settings. Level of Evidence Level III/IV Key Concepts 1. An AI language model adjusted its explanations based on the child’s age, providing simpler language for younger children and more advanced explanations for older children. 2. Across six established readability measures, explanations written for younger children were consistently easier to understand than those created for older age groups. 3. Seven board-certified pediatric orthopedic surgeons found the AI-generated explanations to be highly accurate, with an average rating of 4.15 out of 5. 4. Physicians were generally comfortable with the use of AI-generated educational explanations in practice, and their ratings were highly consistent with one another. 5. When reviewed by a physician, generative AI could be a helpful tool for improving patient education, health literacy, and shared decision-making in pediatric orthopedic care.