A Systematic Review of the Utility of Generative Artificial Intelligence in Orthopaedic Surgery Patient Education Material (Preprint)

BACKGROUND Large language models (LLMs), a type of generative Artificial Intelligence (AI), are capable of interpreting and generating human-like text and offer the potential to synthesize and simplify complex medical information into accessible, human-like responses in a seemingly instantaneous manner. However, concerns remain regarding their implementation in the healthcare setting. OBJECTIVE The aim of this systematic review is to assess the accuracy, readability and quality of responses provided by LLMs when responding to frequently asked questions (FAQs) by patients, generating patient education materials (PEMs) and re-writing pre-existing PEMs within the field of orthopaedic surgery. METHODS A scoping literature review following PRISMA guidelines was conducted using PubMed, Ovid and Embase databases. Papers where LLMs were used to generate, re-write or modify PEMs or respond to frequently asked questions in orthopaedics with outputs evaluated using any validated quality, accuracy or readability assessment tool were included. Studies that compared the output between different LLMs were also included. Papers that were available as abstract-only, non-English language and did not assess LLMs were excluded. This systematic review was registered with PROSPERO (CRD420251087612). RESULTS A total of 94 studies were identified in the literature search and 32 met the inclusion criteria. All LLM models provided moderate to highly accurate responses that were relevant, however factual inaccuracies and an inability to handle clinical nuance were noted with all material produced by LLMs. 12/32 (37.5%) studies utilized instruction based fine tuning (IBFT), 2/32 (6.25%) studies utilized reinforcement learning human feedback (RLHF) and 1/32 (3.13%) studies utilized both IBFT and RLHF to fine-tune and optimize the LLM response. LLM responses that were not fine-tuned provided responses that exceeded average patient literacy levels. Fine-tuned ChatGPT-4.0 models significantly improved clarity and reduced complexity without compromising clinical accuracy. Comparative studies indicated gradual LLM improvement. CONCLUSIONS LLMs demonstrated moderate to high accuracy when producing responses that could be used for patient education. They offer information in an easily accessible, seemingly instantaneous manner that is either free of cost or requires minimal expenditure. However, these responses consistently produce inaccuracies and lack clinical nuance. Whilst fine-tuning improved LLM response, and gradual model improvement was noted, constant expert oversight remains essential as uninformed LLM use risks the potential of mis-information and error for patients. Inadequate prompting and a high reading level of responses are all limitations to LLM safety and utility in healthcare. LLMs will continue to evolve rapidly and ongoing research in to the use of LLMs in patient education in orthopaedic surgery is necessary, especially with well-structured methods where LLMs receive adequate IBFT and RLHF to optimize their function. CLINICALTRIAL This systematic review was registered with PROSPERO (CRD420251087612).