An LLM-based decision aid for collaborative decision-making in juvenile idiopathic arthritis: A multimethod evaluation

Juvenile idiopathic arthritis (JIA) is the most common pediatric chronic inflammatory rheumatic condition, with substantial quality-of-life impacts. Treatment selection is complex due to limited guideline evidence, and difficult trade-offs. Decisions are often solely physician driven. We evaluated the feasibility of Large Language Model-based Decision Aid (LLM-DA) designed to support collaborative rather than solely physician decision-making in JIA care. A multi-method feasibility study was conducted in accordance with the TRIPOD-LLM checklist. The aims were twofold: first, to assess the quality of the LLM-DA (OpenAI’s o3 model) based on evaluations from three pediatric rheumatologists and an exploratory assessment by an LLM-as-a-Judge (OpenAI’s GPT-5 model); and second, to characterize perceived value of the tool from the patient perspective through a mixed methods evaluation (n = 34). For 10 de-identified use cases, rheumatologists and LLM-Judge generated treatment recommendations and scored the LLM-DA outputs on safety, factual consistency, treatment appropriateness, and preference acknowledgment. Rheumatologists rated 97% of outputs as acceptable for safety and preference acknowledgment, 60% for factual consistency, 70% for appropriateness. Treatment recommendation agreement among all evaluators and real-world data was low at <30%. Inter-rater agreement for scores among rheumatologists was almost perfect for safety and preference acknowledgement, moderate for treatment appropriateness, and poor for factual consistency, and remained the same or improved with LLM-as-a-Judge. Variability in evaluator scoring and treatment recommendations may reflect the inherent heterogeneity in acceptable JIA treatment pathways and potential limitations in the LLM-DA’s decision-making logic. Five patient themes were identified: the tool complements rheumatologist expertise, prepares patients through reflection and actionable information, builds confidence for self-advocacy, has variable value depending on the patient-physician relationship, and has potential for broader JIA self-management. These exploratory findings provide a proof-of-concept for using AI to support collaborative decision-making, while highlighting the need for future studies to determine whether LLMs can reliably support clinical practice.