A multidisciplinary assessment of ChatGPT’s knowledge of amyloidosis

Amyloidosis is a rare, multisystem disease with several subtypes including AA (secondary), AL (amyloid light chain), and ATTR (transthyretin amyloidosis). In addition to variable symptoms and multidisciplinary management, amyloidosis being a rare disease further contributes to patients being at risk for decreased health literacy regarding their condition. Increased access to education materials containing simple, plain language may bridge literacy gaps and improve outcomes for patients with rare diseases such as amyloidosis. The large language model (LLM), Chat Generative Pre-Trained Transformer (ChatGPT), may be a powerful tool for improving the availability of accurate and easy to understand education materials. Amyloidosis-related questions from cardiology, gastroenterology, and neurology were sourced from esteemed medical societies and institutions along with amyloidosis Facebook support groups and inputted into ChatGPT-3.5 and GPT-4. Answers were graded on 4-point scale with both models responding to the majority of questions with either “comprehensive” or “correct but inadequate” answers with only 1 (1.2%) answer by GPT-3.5 graded as “completely inaccurate”. When assessing reproducibility, GPT-3.5 scored reliably on more than 83.3% of responses, while GPT-4 produced above 98.2% consistent answers. Our findings show that ChatGPT can potentially serve as a supplemental tool in disseminating vital health education to patients living with amyloidosis.