A Comparative Analysis of the Accuracy of Retrieval Augmented Generation Versus Large Language Models for Gout-Related Queries

Importance: To better understand ways to improve the answers from large language models by feeding them disease specific guidelines which can potentially act as a clinical decision support tool in the future Objective: To demonstrate the effect of retrieval augmentation on answers related to gout in terms of accuracy, conciseness and unambiguity. Design: Observational. Setting: Virtual experimental setting. Participants: ChatGPT 3.5 and Retrieval augmented generation powered ChatGPT. Exposure: Nine questions derived from the 2020 American College of Rheumatology guidelines on gout. Main outcomes: A zero to two subjective scale to measure the accuracy, conciseness, and unambiguity of answers, objective measure for word count, and text similarity using Levenshtein distance. Results: There was a significant difference in the accuracy, conciseness, and unambiguity score between RAG and ChatGPT with the mean difference of 0.78; t(8)=2.4, p = 0.04, mean difference of 1.56; t(8)=8.85, p=0.00, and mean difference of 1;t(8)=3.46, p=0.00 respectively. In addition, the word count was also significantly lower in the RAG group with mean difference of 250.78; t(8)=23.99, p=0.00. The mean similarity score was 0.16(95% CI=0.06 to 0.26). Conclusion and relevance: These findings suggest that incorporating domain specific knowledge to LLM using RAG greatly enhance the accuracy, conciseness, and unambiguity of answers related to gout.