Evaluation of AI-Generated Responses to Pediatric Vaccination Information Requests: Cross-Sectional Content Analysis (Preprint)

BACKGROUND Artificial intelligence (AI) language models are demonstrating increased efficacy in providing recommendations to queries for pediatric vaccinations. However, it is unclear whether newer AI models, such as ChatGPT and BARD (Gemini), can improve the accuracy of these recommendations. OBJECTIVE To qualitatively evaluate the appropriateness of AI models responses to fundamental pediatric vaccination questions. METHODS A questionnaire of 15 questions addressing recommendations for pediatric vaccination schedule, COVID vaccination, vaccines needed for travel, and vaccines for specific diseases, based on guideline-based prevention topics, was utilized. Each question was posed three times to two AI models, ChatGPT and BARD, and the responses were recorded. A total of three board-certified pediatricians graded each set of responses. These responses were graded as appropriate or inappropriate and which model provided better answers in comparison. A set of responses was graded as inappropriate if any of the three responses contained inaccurate, misleading, or harmful information. RESULTS ChatGPT's responses were graded as appropriate for 14 of 15 questions (93%), while BARD's responses were graded as appropriate for 11 of 15 questions (73%). ChatGPT was rated as superior to BARD for 80% of the questions, while BARD was rated as superior to ChatGPT for 20% of the questions. CONCLUSIONS Both AI models provided largely appropriate answers to simple pediatric vaccination questions as evaluated by pediatricians. However, responses from ChatGPT were found to be superior to BARD.