C56-18 AI for the Sub-I: Assessment of Chatbot Use by Senior Medical Students During a Critical Care Clerkship

Abstract Rationale Artificial intelligence (AI) is rapidly transforming healthcare, yet integration into undergraduate medical education remains limited. To effectively integrate AI, it is necessary to understand medical students’ perceptions regarding AI use, and to create exercises that are effective in fostering critical thinking and AI literacy with regards to their use for clinical queries. Methods Fourth-year medical students rotating through the ICU on their critical care clerkship completed a pre-survey assessing familiarity, comfort, and use of AI in clinical settings. Students then analyzed a standardized case, generating a summary statement, differential diagnosis, and management plan as part of a required assignment. They subsequently used their summary statement as a prompt for an AI chatbot of their choice to perform the same task. Students compared their responses with the AI output and completed a reflective post-survey. Responses were scored by faculty reviewers as detailed, basic, or no analysis. For analysis, we summarized post-survey responses by verification method, reporting frequencies and percentages. To assess trends in the odds of the outcomes over time, we fitted a logistic regression model including both time and time-squared terms, but retained only the linear term since there was no evidence of a quadratic trend in all outcomes. Results Pre-assignment survey data (n = 193) demonstrated that 50% (n = 97) had tried using a chatbot to assist in medical education or clinical work, and subsequent analysis demonstrated a consistent level of AI familiarity and clinical use over time from 2024-2025. The vast majority of students (94%, n = 182) felt AI had a role in healthcare with most common applications being administrative, education and documentation. Post-assignment survey responses (n = 188) demonstrated strong critical engagement, most notably seen in recognition of omission/irrelevant content (27% detailed, 56% basic; 83% ≥ basic), consideration of bias/harm (32% detailed, 48% basic; 80% ≥ basic), and identification of AI limitations (40% detailed, 51% basic; 91% ≥ basic). In terms of AI output’s alignment with traditional medical resources, 29% of students overall felt that AI aligned comprehensively, while 71% felt there was only a partial alignment. Hallucinations were detected by 21% of respondents (7% significant hallucination, 14% minor hallucination). Conclusions Despite minimal prior exposure, most students demonstrated at least basic critical analysis of AI-generated reasoning. Structured, case-based exercises like this can effectively promote AI literacy in clinical care. This helps prepare medical students to navigate and critically engage with emerging AI technologies in modern medicine. This abstract is funded by: None