Chemistry Students’ Artificial Intelligence Literacy through their Critical Reflections of Chatbot Responses

Instructors use of Artificial Intelligence (AI) language models (i.e., chatbots) as an educational resource will require an understanding of students' AI literacy, namely their ability to critically reflect on the relevance, trustworthiness, and quality of these tools in the context of chemistry. This study sought to describe students' AI literacy via open-ended surveys of general chemistry I students and students in an upper-level chemistry elective. Thematic analysis was used to create a deeper understanding of chemistry students' AI literacy when considering chatbots. Based on students' responses, they were categorized as either with reservations toward chatbots or without reservations toward chatbots. Thematically, students tended to either reason with the utility/benefit of the tool or reason with concern toward the accuracy of the tool. Results suggest that students are more of a range between these two extremes. This new deeper understanding of AI literacy in chemistry can support instructional practices and inform future research efforts in AI literacy.