Outsmarting ChatGPT in Chemistry: Using Student-Formulated Questions for Critical AI Evaluation

The use of Artificial Intelligence (AI) is increasingly impacting education, and there is a growing sector of educators who have expressed the need to incorporate it into classrooms in innovative ways that both engage students and foster critical thinking. We implemented an activity in an Introductory Chemistry course at the University of Florida during the Fall 2024 term, where students were tasked with “outsmarting” ChatGPT by formulating a multiple-choice question that the AI failed to answer correctly. Students were asked to submit a screenshot of the question, the AI response, and a slide explaining the correct answer. This activity helped to identify some topics where misconceptions or incomplete knowledge were evidenced by the formulated questions. In some cases, poorly formulated or ambiguous questions exposed gaps in student comprehension of some topics. The activity also helped to identify some of ChatGPT’s limitations at the time the activity was carried out. Persistent errors were observed in mathematical calculations, consistent with prior reports, as well as instances where the AI generated correct explanations but incorrect final answers. Topics with the most student submissions were stoichiometry, electronic structure, and significant figures.