Is ChatGPT the Academic Catalyst We've all been Waiting For?

The excitement around ChatGPT 3.5 underscores its potential to transform various fields in education, including STEM. However, we must approach these claims cautiously. While AI can enhance STEM education, there are ethical concerns and potential inaccuracies linked to unsupervised automated responses. To comprehensively evaluate ChatGPT's influence on STEM, we conducted a controlled experiment that involved answering a question set in mathematics and CS in a time-limited session. To avoid bias, we recruited four groups of math and CS students with similar abilities -each group comprised five students. Two groups utilized ChatGPT, while the other two did not. Students who used ChatGPT were tasked with explaining how and where they employed the tool. Conversely, students who did not use ChatGPT were asked to showcase their problem-solving process. We analyzed the responses from these four groups, alongside the analysis of ChatGPT conversations for those who employed ChatGPT. Performance, confidence level, and completion time of each participant were recorded. Experts in mathematics and CS were then consulted to review participant responses. These experts were subsequently interviewed to gain deeper insights and draw conclusive findings. Our findings show that students who didn't use ChatGPT in Mathematics scored better than those who did, Specifically, ChatGPT provided the correct working process but yielded a wrong final answer due to arithmetic mistakes. Similarly, in programming, ChatGPT led to less elegant code. Our findings provide valuable insights into the benefits and challenges of AI integration in these fields, helping educators and students to adapt to AI advancements.