LLM Tutors for Computer Science Courses

Chatbots can provide quick feedback and also answer follow-up questions for students. This study aims to explore the use of chatbots for courses by taking advantage of the pre-defined course assignments to give the chatbot a sample solution to reduce errors. The study focuses on the ability of chatbots to provide high-level plans, pseudocode, and debug students' code. The results showed that the chatbot excelled at providing high-level plans and pseudocode that was often very similar to the sample solution. However, the chatbot also encountered some limitations as it struggled to accurately find errors when debugging and sometimes would incorrectly flag a portion of code that was correct. Overall, chatbots have potential in computer science course education as an assistant, but still require more improvement until it is reliable.