Student Interaction with NewtBot: An LLM-as-tutor Chatbot for Secondary Physics Education

Chatbots based on state-of-the-art large language models (LLMs) hold potential to act as beneficial educational tools. However, challenges to LLMs in education include concerns about not only the accuracy and interpretability of AI-generated text, but also about productive student engagement and positive user experience with LLM chatbots. In this paper, we introduce a physics education chatbot called NewtBot. We designed NewtBot to act as a personalized automated tutor to support secondary students' learning as they complete physics tasks. NewtBot's web interface has a modifiable back-end that internally prompts GPT-3.5 to produce different LLM behaviors. In a user study with German secondary school students (n=50), we evaluated student interactions with three different configurations of the GPT-3.5 back-end: a general-purpose "baseline" model, a setting-specific "tutor" model, and a problem-specific "feedback" model. We find that students had overall positive experiences using NewtBot, and that the setting-specific "tutor" model had the highest user experience ratings. Additionally, despite a majority of participants (72%) expressing apprehensions about using chatbots for school, 70% said they would use NewtBot to help with their physics school work.