An Investigation on the Effectiveness of Chatbots in Evaluating Writing Assignments in Efl Contexts

This study aims to explore the effectiveness of chatbots, specifically ChatGPT, in evaluating writing assignments. 51 English as a foreign language (EFL) learners from Türkiye took part in the study voluntarily. Based on a quasi-experimental research design, the study involved dividing the participants into two non-equivalent groups. The participants in the control group received feedback on their writing from a teacher, while the experimental group was given feedback by the ChatGPT. Thus, the research indicated how Artificial Intelligence (AI)-generated feedback was influential in evaluating writing assignments compared to human-generated feedback. The feedback given by the teacher and the ChatGPT was analyzed via a content analysis, and the results showed that the ChatGPT and the teacher feedback had differences in the language style, register, content, and accuracy. Besides, students’ perceptions of the feedback were revealed through the Student Perceptions of Writing Feedback Scale of Marss (2016) and analyzed using Independent Samples t-Tests on the SPSS. The statistical tests indicated that the students’ perceptions did not vary whether the feedback was AI or human-generated. This study suggests that AI-generated feedback can match human feedback in terms of students’ perceptions. Still, the differences regarding feedback features show the strengths of a teacher and a bot for giving feedback by suggesting a more complementary role for AI technology in educational settings.