Teachers in the loop? An analysis of automatic assessment systems under Article 22 GDPR

The possibility to utilize AI in assessments may radically transform higher education by offering novel ways of understanding student learning, previously considered unfeasible.1 AI-based assessments not only promise benefits for students such as immediate feedback and impartial grading, but they also pledge to reduce teacher workload and fatigue, allowing instructors to focus on ‘higher-level pedagogic work’ rather than ‘routine and procedural tasks’.2 While current imaginaries about the deployment of AI-based assessment suggest increased efficiency, cost-effectiveness, and the solution to an array of problems in the educational context, critical commentators contend that the deployment of these technologies might also lead to ‘disadvantageous and undesirable outcomes’.3 In fact, the recent A-level algorithmic assessment controversy in the UK provides a concrete example of the harms that may arise from the deployment of automatic assessment in an educational context.4 The aim of this article is to explore whether and to what extent the requirements set forth in Article 22 of the General Data Protection Regulation (GDPR) apply within the context of AI-based automatic assessment systems, focusing particularly on the legal obligation to ensure that a teacher remains in the loop, for example by being capable of overseeing and overriding decisions when necessary.5 Although fully automated classrooms and robot teachers are unlikely to enter higher education institutions anytime soon, this article contends that it is important to critically consider how AI-based assessment technology is changing public sector decision-making within higher education and whether Article 22 requires a human-centred system of control.6 This is because AI-based assessment is already widely available on the market and being deployed in higher education, making it capable of transforming everyday educational practices as well as shaping educational imaginaries.7