Learning analytics and AI: Politics, pedagogy and practices

It is arguably human nature—and most certainly bureaucratic nature—to want to quantify our successes and failures. The ability to calculate a numerical value to represent the progress of an individual or an institution, a country even, is now central to evidence-based policy and practice. The counterpoint to “treasuring what we measure,” is of course, “what counts can’t always be measured, and what’s measurable doesn’t always count.” All institutions, education being no exception, have long wrestled with the tension that the powerful abstractions afforded by quantitative analysis also lose vital detail as context is stripped out. This is what makes the design of metrics to gauge the quality of nuanced human processes and outcomes (such as teaching and learning) so controversial. Across society, we see this debate now playing out in all spheres of human life, with ethical frameworks and professional codes of practice proliferating (eg, Asilomar, 2017; Floridi et al., 2018; IEEE, 2017; Montreal, 2017; Partnership on AI [PAI], 2018)—although it is not clear that these are making much impact yet on computing companies (Whittaker et al., 2018, Section 2.3). How this debate should unfold in education and lifelong learning is the focus of this special issue, which brings together leading scholars in the field of Learning Analytics (LA) and Artificial Intelligence in Education (AIED)—fields that are viewed with varying degrees of excitement and suspicion by parents, students, teachers, journalists and scholars. The fears are reasonable: that quantification and autonomous systems provide a new wave of power tools to track and quantify human activity in ever higher resolution—a dream for bureaucrats, marketeers and researchers—but offer little to advance everyday teaching and learning in productive directions. This fear is justified in our post-Snowden era of pervasive surveillance, and post-Cambridge Analytica data breaches. Partly however, this fear is also born of lack of awareness about the diverse forms that LA/AIED take, which is equally understandable—to outsiders, these are new and opaque technologies. It follows that if we do not want to see concerned students, parents and unions protesting against AI in education, we need urgently to communicate in accessible terms what the benefits of these new tools are, and equally, how seriously the community is engaging with their potential to be used to the detriment of society. Politics, Pedagogy and Practices cannot, of course, be neatly split into separate analytical threads: they are mutually constitutive. Building on the critical analyses of information infrastructures (Bowker & Star, 1999; Star & Ruhleder, 1996) and knowledge infrastructures (Edwards, 2010; Edwards et al., 2013), we may now be seeing the emergence of “educational knowledge infrastructures” (Buckingham Shum, 2018)—these only exert political power through practices (in policy and design), which translate educational worldviews (whether implicitly or explicitly recognised: Knight, Buckingham Shum, & Littleton, 2014) into code (eg, in database schemas, algorithms and user interfaces), involving and excluding stakeholders, for instance, through the networks of institutions funding and building such infrastructure (Luckin & Cukurova, 2019; Prinsloo, 2019; Williamson, 2019), and the design methodologies employed to make design decisions (Buckingham Shum, Ferguson, & Martinez-Maldonado, 2019; Richards & Dignum, 2019; Mavrikis, Geraniou, Gutierrez Santos, & Poulovassilis, 2019). In the context of this 50th Anniversary Special Issue of the British Journal of Educational Technology, authors from a range of disciplinary backgrounds and outlooks were challenged to make the state of the art in their fields accessible to a broad audience, and to give glimpses of the road ahead to 2025. The papers are therefore primarily reflective, “big picture” narratives, reviewing and discussing existing literature and case studies, and looking forward to what could, or should, be on the horizon. Together, they provide an eclectic set of lenses for thinking about LA/AIED at a range of scales—from the macroscale of national and international policy and stakeholder networks, to the meso-scale of institutional strategy, down to the micro-scale of how we make cognitive models more intelligible, or design decisions more ethical. These papers thus provide a complement to the BJET ‘sister’ special issue on LA/AIED (Starčič, 2019), which provides a more conventional set of papers reporting new empirical evidence and technical advances. We start with two papers from the perspective of critical data studies and sociotechnical infrastructure analysis. In “Policy networks, performance metrics, and platform markets: Charting the expanding data infrastructure of higher education,” Williamson (2019) recognises that while educational data scientists (such as those in this special issue) are genuine in their pursuit of student/teacher-centred LA/AIED, the politicisation and commercialisation of national and international scale data infrastructure cannot be ignored. Drawing on a UK programme currently underway, Williamson highlights numerous concerns, such as how LA/AIED technology may in fact be appropriated by other agendas, and translated into performance metrics. Consequently, he calls for policy frameworks for ethical, pedagogically valuable uses of student data in higher education. In “A social cartography of analytics in education as performative politics,” Prinsloo (2019) critiques “the data imaginary” in higher education, arguing that this narrative ascribes too much trust and power in the potential of data analysis to represent the complex reality of student progress. Following an analysis of the main assumptions in evidence-based management, and the increasing power of quantitative metrics, he presents a “social cartography” of data analytics, which should be thought of “not only as representational, but as actant, and as performative politics.” The changing ecosystem of stakeholders who constitute LA/AIED continues as the focus in “Designing Educational Technologies in the Age of AI: A Learning Sciences Driven Approach,” by Luckin and Cukurova (2019). Educational technology companies are interpreting ideas that have been in research labs for decades, and powered by big data and data science, are taking products to market. This is both exciting and dangerous, since as the authors put it, “most commercial AI developers know little about Learning Sciences research, indeed they often know little about learning or teaching.” Luckin and Cukurova present three examples illustrating how the learning sciences assist in making sense of rich learner data traces, before discussing models for the more effective sharing of such academic knowledge in forms that can be applied by LA/AIED companies, by forging inter-stakeholder partnerships between developers, educators and researchers. LA/AIED is now in commercial products and robust open source platforms, making them ready for adoption at scale—but what does this process look like? The innovation capacity of individual higher education institutions comes into focus in “Complexity leadership in learning analytics: Drivers, challenges, and opportunities,” by Tsai, Poquet, Gašević, Dawson, and Pardo (2019). They document the organisational enablers and obstacles to sustained adoption, and the forms of leadership required to navigate such organisational change. Drawing on their analysis of 21 universities’ adoption processes, and 23 leadership interviews, they distil some common, critical challenges, in the light of which they argue that leadership models based on complexity theory provide key principles to accelerate sustainable LA. Data ethics run throughout this special issue, but the next paper makes this the central focus. In “Practical ethics for building learning analytics,” Kitto and Knight (2019) identify several tensions (illustrated by “edge cases”) in current thinking about ethics in relation to LA/AIED, focussing in particular on how current frameworks fail to provide practical guidance for the teams who design and build such infrastructure. They propose an open database of such edge cases, as “a middle space where technical builders of systems could more deeply interface with those concerned with policy, law and ethics.” Attention then switches to the new societal challenge of learning for a lifetime, and its implications for how LA/AIED is conceived. In “From data to personal user models for life-long, life-wide learners,” Kay and Kummerfeld (2019) map out new challenges for the AIED community, specifically, how the established concept of “the learner model” (what the AI believes the learner understands) must evolve. They identify a set of “competency questions” to which learners should be able to get intelligible answers, if they have genuine control over their learning data, and be equipped to use it to scaffold metacognitive processes. One of the most visible forms of AI that the public now engages with are software agents, for instance, automating telephone and textchat helplines, and enabling voice interfaces to smartphones and other devices. In “Supporting and challenging learners through pedagogical agents who know their learner: Addressing ethical issues through designing for values,” Richards and Dignum (2019) explain what is currently known about the effective use of pedagogical agents (PAs) in learning environments. They describe the different forms these agents can take, what roles they play, the underlying theories, and how they may develop in the future. Richards and Dignum then weave ethical considerations into this picture, arguing for a “design for values” approach to the design of ethically and socially responsible agents. As we introduced above, there is a belief, held in certain scholarly communities and some of the teaching profession, that LA/AIED equates to intelligent tutoring systems (ITSs), delivered through a narrow, instructivist pedagogical model focussed on skills mastery through “drill and practice.” ITS (often branded “adaptive learning”) may also be seen as particularly threatening to the teaching profession, since the value proposition is to more efficiently handle the task of guiding the learner to master the curriculum with 24/7 coaching feedback. Tackling these concerns heads on in “Escape from the Skinner Box: The case for contemporary intelligent learning environments,” du Boulay (2019) first summarises key evidence from 40 years' ITS research. He turns next to examples of more diverse pedagogies that operate at the “screen level” (the capabilities of an ITS), and the “classroom level” (how teachers can orchestrate the students’ ITS experience in different ways). The agency of both students and teachers in these scenarios is greater than the reductionist image that critics paint. The question of the pedagogical assumptions underpinning LA/AIED continues in “Intelligent Analysis and Data Visualisation for Teacher Assistance tools: The case of exploratory learning,” by Mavrikis et al. (2019). Tackling together the concerns around pedagogy, and the automation of teachers, the authors describe how LA can be designed to augment teachers' awareness of their students' progress in exploratory learning tasks. They describe the design rationale behind a suite of “Teacher Assistance tools,” and their empirical evaluation in use. The ITS theme continues in part in a third paper, with a critical reflection on how the AIED community makes its models more transparent, and the criteria used to judge the quality of intelligent systems. In “Explanatory learner models: Why machine learning (alone) is not the answer,” Rosé et al. (2019) argue for “explanatory learner models” that provide more interpretable, actionable output than “black box” models, which may be inscrutable even to their own developers. They present examples of how their systems accomplish this with different kinds of data, across mathematics and writing courses, in order to improve the learner model, generate actionable feedback, and critically, keep the human “in the loop,” rather than automate them out. Distinctively, the authors describe how other disciplines (social sciences, education and design) have contributed important insights and algorithmic improvements, by evaluating AIED in authentic contexts. They conclude that LA/AIED tools should be designed and evaluated with such interdisciplinary skillsets, to understand the human dimensions of use that more technology-centric approaches miss. We explicitly called for insider accounts from system builders regarding the assumptions they make about “politics, pedagogy and practices.” In “The heart of educational data infrastructures = conscious humanity and scientific responsibility, not infinite data and limitless experimentation,” Johanes and Thille (2019) describe not their own system building efforts, but the insights they gained from in-depth interviews with 11 data infrastructure teams building LA/AIED systems. The authors seek to give these teams a voice, to rebalance concerns that technologists do not care about ethics, or even if they do, they cannot translate this into design decisions. Their goal is that “researchers, policymakers, and infrastructure builders can use these accounts to better understand the building process and experience.” They document how infrastructure developers reflect deeply on, and ultimately make decisions about, the social and ethical dimensions of their creations. We invited papers that would provide critical accounts of the practices employed in the design of LA/AIED, and of the practices adopted by educators deploying these tools (Kay & Kummerfeld, 2019; Luckin & Cukurova, 2019; Richards & Dignum, 2019; Rosé et al., 2019). With respect to pedagogy, we anticipated critical narratives that explain how the use of quantification does not need to equate to the implementation of behaviourist or instructivist approaches to teaching and learning (du Boulay, 2019; Mavrikis et al., 2019). Finally, within the theme of politics, our goal was to garner accounts that recognise the political significance and the potential for power and influence that the intelligence infrastructure constructed through the use of big data, AI and analytics brings to the world. Accounts that would help us to be appropriately vigilant and judiciously embracing with respect to these technologies and the quantifiable approaches they can bring to education (Kitto & Knight, 2019; Prinsloo, 2019; Richards & Dignum, 2019; Tsai et al., 2019; Williamson, 2019). We were also keen for clarifications of the that the three of our Politics, Pedagogy and Practices are deeply and Johanes & 2019; Kitto & Knight, 2019). The making theme was with who the that need to be in to concerns about the use of data to control and and the need to the we approach ethics, issues of We need to to understand data, and to be to the for policy pedagogical and order to the need for data infrastructures that tension between on the and on the the quantification of through metrics 2019). between stakeholders is to as we into educational we should the fact that higher education institutions are complex as clear by Tsai et al. (2019). Prinsloo (2019) the need to to making visible and the and context that all that we The key to be by stakeholders, within education as as within also for Mavrikis et al. Their paper the in which teachers can be by information by and how teachers need to be part of the it comes to the design and implementation of data approaches within education. however, all these political concerns need to be by the of pedagogy that we must put in to the benefits that analytics and AI can As we through the and the we need to about It is important that we question the behind all the about and policy, and that they are by what we know and understand about we do not this then we may make decisions that and du Boulay (2019) challenges the in the for papers for this special issue analytics and AI equate to a pedagogy from the Boulay and the in pedagogy that can be in learning that educational could from the and into their through the we and their learning It be important that new technologies rather than the of teachers, a by Mavrikis et al. (2019). A particularly valuable of the approach we authors to can be seen in the that the papers in this issue build on much valuable research. Kay and Kummerfeld on from AI in Education with research on to the of a for and a set of design with the of learners to develop their metacognitive skills as as control of their In they focus on the need for interfaces that reflection and data and The is but together research AI and in of their We see a for in the of Rosé et al., who the need for the criteria of the machine learning to be with those of authentic in education also Buckingham Shum, & it comes to the practices of teaching and learning with analytics and and the practices in building analytics and AI then we certainly need to that we are in our We need the educators to be with the developers and and this narrative must to policy this the paper by Richards and Dignum about the societal impact and ethical issues that must be into designing The authors that we do not yet understand the implications of in human roles for both teachers and and as lifelong They that both and policy need to be of both the benefits and with the use of AI in and present a design approach in the of learner They the challenge of of learning between education and the and the potential that and reality could bring to help this They make a case for the value of but they also recognise the need for and tools to responsible Johanes and Thille also focus on the need for between different They that those who build data and analytics technology only data as and ready for rather than as products of social processes. They the from their interviews with system builders in higher education, to the which now been by those the The of the infrastructure builders are and the that makes clear that those who the builders the fact that system builders are engaging deeply with what the critics The fact that technologies currently little of human is that these system builders but this is that critics do not always Johanes and Thille us that we must not that the technologies by system builders are the technologies that the system builders would to be they are the that can be with the current of the tools technical and Their is powerful and particularly to a special issue such as where we have to and is that the ethical and personal that the with these concerns have not an in existing In the context of to and Mavrikis & 2019), we recognise the lack of in this special issue, and must for invited the authors on the of their experience and of rather than This however, a reflection of the state of the art in With the of paper from the is from and of the papers focus explicitly on how LA/AIED and The they have and is all from the world. All but 11 in the papers have first authors is in the We cannot on analytics and AI for the a and as a community we must seek to the of our The of the are different to those of the of the world. we can a from the in which this reality been and the are and those in the as our was by the ideas and the that between and this we that LA/AIED be by and influence are technology infrastructure and are and in LA/AIED is greater than of the where our authors and our data are or and are the most in the and the in the and yet no authors or data in this issue from these both however, and can be educational infrastructure it should be no to see politics, pedagogies and as an we see and commercial in AIED that the of the world. as Knight et al. learning analytics are our technology is always to advance the of teaching and and the of of the is much than that of the LA/AIED is at present by companies tutoring for skills often in for conventional (eg, & 2018; & 2019). while are analysis in the the scale of in & 2018; & a of politics, pedagogies and practices that would it is only a of before LA/AIED to other pedagogies (eg, see the examples in 2019), concerns around and scientific always be In is a we all see and LA/AIED must recognise and value this as we into an future. we are to develop the Politics, Pedagogy and Practices that can us through the next we must to design that are and human These are not yet as visible as they could be in the LA/AIED community, but they are those that us from