Managing Data-Driven Development: An Ethnography of Developing Machine Learning for Recruitment

With the increased pervasiveness of Machine Learning (ML) applications in real-world settings, such as healthcare, criminal justice, and recruitment, scholars have started to call for more attention to how ML systems are actually developed in practice. This study draws on a 23-month ethnography of a scale-up vendor that develops a recruiting application powered by ML and neuroscience to support clients in making hiring decisions. Our in-depth analysis reveals that the development of a real-world ML system is different from conventional information system development (ISD), in that it calls for a shift from eliciting domain expertise to feeding the algorithm, from coding domain expertise to training the algorithm, and from adjusting domain expertise to growing the algorithm. While much of the existing scholarship on ISD has focused on interacting with domain experts in order to transfer their knowledge into the system, we demonstrate that data-driven development, required for ML, calls for managing a triangular relationship between developers, clients (“domain experts”), and the algorithm. The paper contributes to the literature on (data-driven) ISD, work on the construction of algorithmic technologies in practice, and responsible artificial intelligence (AI).