Applying Scrum in Data Science Projects

The rise of big data has led to an increase in data science projects conducted by organizations. Such projects aim to create valuable insights by improving decision making or enhancing an organization's service offering through data-driven services. However, the majority of data science projects still fail to deliver the expected value. To increase the success rate of projects, the use of process models or methodologies is recommended in the literature. Nevertheless, organizations are hardly using them because they are considered too rigid and they do not support the typical iterative and open nature of data science projects. To overcome this problem, this research suggests applying Agile methodologies to data science projects. Agile methodologies were originally developed in the software engineering domain and are characterised by their iterative approach towards software development. In this study, we selected the Scrum approach and integrated it into the CRISP-DM methodology for data science projects using a Design Science Research approach. This new methodology was then evaluated in three different case organizations using expert interviews. Analysis of the expert interviews resulted in a further refinement of the Agile data science methodology proposed by this research.