How Scrum can be used for Data Science projects

What’s it about?

In this whitepaper, you will get an overview of the project management technique Scrum with regard to the typical process of a data science project. Based on this, we present the biggest challenges and propose solutions for the integration of Scrum in your data science projects.

In recent years, data science projects have been subject to significant change. From the development of simple Proof of Concepts (PoC), the focus is shifting towards full software applications, focusing on operationalization and industrialization of the solution. Above all, the topic of “agile working“ is becoming increasingly important in the project context of data science and AI. In particular, project management according to the Scrum method has become best practice. To successfully apply Scrum to Data science projects, in addition to the implementation of the central concepts of Scrum — regular communication, completing tasks in small packages, not losing sight of the overall goal, constant adjustment, and improvement of the work situation — some adjustments of the processes should be made.

Since data scientists in the past enjoyed a high degree of freedom and creativity in the context of project implementation, the strict procedures and rigorous time constraints required by Scrum often meet with contradiction and displeasure (Dr. Saltz & Hotz, 2020). In contrast, project managers often cannot understand why the Scrum approach is not immediately accepted by the team, even though it has been common practice in software development for years. How can these problems be solved and how can different views be aligned? Can Scrum be the right approach for data science projects? If so, do the processes have to be adapted or does the data scientists‘ attitude have to change? In this whitepaper, we present solutions for a successfull application of Scrum to data science projects.