Assisted design of end-to-end data science pipelines

When designing data science pipelines, end-users with varying expertise — data analysts, domain experts, and novices — get overwhelmed by the large and growing number of design decisions they have to make. The lack of explicit guidelines on how to apply data preprocessing and modeling algorithms, both separately and as part of an end-to-end workflow, leads to error-prone ad-hoc experimentation, significant manual overhead to the end-user, and performance degradation in deployed applications. In this thesis, we propose a generic, human-in-the-loop framework for the assisted design of end-to-end data science pipelines. We then demonstrate the framework on three subproblems that mainstream automated machine learning (AutoML) and Intelligent Discovery Assistants (IDAs) do not address: validating the predictions of black-box machine learning models on unseen data, validating data quality for dynamic data ingestion, and documenting end-to-end data science experiments for reproducibility and root-cause analysis. We outline the challenges that make each subproblem particularly resource-intensive and overwhelming to the end-user. For each, we propose a semi-automated solution that requires little to no manual overhead. As opposed to the existing work, our framework provides real-time recommendations to design effective data science pipelines tailored to the needs of the end-user. It is capable of extending to new data science tasks, operators, and evaluation procedures and improves over time due to an ever-expanding store of previously executed pipelines. We demonstrate our framework and evaluate the quality of real-time recommendations under the ever-growing set of supported data science operators, tasks, and evaluation processes. Furthermore, we articulate how end-users can extend the framework to their needs.