From suspicious results to insights: a study on debugging practices in scientific data analysis workflows

Scientific progress more and more depends on computational analysis, performed by increasingly complex data analysis workflows. When these workflows inevitably show unexpected behavior or even fail, scientists need to establish a cause-effect relationship between the observable problem and the actual defect. So-called outcome-preserving input reduction was recently proposed as a semi-automated technique to support this debugging activity. However, its adoption in practice depends on the assumptions that (1) reducing input data to a minimal subset will sharpen debugging focus and simplify problem-solving, and that (2) users can accurately formulate debugging queries based on suspicious output. To get a better picture of how scientists currently approach debugging in general and to assess the validity of the assumptions above in particular, we performed an expert survey in which we confronted participants with an example case showcasing a common data analysis problem. We found that most participants rely on inspecting intermediate results when debugging. About half of them propose to use data exploration strategies, while only a few mentioned using smaller input data sets. However, more than 70% of participants were able to adequately describe the suspicious result and, given a minimal reproducible input data set, about 60% correctly or partially identified the values causing the problem and found the reduced data set helpful for the sake of debugging. Results were consistent across participants from both computer science and natural science backgrounds.