Reuse and reproduce bioinformatic pipelines using scientific workflow systems

Data analysis pipelines have emerged as a well-established and efficient approach for defining and executing bioinformatics data analysis and experiments. Scripting languages, such as Python and R, with potential help of notebooks, are widely utilised and beneficial for constructing small-scale pipelines, conducting final analyses, and creating visualisations for individual users. However, it is increasingly acknowledged that they fall short in enabling the development of large-scale, shareable, maintainable, and reusable pipelines capable of processing extensive datasets and operating on high-performance computing clusters. This chapter first provides an overview of the essential features involved in constructing large-scale data pipelines. It highlights the user needs to be considered when building such pipelines. It then explores the existing solutions available to meet these requirements. Subsequently, the advantages of employing scientific workflow systems to achieve modularity, reproducibility, and reusability in bioinformatics data analysis pipelines is strongly emphasised. Lastly, a conclusion is drawn and current challenges are discussed.