pileup-hi: an ultra-high-throughput, customizable alignment pileup program for large datasets

MOTIVATION: Recent advancements in next-generation sequencing have combined massive data throughput with high read accuracy to facilitate large-scale, rapid, and sensitive analysis of genetic variation. High-depth, high-quality platforms such as the Illumina NovaSeq X and Ultima Genomics UG 100 are increasingly used in time-sensitive clinical contexts such as cancer screening to detect mutations as low as 0.01% without sequence error correction. This process usually involves the bioinformatic construction of a pileup, or a list of nucleotides aligned to one or more positions in a sequence alignment, to identify variants. Efficient pileup software is required to process large-scale sequencing datasets rapidly to allow for timely clinical decision-making. While foundational to many analytical pipelines, the de facto standard pileup software samtools mpileup faces scalability challenges with larger datasets and is restricted to one output format. RESULTS: We present pileup-hi, a multi-threaded pileup engine that is scalable to alignments containing billions of reads and extendable to support custom output formats. When set to emit the default mpileup format, pileup-hi is up to 13x faster than samtools mpileup, 3.2x faster than sambamba mpileup, and up to 8x faster than perbase base-depth. The default output of pileup-hi has binary equivalence to the output of samtools mpileup across benchmark files and a variety of samtools regression tests. We present a new pileup-derived format that provides depth-invariant data storage proportional only to the reference genome length and number of unique indels. AVAILABILITY AND IMPLEMENTATION: Pileup-hi is implemented in the Rust programming language and distributed as open-source code and precompiled binaries. Source code and installation instructions can be found at https://github.com/greninger-lab/pileup-hi. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.