The Koala Benchmarks for the Shell: Characterization, Implications, and Extensions

Koala is a benchmark suite for research targeting the Unix and Linux shell. Koala combines a systematic collection of diverse shell programs collected from tasks found in the wild, various real inputs to these programs that facilitate small and large deployments, extensive analysis and characterization aiding their understanding, and additional infrastructure and tooling aimed at usability and reproducibility in systems research. The Koala benchmarks perform a variety of common shell tasks; they combine all major language features of the POSIX shell; they use a variety of POSIX, GNU Coreutils, and third-party components; and they operate on inputs of varying size and composition—available on both permanent archival storage and scalable cloud storage. Applying Koala to six systems concerned with shell execution, ranging from the underlying interpreter to bolt-on incremental execution, offers a broader perspective on their trade-offs, generalizes their key results, and contributes to a better understanding of these systems.