We will describe ccwl and ravanan---Common Workflow Language (CWL) tools we have developed---that allow running scientific workflows on HPC clusters with concision, reproducibility and performance. ravanan uses propagator networks to provide high cluster utilization. Intermediate computational steps are run as early as possible without blocking, even when only partial results are available from previous steps. In addition, ravanan provides strong caching and reproducibility guarantees so that computation never has to be repeated.
Arun Isaac (Mon,) studied this question.