The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and shareable structure, it is difficult for the community to contribute, update and extend existing benchmarking studies to ensure long-term relevancy. To address this challenge, we present BenchHub, a community-oriented ecosystem with a modular R6-based structure that enables living benchmarking. BenchHub comprises three key components: (i) a Trio database that links datasets, performance metrics, and supporting evidence (e.g. ground truth), (ii) a BenchmarkStudy structure that captures the different benchmark study designs, and (iii) a series of tools together with vignettes and interactive platform that allow users to gain insights from the benchmarking results. Together, these components streamline the benchmarking process for benchmark study developers, methods contributors, and benchmark consumers, promoting reproducibility, comparability, and long-term sustainability in computational biology.
Yang et al. (Tue,) studied this question.