Answers are grounded in this paper and cite their sources.
A standardized, elaborate and well-argued performance metric for benchmarking anytime algorithms on noisy optimization problems is missing in the current benchmarking frameworks. This research aims at developing such a metric as an artifact by following the Design Science Research methodology. Existing noise handling practices and benchmarking guidelines are reviewed. Benchmarking experiments, including a test suite containing 13 artificial test functions, were conducted with two state-of-the-art algorithms to assess the performance metric. Based on the experimental results and the existing literature, this study concludes that, rather than relying on a single quantitative value, algorithm performance should be assessed through a comprehensive performance report comprising the numerical loss value, the number of function evaluations, wall-clock runtime, and a performance profile illustrating the trajectory of loss values over time. This approach aims to maximize the reliability, expressiveness, and informational value of performance assessments.
No takes yet. Share an insight, caveat, or question.
Boeh et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: