Answers are grounded in this paper and cite their sources.
Large Language Models (LLMs) are increasingly used to design optimization algorithms, in a paradigm referred to as LLM-driven Automated Algorithm Design (LLM4AAD). Such a shift challenges established benchmarking practice in optimization, which was built around evaluating a single fixed algorithm. In contrast, LLM4AAD systems produce a generated algorithm through a stochastic, LLM-driven generation process, and both the generation and the generated algorithms deserve attention in benchmarking. This position paper adapts and extends established benchmarking principles from evolutionary optimization to the specifics of LLM-driven algorithm design. We formalize the algorithm design problem, distinguishing the generated algorithm from the generation process that produces it, and argue that the two require distinct benchmarking targets and metrics. Using two representative examples, we systematically revisit the benchmarking goals established for classical meta-heuristic optimization, assess the current maturity of LLM4AAD practices against each, and introduce a further set of goals specific to the use of LLMs as the generative component. We further examine the reproducibility threats posed by stochastic, evolving, and often closed-source LLM backends, and outline minimum reporting practices to mitigate them. We conclude with a research agenda identifying the community infrastructure and evaluation protocols needed to establish shared benchmarking standards for the field.
No takes yet. Share an insight, caveat, or question.
Stein et al. (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: