Deep reinforcement learning for multi‐restart metaheuristics: an environment design for a hybrid of unbiased exploratory search and covariance matrix adaptation evolution strategy

Multi‐restart metaheuristics can be highly effective for complex optimization problems, yet their performance depends critically on how restarts and algorithmic parameters are selected. This paper introduces a reinforcement learning approach for managing restart‐level decisions and parameter configurations in the UES–CMA‐ES hybrid metaheuristic. Rather than modifying the underlying optimizer, we focus on the design of the reinforcement learning environment, formulating restarts as high‐level decision points governed by a learned policy. We propose an environment that anchors restarts at the final solution of each execution and jointly controls key metaheuristic parameters, guided by a stagnation‐aware reward function. Experiments on the IEEE CEC'13 benchmark suite show that the proposed approach achieves consistent performance improvements over static multi‐restart strategies and simpler environment formulations, with average gains of up to 15–30% on multimodal and composition functions. A single agent trained on 30‐dimensional problems generalizes effectively to 100‐dimensional benchmarks without retraining. These results highlight the central role of environment design and reward shaping in reinforcement learning–enhanced optimization and suggest a principled pathway toward data‐driven metaheuristics that adaptively refine multi‐restart strategies.