Turtles All the Way Down: Parameter Sensitivity in Metaheuristic Hyperparameter Optimisation for Machine Learning Models

Metaheuristic algorithms, particularly particle swarm optimisation (PSO) and differential evolution (DE), are widely used for hyperparameter optimisation of machine learning models, yet metaheuristics are in turn sensitive to their own control parameters. How much this recursive dependency affects hyperparameter optimisation outcomes in practice has not been established. Four metaheuristics, including the self-adaptive variants self-adaptive differential evolution (jDE) and self-adaptive particle swarm optimisation (SAPSO), are evaluated across ten datasets and four machine learning models, alongside four modern adaptive methods and three parameter-free baselines. Meta-parameter choices are shown to cause accuracy variations of up to 3.49%. Neither the classical self-adaptive variants nor the modern methods reduce the sensitivity. No population-based metaheuristic significantly outperforms a properly configured grid search, and grid search attains a near-best mean rank at roughly an order of magnitude fewer function evaluations. The metaheuristics are the stronger optimisers of the cross-validation objective. The additional gains do not transfer to test data. The pattern is consistent with oversearching. Simpler methods are therefore preferable for small tabular classification with low-dimensional hyperparameter spaces.