Importance estimation of hyperparameters in reinforcement learning

Hyperparameters play a critical role in enabling reinforcement learning (RL) agents to achieve high performance, yet their optimization remains computationally demanding. To address this challenge, we evaluate methodologies from the field of sensitivity analysis (SA) for assessing hyperparameter importance, thereby enabling more informed resource allocation through (I) hyperparameter prioritization, (II) hyperparameter fixation, and (III) hyperparameter mapping across the value space. Following a theoretical analysis of RL-specific characteristics and available methodologies from the field of SA, we identify functional ANOVA (fANOVA) as the most promising candidate. Our empirical investigation evaluates the validity, reliability, sample efficiency, and usability of fANOVA within the RL context. The results demonstrate its overall effectiveness in achieving goals I–III, while also highlighting limitations related to the sample efficiency and especially regarding the usage of data generated during hyperparameter optimization.