Benchmarking Deep Learning Optimizers for the classification of Event-Related Potentials based Brain-Computer Interface EEG data

The lack of theoretical convergence guarantees for gradient-based deep learning optimizers across diverse classification and generation tasks leaves the choice of the optimizer to arbitrary considerations. Developing new optimizers requires heavy empirical benchmarking; moreover, their performance improvements are observed on large datasets and their generalizations to small datasets are underexplored. In this work, we test five recent Adaptive gradient-based optimizers: AMSGrad, AdamW, Lion, Aida, and Ranger21 on Electroencephalography (EEG) based Event-Related Potentials (ERP) data for Brain-Computer Interfaces (BCIs). The optimizers are tested on different settings of batch size and epochs to assess their robustness and gain insights into the optimal settings. Except for Ranger21, the optimizers showed stable performances slightly outperforming or on par with AdamW, with Aida showing the best performances on average, even with large batch size. These findings are a step forward in facilitating the choice of the optimizer for reducing the Deep Neural Network model training time.