In this work, we propose a novel activation function, AHReLU, which demonstrates superior performance compared to traditional activation functions such as ReLU, GELU, and Mish. Replacing ReLU with AHReLU in the VGG-16 model improved Top-1 classification accuracy by 0.79%, reaching97.57%. In the CIFAR-100 dataset, AHReLU outperformed ReLU by 0.32%, achieving a Top-1 accuracy of 59.82%. For the SVHN dataset, AHReLU achieved a mean accuracy of 95.38%, slightly surpassing ReLU’s performance of 95.36%. In machine translation tasks, specifically on the WMT 2014 dataset, AHReLU achieved a BLEU score of 27.5, which is 0.2 points higher than ReLU and 1.2 points higher than GELU. These results highlight that AHReLU, with its learnable parameters, outperforms traditional activation functions, leading to better model performance across various datasets and tasks. The introduction of learnable parameters into the activation function is key to the observed improvements, making AHReLU a promising candidate to replace widely used activation functions such as ReLU, GELU, and Mish in deep learning models.
Ullah et al. (Thu,) studied this question.