A Novel Adaptive Activation Function for Convolutional Neural Networks: The Parametric Arctangent Unit (PATU)

The activation function is a critical component in convolutional neural networks that perform nonlinear mappings between inputs and outputs. The rectified linear unit (ReLU) function is currently the baseline activation function in the deep learning community; however, it has several drawbacks that result in inefficient training, such as how the zero value of a negative part treats negative inputs as insignificant information for learning, which results in bias shift in network layers, and how the multilinear structure of ReLU restricts the nonlinear approximation power of networks. To address these limitations, this study proposes the parametric arctangent unit (PATU), which integrates advantageous features of ReLU, (parametric ReLU) PReLU, (exponential linear unit) ELU and Swish. Unlike the PReLU, which employs a learnable linear slope for negative inputs and thus remains piecewise linear, PATU integrates a nonlinear arctangent activation in the negative part. Specifically, PATU retains ReLU’s advantageous positive part, while the negative part employs arctangent to activate negative inputs, and a trainable parameter is implemented to control the saturation of the negative part. The study compared the proposed PATU to four state-of-art activation functions, namely ReLU, PReLU, ELU and Swish, on the CIFAR10 and CIFAR100 datasets using SmallNet, Network in Network, and ResNet18 convolutional neural network architectures. Experimental results demonstrate that PATU outperformed ReLU in terms of accuracy on the CIFAR10 and CIFAR100 datasets by 0.45% and 1.44%, respectively, for the SmallNet architecture; 1.87% and 3.11%, respectively, for the NIN architecture; and 1.19% and 2.92%, respectively, for the ResNet18 architecture. All experiments were repeated five times, and the improvements were statistically significant (Friedman test, p = 0.015). PATU reached the highest mean rank score of 1.16 in the Friedman test for classification accuracy, which is superior to state-of-the-art methods. These results suggest that the proposed PATU is an effective alternative to ReLU.