Hypergeometric Functions as Activation Functions: The Particular Case of Bessel-Type Functions

The choice of the activation functions in neural networks (NN) are of paramount importance in the training process and the performance of NNs. Therefore, the machine learning community has directed its attention to the development of computationally efficient activation functions. In this paper we introduce a new family of activation functions based on the hypergeometric functions. These functions have trainable parameters, and therefore after the training process, the NN will end up with different activation functions. To the best of our knowledge, this work is the first attempt to consider hypergeometric functions as activation functions in NNs. Special attention is given to the Bessel functions of the first kind Jν, which is a sub-family of the general family of hypergeometric functions. The new (Bessel-type) activation functions are implemented on different benchmark data sets and compared to the widely adopted ReLU activation function. The results demonstrate that the Bessel activation functions outperform the ReLU activation functions in both accuracy aspects and computational time.