Saturated linear unit as an universal symmetric activation function for deep learning

There is a number of symmetric activation functions used in artificial neural networks for deep learning. In this paper, we propose a universal activation function based on the Saturated Linear Unit (SaLU) that can be used instead of any known symmetric activation function. It is not necessary for classification tasks to have an exact calculation of the probability of detected classes. The classification decision is made based on the highest probability for the input values. We propose, as a proof of concept, that the two most commonly used hyperbolic tangent and algebraic sigmoid activation functions can be effectively replaced by SaLU by choosing a single parameter. Moreover, the theoretical step function can also be replaced by SaLU for a wider transition range. All derivations use symbolic processing. Also shown is a visualization of the range of inputs that result in a suitable classification. This can help scientists and programmers design complex machine learning algorithms and understand how deep learning algorithms work.