An Improved Taylor Hyperbolic Tangent and Sigmoid Activations for Avoiding Vanishing Gradients in Recurrent Neural Nets

In deep learning, Hyperbolic Tangent (Tanh) and Sigmoid nonlinear activation functions can retain the complex relationship, which is more appropriate in Recurrent Neural Networks (RNNs). The gradients of these activation functions are vital in updating the weights during training the network. However, both functions are vulnerable to the vanishing gradient problem and expensive in exponent operations. It causes gradients to vanish during back propagation that leads to training overheads and low performance. Although most of the studies put forward methods to reduce exponent operations, there is not a viable solution to tackle the gradient issues. Hence, we propose a Taylor expansion of second order to realize Tanh and Sigmoid functions. In particular, Long Short-Term Memory (LSTM) network makes extensive use of these functions as well as gating mechanism to control the flow of information and gradients. In consequence, Taylor expansion Tanh and Sigmoid activation functions based parallel heterogeneous LSTM network integrated with Bayesian hyperparameter optimization is being proposed for multi-step time series prediction. The current model efficacy is evaluated on bench mark datasets Mackey-Glass Series (MGS), Electricity Transformer Temperature hourly 2 (ETTh2), coronavirus daily cumulative cases, Cumulative Deaths (CD-5) and (CD-7), daily New Cases (NC4), and Total Recovery Cases (TRC-8) in India. The model performance is compared with conventional models like the Auto Regressive Integrated Moving Average (ARIMA), Tree-based Pipeline Optimization Tool (TPOT) regressor, LSTM, Gated Recurrent Unit (GRU), transformer, and the proposed model with Tanh and Sigmoid activations. The analysis reveals that the current model achieves remarkable performance in terms of Mean Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Coefficient of Determination (R2 Score) when compared to existing models