TRex: A Smooth Nonlinear Activation Bridging Tanh and ReLU for Stable Deep Learning

Activation functions are fundamental to the representational capacity and optimization dynamics of deep neural networks. Although numerous nonlinearities have been proposed, ranging from classical sigmoid and tanh to modern smooth and trainable functions, no single activation is universally optimal, as each involves trade-offs among gradient flow, stability, computational cost, and expressiveness. This study introduces TRex, a novel activation function that combines the efficiency and linear growth of rectified units with the smooth gradient propagation of saturating functions. TRex features a non-zero, smoothed negative region inspired by tanh while maintaining near-linear behavior for positive inputs, preserving gradients and reducing neuron inactivation. We evaluate TRex against five widely used activation functions (ReLU, ELU, Swish, Mish, and GELU) across eight convolutional architectures (AlexNet, DenseNet-121, EfficientNet-B0, GoogLeNet, LeNet, MobileNet-V2, ResNet-18, and VGGNet) on two benchmark datasets (MNIST and Fashion-MNIST) and a real-world medical imaging dataset (SkinCancer). The results show that TRex achieves competitive accuracy, AUC, and convergence stability across most deep, connectivity-rich architectures while maintaining computational efficiency comparable to those of other smooth activations. These findings highlight TRex as a contextually efficient activation function that enhances gradient flow, generalization, and training stability, particularly in deeper or densely connected architectures, while offering comparable performance in lightweight and mobile-optimized models.