Beyond ReLU and GELU: SoftCap Bounded Activations for Stability and Sparsity

We introduce the SoftCap family, bounded rectifying activations derived from explicit continuity and sparsity constraints rather than empirical search @ramachandran2017searching. The family comprises SoftCap (C⁰), SwishCap (C¹), and SparseCap (C²), all sharing a bounded positive branch a (x) with analytically derived, variance-preserving scalar a^* @glorot2010understanding; @he2015delving; @klambauer2017selu. In high-learning-rate grokking stress tests, SwishCap achieves 100% survival across all tested rates, whereas hard-zero variants exhibit sharp collapse boundaries, indicating that origin smoothness and negative-side gradient transport govern stability more strongly than boundedness alone @power2022grokking; @balduzzi2017shattered. Applied after Q/K projections in Muon-trained ViTs, bounded activations reduce peak pre-softmax attention scores by 3–4×, reducing reliance on explicit clamping @vaswani2017attention; @dosovitskiy2021vit. Under heavy-tailed contamination, they suppress outlier logit gaps by over two orders of magnitude, imposing an architectural confidence ceiling without explicit calibration @ovadia2019can; @guo2017calibration. While trailing ReLU/GELU by 4 pp in standard supervised regimes @nair2010relu; @hendrycks2016gelu, these results establish a constrained design map in which continuity order and notch geometry determine predictable trade-offs across stability, sparsity, and dynamic-range control.