Beyond ReLU and GELU: SoftCap Bounded Activations for Stability, Sparsity, and Quantization

We introduce a constraints-first activation design framework: fix a shared bounded positive branch a (x) with (variance-preserving) scale parameter a and vary only the negative side, yielding the SoftCap family: a C⁰–C² progression of bounded rectifiers including SoftCap (exact-zero), SwishCap (C¹ derivative-matched), and SparseCap (C² quintic notch). These activations are derived in closed form, avoiding benchmark-driven empirical search @ramachandran2017searching. In high-learning-rate stress tests, SwishCap achieves 100% survival, demonstrating that negative-branch gradient flow dictates stability @power2022grokking; @balduzzi2017shattered. In transformers, applying bounded activations to query/key projections reduces peak attention scores by 3–4×, expands the high-learning-rate operating window, and reduces reliance on explicit clamping @vaswani2017attention; @dosovitskiy2021vit. Structurally, SparseCap natively routes 8. 9% of queries to exactly zero, establishing an architectural foundation for attention sparsity without post-hoc thresholding. Beyond attention, fully bounded networks inherently suppress post-activation outliers, substantially reducing performance degradation under INT8 quantization compared to unbounded baselines. Finally, under heavy-tailed contamination, bounded variants suppress outlier logit gaps by ~40–85×, imposing a strict confidence ceiling without explicit calibration @ovadia2019can; @guo2017calibration. Together, these results establish a constrained design map in which continuity order and notch geometry determine predictable trade-offs across high-learning-rate stability, structural sparsity, and dynamic-range control.