Deep Neural Networks with Spectral Activation Functions: Applications to Numerical PDEs

We introduce Orthogonal Polynomial-based Residual Networks (OPResNets), a novel neural architecture that synergistically integrates spectral activation functions (SAFs) with residual learning. Our key innovation is the construction of hierarchical spectral activations using orthogonal polynomial bases, which enables precise control over frequency distributions and facilitates simultaneous multi-scale feature extraction. Comprehensive experiments demonstrate that OPResNets achieve superior performance in balancing accuracy and computational efficiency. Notably, the architecture significantly outperforms conventional activation functions in high-frequency function approximation tasks. Building upon these advantages for high-frequency problems, we further propose a multi-stage OPResNet variant that achieves enhanced accuracy in numerical partial differential equation (PDE) solutions. Extensive numerical experiments demonstrate that both OPResNets and their multi-stage variants achieve superior approximation accuracy in high-frequency regimes compared to conventional architectures.