Supervised Random Feature Regression via Projection Pursuit

Random Feature (RF) methods and Neural Networks (NNs) are two dominant paradigms in nonparametric modeling. RF methods offer computational efficiency but are often limited by their reliance on fixed, data-independent feature representations. NNs achieve strong expressive power through deep architectures, yet at the cost of greater computational complexity, reduced interpretability, and sensitivity to hyperparameter choices—particularly in limited-data settings. This paper proposes a supervised random feature framework based on projection pursuit that bridges these two paradigms. The core estimator, SRF-I, replaces the fixed activation function in classical RF with data-adaptive univariate basis functions learned along random projections in a supervised manner. Building on SRF-I, we develop SRF-II, a blockwise hierarchical extension that aggregates block-level predictions through a projection pursuit regression model, enhancing both scalability and modeling flexibility. We establish an excess risk bound for SRF-I that characterizes the bias–variance trade-off and guarantees consistency under mild conditions. Extensive simulations and real-data experiments demonstrate that the proposed framework combines the computational advantages of RF methods with the expressive power of neural networks, achieving robust performance in limited-data regimes without extensive hyperparameter tuning.