Answers are grounded in this paper and cite their sources.
This article studies the Gram random matrix model G=1/TΣ^TΣ, Σ=σ(WX), classically found in the analysis of random feature maps and random neural networks, where X=[x₁,…,xT]p× T is a (data) matrix of bounded norm, Wn× p is a matrix of independent zero-mean unit variance entries and σ:R is a Lipschitz continuous (activation) function—σ(WX) being understood entry-wise. By means of a key concentration of measure lemma arising from nonasymptotic random matrix arguments, we prove that, as $n,p,T$ grow large at the same rate, the resolvent Q=(G+γ IT)⁻¹, for γ>0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment Φ=T/nE[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.
No takes yet. Share an insight, caveat, or question.
Louart et al. (2018) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: