Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

We introduce a variational Bayesian neural network where the parameters are via a probability distribution on random matrices. Specifically, we a matrix variate Gaussian \{gupta1999matrix} parameter posterior where we explicitly model the covariance among the input and dimensions of each layer. Furthermore, with approximate covariance we can achieve a more efficient way to represent those correlations is also cheaper than fully factorized parameter posteriors. We further that with the "local reprarametrization trick"\{kingma2015variational} on this posterior distribution we arrive at a Process \{rasmussen2006gaussian} interpretation of the hidden in each layer and we, similarly with \{gal2015dropout}, provide with deep Gaussian processes. We continue in taking advantage of duality and incorporate "pseudo-data" \{snelson2005sparse} in our, which in turn allows for more efficient sampling while maintaining the of the original model. The validity of the proposed approach is through extensive experiments.