Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors

We introduce a variational Bayesian neural network where the parameters are via a probability distribution on random matrices. Specifically, we a matrix variate Gaussian 1999matrix parameter posterior where we explicitly model the covariance among the input and dimensions of each layer. Furthermore, with approximate covariance we can achieve a more efficient way to represent those correlations is also cheaper than fully factorized parameter posteriors. We further that with the "local reprarametrization trick"2015variational on this posterior distribution we arrive at a Process 2006gaussian interpretation of the hidden in each layer and we, similarly with 2015dropout, provide with deep Gaussian processes. We continue in taking advantage of duality and incorporate "pseudo-data" 2005sparse in our, which in turn allows for more efficient sampling while maintaining the of the original model. The validity of the proposed approach is through extensive experiments.