Tensorizing Neural Networks

Deep neural networks currently demonstrate state-of-the-art performance in domains. At the same time, models of this class are very demanding in of computational resources. In particular, a large amount of memory is by commonly used fully-connected layers, making it hard to use the on low-end devices and stopping the further increase of the model size. this paper we convert the dense weight matrices of the fully-connected to the Tensor Train format such that the number of parameters is reduced a huge factor and at the same time the expressive power of the layer is. In particular, for the Very Deep VGG networks we report the factor of the dense weight matrix of a fully-connected layer up to200000 times leading to the compression factor of the whole network up to 7.