Variational Tensor Neural Networks for Deep Learning

Abstract Deep neural networks (NNs) encounter scalability limitations when confronted with a vast array of neurons, thereby constraining their achievable network depth. To address this challenge, we propose an integration of tensor networks (TN) into NN frameworks, combined with a variational DMRG-inspired training technique. This inturn, results in a scalable tensor neural network (TNN) architecture capable of efficient training over a large parameter space. Our variational algorithm utilizes a local gradient-descent technique, enabling manual or automatic computation of tensor gradients, facilitating design of hybrid TNN models with combined dense and tensor layers. Our training algorithm sheds light on the entanglement structure of the tensorized trainable weights, while also illuminating the expressive power of the TNN as a quantum neural state. We validate the accuracy and efficiency of our method by designing TNN models for regression and classification tasks on diverse datasets. Furthermore, we delve into the expressive power of our algorithm, drawing upon the entanglement structure of the neural network.