On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory

This paper tackles the problem of Lipschitz regularization of Convolutional Networks. Lipschitz regularity is now established as a key property of deep learning with implications in training stability, generalization, against adversarial examples, etc. However, computing the exact of the Lipschitz constant of a neural network is known to be NP-hard. attempts from the literature introduce upper bounds to approximate this that are either efficient but loose or accurate but computationally. In this work, by leveraging the theory of Toeplitz matrices, we a new upper bound for convolutional layers that is both tight and to compute. Based on this result we devise an algorithm to train Lipschitz Convolutional Neural Networks.