Manifold Curvature Estimation for Neural Networks

This paper introduces a novel method for creating a metric to measure curvature of a discretized manifold. For each data point x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</inf> on a manifold, a subspace Sᵢ is matched using a number of neighboring points of x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</inf> . A local subspace is also matched to each neighboring point of x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</inf> . Then, a set of weighted angles between Sᵢ and each neighboring local subspace are computed and the minimum of those weighted angles is used as a measure of curvature of the manifold at x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</inf> . The average curvature for all data points on the manifold is used as metric for the manifold’s curvature estimation. This research also uses the proposed metric to show that each layer of a neural network maps an input manifold to a flatter manifold during the training process. It is observed that each successive block in a neural network generates a manifold with less curvature than that of the previous layer. Another observation is that convolutional layers always generate flatter manifolds. Furthermore, it is shown that this metric can be used as a robustness measure for a neural network. The method has been tested successfully using two datasets MNIST and Extended YaleB datasets.