Principal Components Analysis Of Images Via Back Propagation

The recent discovery of powerful learning algorithms for parallel distributed networks has made it possible to program computation in a new way, by example rather than algorithm. The back propagation algorithm is a gradient descent technique for training such networks. The problem posed to the researcher using such an algorithm is discovering how it did solve the problem if a solution is found. In this paper we apply back propagation to a well understood problem in image analysis, i.e., bandwidth compression, and analyze the internal representation developed by the network. The network used consists of nonlinear units that compute a sigmoidal function of their inputs. It is found that the learning algorithm produces a nearly linear transformation of a Principal Components Analysis of the image, and the units in the network tend to stay in the linear range of the sigmoid function. The particular transform found departs from the standard Principal Components solution in that near-equal variance of the coefficients results, depending on the encoding used. While the solution found is basically linear, such networks can also use the nonlinearity to solve encoding problems where the Principal Components solution is degenerate.