Linear Discriminant Analysis with few training data

Statistically-optimal Linear Discriminant Analysis (LDA) is formulated as a maximization that involves the nominal statistics of the classes to be discriminated. In practice, however, these nominal statistics are unknown and estimated from a collection of labeled training data. Accordingly, the nominal LDA basis is approximated by the solution of the popular practical LDA problem defined upon these estimates. However, when the available training data are few, the solution to practical LDA is known to lie far from the nominal LDA basis. In this work, we propose a novel algorithm that operates on the estimated class statistics and generates a sequence of bases that converges to the solution of practical LDA. Importantly, our studies illustrate that early elements of the proposed sequence exhibit significantly higher approximation to the nominal LDA basis than the converging point and, thus, offer the means for superior classification performance.