On-line learning from finite training sets

We analyse on-line (gradient descent) learning of a rule from a finite set of training examples at non-infinitesimal learning rates η, calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an infinite hierarchy of order parameters using "within-sample self-averaging". The resulting dynamics is non-perturbative in η, with a slow mode appearing only above a finite threshold ηmin. Optimal settings of η for given final learning time are determined and the results are compared with offline gradient descent.