On-line versus off-line learning in the linear perceptron: A comparative study

The spherical perceptron with N inputs and a linear output does not present optimal generalization if trained by minimization of the standard quadratic cost function E=1/2 J_μ=1^αN (b_μ-h_μ{)}²$, where ${b}_{{{μ}}}$ and ${h}_{{{μ}}}$ are the outputs from the rule (teacher) and hypothesis (student) networks for the example {μ} and there are {α}N examples. We derive an optimal algorithm for on-line learning of examples which outperforms the iterative (off-line) standard algorithm for {α} up to 0.71. The on-line optimized algorithm suggests a class of cost functions for off-line learning, which we then proceed to study using the replica method. The optimized cost function within that class has the suggestive form E={α}N[{Γ}(1/{α}N) J_μ=1^αN [-lnP(b_μ{}h_μ)]-{Γ} lnZ], where Z is a normalization constant, P(b_μ{}h_μ) is the conditional probability of the output data b_μ given the hypothesis output h_μ, and {Γ} is a learning parameter analogous to a temperature which decreases in a well defined manner along the learning process.