A simple and practical review of over-fitting in neural network learning

Training a neural network involves the adaptation of its internal parameters for modelling a specific task. The states of the internal parameters during training describe how much experiential knowledge the model has acquired. Although, it is desirable that a trained neural network achieves zero classification error on the training examples while tuning its internal parameters for a task, the amount of generalisation power that is lost while enforcing such a learning constraint on the model is quite important. In this paper, we review from a practical perspective the consequences of enforcing such a learning constraint which results in a model that has learned a smooth mapping function or essentially 'memorised' the training data. In addition, we investigate how the curse of dimensionality relates to such a learning constraint. For our experiments, we consider handwritten character recognition applications using publicly available datasets.