Learning and Information Theory

A general statistical description of learning processes is given, using an extension of classical information theory which is able to characterize a system's subjective uncertainty (subjective entropy) about a set of data and the relevance associated with them. Learning is defined as a process in which the system's subjective entropy or, equivalently, its missing information decreases in time. This definition implies that learning enables the system to optimize its responses upon external stimuli such that its expected profit increases. A physical model is discussed where the system's subjective probabilities, which enter into the definition of its missing information, can be considered as internal physical parameters. It is conjectured that the same might be true in models of brain and memory. Two forms of the associative memory model are briefly discussed.