Learning and Information theory

As a system learns, it evolves its knowledge about the world. If we think of this knowledge being represented by a probabilistic model of the world, we can define entropy of the

In 1970, E. Pfaffelhuber wrote a paper titled "Learning and Information Theory" and submitted to the international journal of neuroscience [doi]. The introduction is very intriguing.

Intuitively, learning means an accumulation of a system's knowledge or information about a set X of data or events x or, equivalently, a decrease of the system's missing information about these data in the lapse of time. Thus, a quantitative definition of learning seems possible, provided one is able to introduce a measure for a system's missing information. As has been pointed out by various authors, Leibovic (1969, "information processing in the nervous system"), Shannon's classical information measure is not appropriate to describe behavioral processes of biological systems, the reason being that this measure is not appropriate to describe behavioral processes of biological systems, the reason being that this measure is based solely upon objective probabilities and cannot, therefore, represent a system's knowledge or beliefs nor can it discriminate between events which are of great importance and others which are irrelevant for an individual system.

The paper ends up talking about Kullback-Leibler divergence as a measure of difference between the actual probability and subjective probability. However, it is not readily usable by any learning system, because the actual probability is not known. This concept is extended by Palm's 1981 paper "Evidence, Information, and Surprise" in Biological Cybernetics [doi].