Information Theory | Research & Encyclopedia Articles

Information Theory

Among the more interesting trends of the past half-century has been the consolidation of probability, statistics, combinatorial optimization, information theory, and computer science into a single imposing discipline of infomatics.

Minimal Belief Change

Of special philosophical interest is the enrichment of Bayesian inference by a rule of belief change that goes by minimizing the distance between two probability distributions, P = (p₁, … , p_k and Q = (q₁, … , q₂), as measured by the expected log-likelihood ratio:
(1)    � 0A0;
The likelihood ratio, P(e|h):P(e|k), is a fundamental index of the support that e accords h over k (see the entry "Foundations of Statistics").

Using the visually transparent Gibbs inequality,
(2)     ln x ≤ x − 1
with equality if and only if (iff) x = 1, in the equivalent form ln x ≥ 1 − 1/x, it follows that H(P, Q) ≥ 0 with equality iff P = Q. Notice, however, that H(P, Q) ≠ H(Q, P).

Alan Turing and his wartime assistant, Irving John Good, used H(P, Q) in their code-breaking work, but it was not until 1959 that another wartime code breaker, Solomon Kullback, developed its properties systematically in his book Information Theory and Statistics (1959), unleashing a floodtide of applications to classification, contingency tables, pattern recognition, and other topics.

Information Theory article

Read the rest of this article.
This article contains 4,647 words (approx. 15 pages at 300 words per page).