On Information in Statistics

The three familiar definitions of statistical information, Fisher (1925), Shannon (1948), and Kullback (1959), are closely tied to asymptotic properties, hypothesis testing, and a general principle that information should be additive. A definition of information is proposed here in the framework of an important kind of statistical model. It has an interpretation for small samples, has several optimum properties, and in most cases is not additive with independent observations. The Fisher, Shannon and Kullback informations are aspects of this information. A variety of definitions of information may be found in the statistical literature. Perhaps the oldest and most familiar is that due to Fisher (1925). For a real parameter and a density function satisfying Cramer-Rao regularity conditions it has the form align*IF () \\ align* for a vector parameter it becomes a matrix align*IF () = cov \ (/) f (x) \ \\ with a change of sign it measures concentration and is thereby more appropriate for statistics: IS () = f (x) f (x) dx. Kullback (1959) considers a definition of information for "discriminating in favor of H₁ (₁) against H₂ (₂) ": IK (₁, ₂) = f (x ₁) /f (x ₂) f (x ₁) dx. These information functions are additive with independent observation; in fact, additivity is taken as an essential property in most developments of information. The three information functions are defined for quite general statistical models, (mild regularity required for Fisher's definition). And the Fisher and the Kullback definitions are tied closely to large sample theory and to Bayes' theory respectively. The emphasis in this paper is not on information in a general model based on general principles. Rather it is on information in an important and somewhat special model--the location model and, more generally, the transformation-parameter model.