Abstract

Given a collection of random variables X(i) where i belongs to a finite nonempty set N, the corresponding multiinformation function ascribes (to every subset A of N) the relative entropy of the joint distribution of X(A) (that is, the random vector composed of the variables X(i) where i belongs to A) with respect to the product of distributions of individual random variables X(i) through i in A. We argue that it is a useful tool for solving the problems concerning stochastic (conditional) dependence and independence (at least in discrete case). First, it makes possible to express the conditional mutual information between X(A) and X(B) given X(C) (for every disjoint subsets A,B,C of N) which can be considered as a good measure of conditional stochastic depedence. Second, one can introduce reasonable measures of dependence of level r among variables of X(A) (where A is a subset of N and 0 < r < card A) which are expressible by means of the multiinformation function. Third, it enables one to derive theoretical results on (nonexistence of an) axiomatic characterization of stochastic conditional independence models.

In fact, the last part of the paper is nothing but a didactive proof of the result from the paper:

M. Studeny: Conditional independence relations have no finite complete characterization. In Information Theory, Statistical Decision Functions and Random Processes. Transactions of the 11th Prague Conference vol. B (S. Kubik, J.A. Visek eds.), Kluwer, Dordrecht - Boston - London (also Academia, Prague) 1992, pp. 377-396.

AMS classification 94A17, 68T30 (03B30, 62H05)

Keywords

multiinformation

conditional mutual information

relative entropy

level-specific measures of dependence

(regular) inference rule

perfect rule

A pdf copy (converted postscript version) (366kB) is available.

The results of the paper has a connection to the following works:

• T.S. Han: Nonnegative entropy of multivariate symmetric correlations. Information and Control 36 (1978), pp. 113-156.
• M. Studeny, P. Bocek: CI-models arising among 4 random variables. In Proceedings of the 3rd workshop WUPES, September 11-15, 1994, Trest, Czech Republic, pp. 268-282.