Resumen de Information-estimation relationships over binomial, negative binomial and Poisson models
Camilo Gil Taborda
This thesis presents several relationships between information theory and estimation theory over random transformations that are governed through probability mass functions of the type binomial, negative binomial and Poisson. The pioneer expressions that arose relating these fields date back to the 60's when Duncan proved that the input-output mutual information of a channel affected by Gaussian noise can be expressed as a time integral of the causal minimum mean square error. With the time, additional works due to Zakai1, Kadota, Mayer-Wolf, Lipster and Guo et al. -among others- suggested the fact that there could be a hidden structure relating concepts such as the mutual information with some estimation quantities over a wide range of scenarios. The most prominent work in this field states that, over a real-valued Gaussian channel, the derivative of the input-output mutual information with respect to the signal to noise ratio is proportional to the mean square error achieved when measuring the loss between the input X and its conditional mean estimate based on an observation on the output. The minimum value of the mean square error is achieved precisely by the conditional mean estimate of the input, which gives rise to the well known “I-MMSE" relationship. Similar expressions can be derived by studying the derivative of the relative entropy between two distributions obtained at the output of a Gaussian channel. The expressions proved for the Gaussian channel translate verbatim to the Poisson channel where the main difference lies in the loss function used to state the connection between information and estimation. In this framework, regarding the derivative of the input-output mutual information, it is further known that the considered loss function achieves its minimum value when is measured the difference between the input and its conditional mean estimate. This behavior has two main implications: in the context of the information-estimation relationships, it gives rise to the “I-MMLE" relationship over the Poisson model; second, it converts the loss function to a Bregman divergence, a property that is shared with the square distance used to state information-estimation relations in the Gaussian channel. Based on the previous results we explore similar relationships in the context of the binomial and negative binomial models. In each model, using a deterministic input preprocessing, we develop several information-estimation relationships, depending solely on input statistics and its respective conditional estimates, that in some scenarios are given through Bregman divergences as was done formerly for the Gaussian and Poisson models. Working over models whose mean is given by a linear scaling of the input X through a parameter ?, we show for the binomial and negative binomial models, that the derivative of the input-output mutual information is given through a Bregman divergence where the arguments are the mean of the model and its conditional estimate. This condition gives rise to relationships that are of the same kind as the “I-MMSE" and the “I-MMLE" found initially for the Gaussian and Poisson models. Similar expressions are developed for the relative entropy, where the arguments of the Bregman divergence are the conditional mean estimate of the model ?X assuming that X ~ PX and its correspondent mismatched version when X ~ QX. Making the input scaling factor tends to zero, we show that the derivative of the input-output mutual information is proportional to the expectation of a Bregman divergence between the input X and its mean E[X]. This behavior is similar to that proved for the case of the Gaussian channel where, when the signal to noise ratio goes to zero, the derivative of the mutual information tends to the variance of the input. Furthermore, using an arbitrary input preprocessing function that is not necessarily linear, we prove that several scenarios lead to information-estimation expressions that are given through Bregman divergences even though this is not always the case. In those cases where the information-estimation relationship is given through the minimum of a Bregman divergence, an information-estimation relationship similar to the “I-MMSE" and “I-MLE" relationships can be stated. Finally, we provide conditions for which the results obtained for the binomial and negative binomial models converge asymptotically to information-estimation relationships over the Poisson model. This technique let us present connections between information and estimation over the Poisson model that cover wider scenarios than those studied so far.
