Information mutual

Mutual information, effectively measures the degree to which two probability distributions or, in the context of CA, two sites or blocks - are correlated. Given probability distributions pi and pj and the joint probability distribution py, 1 is defined by  

Numerical experiments by Langton [langQOa] show once again that when Xm is plotted against A, there is a region for small A for which Xm is essentially zero, a critical A at which Xm jumps to some moderate value followed by a slow decay as 

A increases from that point on. From the perspective of computation theory taken in his paper, Langton argues that it is the set of intermediate values of A that are most important. 

A particularly revealing plot in this regard, shown schematically in figure 3.43, is a graph of Im versus H, where is computed between a site and itself at the next iteration step and H is normalized to i ,ax = 1- 

Notice the sharp maximal value of Im at a critical He ( 0.32), suggesting that there exists an optimal entropy for which CAs yield large spatial and temporal correlations. Langton conjectures that this results from two competing requirements that must both be satisfied in order for an effective computational ability to exist information storage, which involves lowering entropy, and information transmission, which involves increasing entropy. 

The concept of mutual information originates from work in information theory [52] and can be seen as a generalization of the correlation coefficient. The mutual information between a class c and an input feature Xj is the amount to which the knowledge provided to the feature vector decreases the uncertainty about the class. 

Mutual information is calculated from the probability distributions, p(x), p(y) and p(x,y). p(x) is the distribution of the values for a certain variable x. p(x,y) is the joint probability between the x variable and the dependent variable y. A comparison of the joint probability p(x,y) with p(x)p(y) is made. For statistically independent data [53] we have that  

If these probabilities are not the same, there is a dependence between the two distributions and no prior assumptions about its form are made. 

The standard way of creating probability distributions from histograms is only optimal for dense data. For this reason, an alternative method is employed that is based on kernel density estimation [54]. These probability distributions estimated from this approach are subsequently used in the formula for the calculation of the mutual information, I(x,y) [53]. 

y) has a large magnitude if one distribution provides much information about the other, small if it provides little. In a purely linear Gaussian situation, I(x,y) reduces to correlation and provides identical results. 

---

Fig. 3.43 Schematic form of a plot of mutual information Xm vs. average single-site entropy H.

Dynamical Entropy In order to capture the dynamics of a CML pattern, Kaneko has constructed what amounts to it mutual information between two successive patterns at a given time interval [kaneko93]. It is defined by first obtaining an estimate, through spatio-temporal samplings, of the probability transition matrix Td,d = transition horn domain of size D to a domain of size D. The dynamical entropy, Sd, is then given by... 

Mutual Information.—In the preceding sections, self informa- tion was defined and interpreted as a fundamental quantity associated with a discrete memoryless communication source. In this section we define, and in the next section interpret, a measure of the information being transmitted over a communication system. One might at first be tempted to simply analyze the self information at each point in the system, but if the channel output is statistically independent of the input, the self information at the output of the channel bears no connection to the self information of the source. What is needed instead is a measure of the information in the channel output about the channel input. 

Mutual information is thus a random variable since it is a real valued function defined on the points of an ensemble. Consequently, it has an average, variance, distribution function, and moment generating function. It is important to note that mutual information has been defined only on product ensembles, and only as a function of two events, x and y, which are sample points in the two ensembles of which the product ensemble is formed. Mutual information is sometimes defined as a function of any two events in an ensemble, but in this case it is not a random variable. It should also be noted that the mutual... 

The average mutual information between the input and output sequences is given by... 

Theorem 4-8. Let C be the capacity of a discrete memoryless channel, and let 7(x y) be the average mutual information between input and output sequences of length N for an arbitrary input probability measure, Pr(x). Then... 

If the channel inputs are statistically independent, and if the individual letter probabilities are such as to give channel capacity, then the average mutual information transmitted by N letters is NO. 

These results are sometimes interpreted as a converse to the coding theorem. That is, we have shown that the mutual information per channel symbol between a source and destination is limited by the capacity of the channel. The problem is that we have not demonstrated any relationship between error probability and source rate when the source rate is greater than the mutual information. Unfortunately, this is not as trivial to obtain as it might appear. In the next section, we find a lower bound to Hie probability of error when the source rate is greater than C. 

Converse to Coding Theorem.—We shall show in this Section that reliable communication at rates greater than channel capacity is impossible. Although only discrete memoryless sources and channels will be considered here, it will be obvious that the results are virtually independent of the type of channel. It was shown in the last section that the average mutual information between source and destination can be no greater than channel capacity then if the source rate is greater than capacity, some information is lost. The problem is to relate this lost information, or equivocation, to the probability of error between the source and destination. 

Since the coder and decoder cannot increase the average mutual information (see Eq. (4-62)). 

Problem—Show that Eq. (4-178) is the mutual information in nats between channel input and output when F(x) is the distribution function for channel inputs. Use Eq. (4-41) for quantized input and output and pass to the limit. 

Multiple roots of matrices, 68 Mutual information, 205 average, 206... 

Maximum Mutual Information Based Sensor Selection Algorithm... 

Active participation to the current cycle is decided based on the mutual information gained with the last observation. This event can be formulated as... 

We run Monte Carlo simulations to examine the performance of the sensor selection algorithm based on the maximization of mutual information for the distributed data fusion architecture. We examine two scenarios first is the sparser one, which consists of 50 sensors which are randomly deployed in the 200 m x 200 m area. The second is a denser scenario in which 100 sensors are deployed in the same area. All data points in the graphs represent the means of ten runs. A target moves in the area according to the process model described in Section 4. We utilize the Neyman-Pearson detector [20, 30] with a = 0.05, L = 100, r) = 2, 2-dB antenna gain, -30-dB sensor transmission power and -6-dB noise power. 

A mutual information based information measure is adopted to select the most informative subset of sensors to actively participate in the distributed data fusion framework. The duty of the sensors is to accurately localize and track the targets. Simulation results show 36% energy saving for a given tracking quality can be achievable by selecting the sensors to cooperate according to the mutual information metric. 

E. Ertin, J. W. Fisher, and L. C. Potter, Maximum mutual information principle for dynamic sensor query problems , in Proceedings of the 2nd International Workshop on Information Processing in Sensor Networks, Palo Alto, USA, April 2003, pp. 405-416. 

This is the mutual information between the target variable (range and Doppler) X and the processed (with a matched filter) radar return Y, resulting from the use of the waveform identity matrix. We use this expected information as the MoE of the waveform more information we extract from the situation the better. 

In the case of a finite number of waveforms in the library, we observe that the utility of the rotation library improves with the number of waveforms in the library. We can show that there exists a unique 0 which maximizes the mutual information I(X]Y) and, in a similar fashion to the pure chirp library case,... 

As in the previous experiments, at each epoch we would like to select a waveform (or really the error covariance matrix associated with a measurement using this waveform) so that the measurement will minimize the uncertainty of the dynamic model of the target. We study two possible measures entropy of the a posteriori pdf of the models and mutual information between the dynamic model pdf and measurement history. Both of these involve making modifications to the LMIPDA-IMM approach that are described in [5]. Since we want to minimize the entropy before taking the measurement, we need to consider the expected value of the cost. To do this we replace the measurement z in the IMM equations by its expected value. In the case of the second measure, for a model we have... 

Figure 7. Cost Function and Correct Maneuver Identification for Mutual Information Cost.-...

---