Relative entropy

We mentioned above that a typical problem for a Boltzman Machine is to obtain a set of weights such that the states of the visible neurons take on some desired probability distribution. For example, the task may he to teach the net to learn that the first component of an Ai-component input vector has value +1 40% of the time. To accompli.sh this, a Boltzman Machine uses the familiar gradient-descent technique, but not on the energy of the net instead, it maximizes the relative entropy of the system. 

On the other hand, we also have a set of desired probabilities P that we want the Boltzman Machine to learn. From elementary information theory, we know that the relative entropy... 

In this expression, the term (1 + Of) is interpreted as the probability that the hypothesis represented by the neuron is true (with Of = +1 meaning true and Of = —i- 1 meaning false), and the term (1 + Sf) is interpreted as the set of desired probabilities. With these, interpretations, / effectively yields the relative entropy between these two sets of probability measnres. 

Within group (i), square-planar EtPt(CO)(AsPh3)Cl inserts more rapidly than six-coordinate EtIr(CO)2(AsPh3)Cl2. In THE at 40°C, the relative k s are 9 and 1. Comparison of group (ii) alkyl carbonyls reveals the order MeMn(CO)5 > CpMo(CO)3Me > CpFe(CO)2Me. The ratios of the k s are 23 I and 100 1, respectively, in THF at 25° and 50.7°C. The higher reactivity of manganese than of molybdenum is a consequence of the relative entropies, whereas the lowest reactivity of iron is caused by its Jff (Table III). 

According to the latter model, the crystal is described as formed of anumber of equal scatterers, all randomly, identically and independently distributed. This simplified picture and the interpretation of the electron density as a probability distribution to generate a statistical ensemble of structures lead to the selection of the map having maximum relative entropy with respect to some prior-prejudice distribution m(x) [27, 28],... 

The name of the distribution is due to the fact that the saddle point X can also be obtained as the vector ofLagrange multipliers needed to find the distribution q = qME for which the relative entropy,... 

Carlsson, J. Aqvist, J., Absolute and relative entropies from computer simulation with applications to ligand binding, J. Phys. Chem. B 2005,109, 6448-6456. 

J. Schlitter, Estimation of absolute and relative entropies of macromolecules using the... 

Figure 8.3 Phase sequence and transition temperatures of a pure sample of the mesogen PYP 906. Plot illustrates relative entropy of various fluid phases, showing how each becomes thermodynamic minimum in some temperature range.

With the scores scaled in bit units, the relative entropy of the target and background frequencies is computed as... 

Fig. 3. Relative entropies for BLOSUM clustering percentages and Dayhoff PAM units.

The formula (101) gives a non-negative entropy production in agreement with the second law of thermodynamics. Indeed, the non-negativity of the right-hand member is guaranteed by the fact that the difference between Eqs. (98) and (97) is a relative entropy that is known to be non-negative ... 

Relative Entropies and Energies of Activation for Semicarbazone Formation at 12.5°... 

The relative entropy changes for all the processes discussed, pure equilibrium (C-E), pure frozen (C-F), kinetic rate controlling (C-I-G-A), modified Bray (C-B-R) and approximate Bray (C-T-S), are all shown on Fig. II. C. 1. In order to see somewhat more clearly the relative order of the performance for each procedure, the three-dimensional graph given in figure n. C. 2. is represented by... 

Another approach, namely a Minimum Relative Entropy (MRE) inversion was used by Woodbury et al. [70]. The MRE inversion is a method of statistical inference. Woodbury and co-workers also utilized the closed form solution, Eq. (46) and discretized it in the form... 

Woodbury AD, Ulrych TJ (1996) Minimum relative entropy inversion theory and application to recovering the release history of groundwater contaminant. Water Resour Res 32 2671-2681... 

Woodbury A, Sudicky E, Ulrych TJ, Ludwig R (1998) Three-dimensional plume source reconstruction using minimum relative entropy inversion. J Contam Hydrol 32 131-158... 

The most likely model, in the absence of other information, is one with the most positive enUopy. Discuss the relative entropies of the three models. 

Table 1 summarizes these parameters characterizing the keto-enol equilibria, where A refers to the difference between the enol and keto forms. The enol forms are significantly more stable, consistent with the inclusion of an intramolecular hydrogen bond in the structures and concurrent resonance stabilization. The low frequency torsional vibration of the keto forms can account for their significantly greater relative entropy. 

The data for heats of formation and relative entropy for the above calculations are not always readily available, except for the more commonly studied explo-... 

Kofke has provided a useful analysis to determine when numerical problems will occur in the calculation of the JE, and considers a pedagogical example. It is demonstrated that convergence will be problematic when the probability distributions of the work for the forward and reverse protocols are well separated. This will generally increase as the number of particles, rate of change of the parameter and size of the perturbation are increased. Kofke proposes using the relative entropy to assist in the assessment of the accuracy of the results obtained. 

Recent papers have noted the connection of the FR to Kullback-Leibler distance, or relative entropy which measures the irreversibility of the system and Sevick et al consider a similar property—the departure of the average of the time-averaged dissipation function from zero—as a measure of irreversibility. The effect of system on size on reversibility is discussed in ref. 213. 

A very useful criterion in this respect is given by the maximum entropy principle in the sense of Jaynes." The ingredients of the maximum entropy principle are (i) some reference probability distribution on the pure states and (ii) a way to estimate the quality of some given probability distribution p. on the pure states with respect to the reference distribution. As our reference probability distribution, we shall take the equidistribution defined in Eq. (30), for a two-level system (this definition of equipartition can be generalized to arbitrary dxd matrices, being the canonical measure on the d-dimensional complex projective plane - " ). The relative entropy of some probability distribution pf [see Eq. (35)] with respect to yXgqp is defined as... 

A nonmetric measure of relative entropy is the Kullback-Leibler divergence (also called information divergence, information gain, or relative entropy) that is a measure of the difference between two probability distributions P and Q, defined as... 

SAMPLE PROBLEM 20.1 Predicting Relative Entropy Values... 

Figure 5. Relative entropy of a 38-mer (210) lattice protein model determined by the ESMC procedure with enhanced sampling procedures (including both the CBMC and jump-walking techniques). The inset shows the standard deviations of the computed entropy function.

---