Relative entropy function

There are other metrics of information content, and several of them are based on the Shannon entropyAbout 10 years after introduction of the Shannon entropy concept, Jaynes formulated the maximum entropy approach, which is often referred to as Jaynes entropy and is closely related to Shannon s work. Jaynes introduction of the notion of maximum entropy has become an important approach to any study of statistical inference where all or part of a model system s probability distribution remains unknown. Jaynes entropy, or relations, which guide the parameterization to achieve a model of minimum bias, are built on the Kullback-Leibler (KL) function, sometimes referred to as the cross-entropy or relative entropy function, which is often used and shown (in which p and q represent two probability distributions indexed by k), as... 

Recent papers have noted the connection of the FR to Kullback-Leibler distance, or relative entropy which measures the irreversibility of the system and Sevick et al consider a similar property—the departure of the average of the time-averaged dissipation function from zero—as a measure of irreversibility. The effect of system on size on reversibility is discussed in ref. 213. 

Table 1 lists the thermodynamic functions of some inhibitors studied previously. The free energy change, AG°, is directly determined from the logarithm of the inhibition constant (i.e. RTlnKj), the enthalpy, entropy and heat capacity are the fitting results by using eq 7 where K, is replaced by Kj. Based on Scheme I, by using the relative thermodynamic functions (AAG°, AAH°, T°AAS°) given by... 

In addition to the estimated properties, we measured the thermochemistry of several important vapor species. These measurements were conducted in a Knudsen effusion cell using special line-of-sight vaporization under subambient pressures with flowing O2 and H2O vapor mixtures [4]. The gaseous species over silica [5], manganese oxide [6], lanthana, alumina, and palladium metal were detected and relative partial pressures measured as a function of temperature. These vapor pressure measurements were calibrated by using the known metal atom or binary metal oxide volatility as a calibration source. Oxide species concentrations were measured relative to that of a reference compound, e.g., metal atom. The identification of oxide and hydroxide compounds was facilitated by Ae technique of threshold electron ionization [7]. These data were then evaluated using estimated entropy functions and the third law temperatures. 

A major drawback of MD and MC techniques is that they calculate average properties. The free energy and entropy functions cannot be expressed as simple averages of functions of the state point y. They are directly connected to the logarithm of the partition function, and our methods do not give us the partition function itself. Nonetheless, calculating free energies is important, especially when we wish to determine the relative thermodynamic stability of different phases. How can we approach this problem ... 

Figure 5. Relative entropy of a 38-mer (210) lattice protein model determined by the ESMC procedure with enhanced sampling procedures (including both the CBMC and jump-walking techniques). The inset shows the standard deviations of the computed entropy function.

This formulation for the log evidence for model class Af, shows that it is the difference between two terms the first term is the posterior mean of the log-likelihood function, which is a measure of the average data fit for model class Mj, while the second term is the relative entropy between the prior and posterior distributions, which is a measure of the information gained about the parameters 0/ from the data V. Therefore, the log evidence is comprised of a data-fit term and a term which provides a penalty against more complex models that extract more information from the data. This gives an intuitive understanding of why the application of Bayes Theorem at the model class level automatically enforces Ockham s razor Pluralitas non est ponenda sine neccesitate ( entities should not be multiplied unnecessarily ). Although this information-theoretic interpretation was initially presented in Beck Yuen (2004),... 

For this example and Example 6.1, fe should not be Boltzmann s constant. Boltzmann s constant is appropriate only when you need to put entropy into units that interconvert with energy, for thermodynamics and molecular science. For other types of probability distributions, k is chosen to suit the purposes at hand, so fe = 1 would be simplest here. The entropy function just reports the relative flatness of a distribution function. The limiting cases are the most ordered, S = 0 (everybody wears the same color socks) and the most disordered, 5/fe = In t = In 5 = 1.61 (all five sock colors are equally Ukely). 

The Kullback-Leibler formulation evaluates the relative entropy between two data distributions. However, it is not symmetrical with respect to the two distributions under comparison that is, one must declare one distribution as the base set or reference from which the other is assumed to depart. Concerning the connection between Jaynes entropy and the Kullback-Leibler function, maximum entropy is achieved when qi is replaced with a distribution about which there is prior knowledge and pk is adjusted so as to maximize KL. Prior knowledge could, for example, be the mean or expectation value of a data distribution. Importantly, because of the quotient involved (Eq. [4]), the Kullback-Leibler function becomes undefined if any bin is unpopulated. This renders this function inappropriate for the purposes of estimating information content in chemical descriptor sets, which is discussed below. 

An exceptionally badly reported kinetic study in which a linear correlation of rate coefficient with acidity function was claimed was that of Mackor et al. 11, who studied the dedeuteration of benzene and some alkylbenzenes in sulphuric acid-trifluoroacetic acid at 25 °C. Rates were given only in the form of a log rate coefficient versus —H0 plot and rate coefficients and entropies of activation (measured relative to p-xylene) together with heats of activation (determined over a temperature range which was not quoted) were also given (Table 129). However,... 

The most common states of a pure substance are solid, liquid, or gas (vapor), state property See state function. state symbol A symbol (abbreviation) denoting the state of a species. Examples s (solid) I (liquid) g (gas) aq (aqueous solution), statistical entropy The entropy calculated from statistical thermodynamics S = k In W. statistical thermodynamics The interpretation of the laws of thermodynamics in terms of the behavior of large numbers of atoms and molecules, steady-state approximation The assumption that the net rate of formation of reaction intermediates is 0. Stefan-Boltzmann law The total intensity of radiation emitted by a heated black body is proportional to the fourth power of the absolute temperature, stereoisomers Isomers in which atoms have the same partners arranged differently in space, stereoregular polymer A polymer in which each unit or pair of repeating units has the same relative orientation, steric factor (P) An empirical factor that takes into account the steric requirement of a reaction, steric requirement A constraint on an elementary reaction in which the successful collision of two molecules depends on their relative orientation. 

Other thermodynamic quantities such as chemical potential and entropy also follow directly from the partition function, as we demonstrate later on. However, to illustrate what a partition function means, we will first discuss two relatively simple but instructive examples. 

---