Maximal entropy distribution

There are two ways to implement this program. One is to directly discuss the distribution of final states. This is known as surprisal analysis . In this simpler procedure one does not ask how this distribution came about Instead, one seeks the coarsest or most statistical" (= of maximal entropy) distribution of final states, subject to constraints. The last proviso is, of course, essential. If no constraints axe imposed, one will obtain as an answer the prior distribution. It is the constraints that generate a distribution which contains just that minimal dynamical detail as is necessary to generate the answers of interest. Few simple and physically obvious constraints are often sufficient [1, 3, 23] to account for even extreme deviations from the prior distribution. 

The commercially available software (Maximum Entropy Data Consultant Ltd, Cambridge, UK) allows reconstruction of the distribution a.(z) (or f(z)) which has the maximal entropy S subject to the constraint of the chi-squared value. The quantified version of this software has a full Bayesian approach and includes a precise statement of the accuracy of quantities of interest, i.e. position, surface and broadness of peaks in the distribution. The distributions are recovered by using an automatic stopping criterion for successive iterates, which is based on a Gaussian approximation of the likelihood. 

The prior distribution is often not what is observed, and there can be extreme deviations from it. [1, 3, 23] By our terminology this means that the dynamics do impose constraints on what can happen. How can one explicitly impose such constraints without a full dynamical computation At this point I appeal again to the reproducibility of the results of interest as ensured by the Monte Carlo theorem. The very reproducibility implies that much of the computed detail is not relevant to the results of interest. What one therefore seeks is the crudest possible division into cells in phase space that is consistent with the given values of the constraints. This distribution is known as one of maximal entropy". 

The distribution of fragment size is computed as one of the maximal entropy subject to two constraints (1) conservation of matter ... 

The reason we employ two rather distinct methods of inquiry is that neither, by itself, is free of open methodological issues. The method of molecular dynamics has been extensively applied, inter alia, to cluster impact. However, there are two problems. One is that the results are only as reliable as the potential energy function that is used as input. For a problem containing many open shell reactive atoms, one does not have well tested semiempirical approximations for the potential. We used the many body potential which we used for the reactive system in our earlier studies on rare gas clusters containing several N2/O2 molecules (see Sec. 3.4). The other limitation of the MD simulation is that it fails to incorporate the possibility of electronic excitation. This will be discussed fmther below. The second method that we used is, in many ways, complementary to MD. It does not require the potential as an input and it can readily allow for electronically excited as well as for charged products. It seeks to compute that distribution of products which is of maximal entropy subject to the constraints on the system (conservation of chemical elements, charge and... 

Internally equilibrated subsystems, which act as free energy reservoirs, are already as random as possible given their boundary conditions, even if they are not in equilibrium with one another because of some bottleneck. Tlius, the only kinds of perturbation that can arise and be stabilized when they are coupled are those that make the joint system less constrained than the subsystems originally were. (This is Boltzmann s H-theorem [9] only a less constrained joint system has a liigher maximal entropy than die sum of entropies from the subsystems independently and can stably adopt a different form.) The flows that relax reservoir constraints are thermochemical relaxation processes toward the equilibrium state for tlte joint ensemble. The processes by wliich such equilibration takes place are by assumption not reachable within the equilibrium distribution of either subsystem. As the nature of the relaxation phenomenon often depends on aspects of the crosssystem coupling that are much more specific than the constraints that define either reservoir, they are often correspondingly more complex than the typical processes... 

To implement the procedure of maximal entropy, we need to write down the entropy. Here care need be exercised since the usual expressions for the entropy are only valid for that distribution which, in the absence of all but the normalization constraint, is uniform. We suggest that this natural distribution is not necessarily P(y). Consider first the case where all bound states and the transition operator are real. Then y = x2 where x is real. In the absence of any constraints we require that the natural distribution remain unchanged under an orthogonal change of the basis set. The distribution that is unchanged by a rotation of x is the distribution uniform in x. The alternative, a distribution uniform in y, will not stay uniform after a rotation of the basis. 

One can show that the result, Eq. (2.15), that (2.12) is the distribution of maximal entropy implies also that P(x) is more uniform than Q(x). 

Experimental and computational results often do deviate from the distributions of maximal entropy subject only to a given total strength. Consider the following simple modification. The spectrum can be regarded as the set of expectation values of the state Pi... 

We now impose not only a given overall strength but also a given envelope. The average local intensity is thus also specified. This requires the introduction of the set of N constraints (3.13), each of which is assigned the Lagrange multiplier yf. The distribution of maximal entropy subject to the three constraints (1)—(3)is... 

The amplitudes of the different lines in the spectrum can be regarded as the components of the optically prepared bright state in the basis of the eigenstates of the system, cf. Eqs. (2)-(3). Already in Sec. II we had several occasions to note that the bright state behaves not unlike a random vector. One can therefore ask what the spectrum will be if we make the approximation that these components are truly random. This requires us to specify, in a technical sense, what we mean by random. This is where entropy comes in. By random we will mean that the distribution of the amplitudes be as uniform as possible and, as such, be a distribution of maximal entropy. 

Consider first the limit where no dynamical input is given. Then only the value of C(t) at / = 0 is imposed as a constraint in determining P(yf) by the procedure of maximal entropy. In this case all the Lagrange multipliers have exactly the same value, namely (x = 1/X.o and so all the distributions P(yf) are the same. One can therefore explicitly carry out the sum over final states in (102), with the result... 

Based on the entropy principle proposed by Shannon (1948) entropy distributions are defined to be those which maximize the information entropy measure... 

The maximum entropy distribution is Gaussian when the second moment is given. Pro e that the probability distribution p, that maximizes the entropy for die rolls subject to a constant value of the second moment (t ) is a Gaussian function. Lfse / = i. 

If all quantum states are equally probable, then one can say that the distribution of final states has a maximal entropy. So far, the maximal value of the entropy is just a restatement of the result that the distribution is as uniform as possible. If the final states are not equally probable, we ask that the entropy of the distribution of states be as large as is allowed by the dynamical constraints. In other words, the working hypothesis is that all states are as probable as possible under the constraints imposed by the dynamics. 

The details of determining a distribution of maximal entropy are spelled out in a number of reviews. Here we just give an example suppose the dynamic constraint is the final vibrational energy. This means that the dynamics imposes on the distribution of final states a mean value of the final vibrational energy... 

In some circles the approach that we follow, that of looking for a distribution of maximal entropy. Section 6.4.2.1, is questioned. This occurs because people sometimes overlook the need to include a prior distribution in Eq. (6.32). In this case it follows that in the absence of constraints all groups of outcomes are equalty probable. Very often this is manifestly not reasonable and correctly so because different groups can differ in how many mutually exclusive outcomes are members of the group. 

We mentioned above that a typical problem for a Boltzman Machine is to obtain a set of weights such that the states of the visible neurons take on some desired probability distribution. For example, the task may he to teach the net to learn that the first component of an Ai-component input vector has value +1 40% of the time. To accompli.sh this, a Boltzman Machine uses the familiar gradient-descent technique, but not on the energy of the net instead, it maximizes the relative entropy of the system. 

The choice of the preferred solution within the feasible set can be achieved by maximizing some function F[a(r)] of the spectrum that introduces the fewest artefacts into the distribution. It has been proved that only the Shannon-Jaynes entropy S will give the least correlated solution 1. All other maximization (or regularization) functions introduce correlations into the solution not demanded by the data. The function S is defined as... 

Consider two liquid substances that are rather similar, such as benzene and toluene or water and ethylene glycol. When moles of the one are mixed with B moles of the other, the composition of the liquid mixture is given by specification of the mole fraction of one of them [e.g., Xa, according to Eq. (2.2)]. The energy or heat of the mutual interactions between the molecules of the components is similar to that of their self interactions, because of the similarity of the two liquids, and the molecules of A and B are distributed completely randomly in the mixture. In such mixtures, the entropy of mixing, which is a measure of the change in the molecular disorder of the system caused by the process of mixing the specified quantities of A and B, attains its maximal value ... 

The corresponding formulation was made by von Neumann2 for quantum mechanics. This entropy-maximizing (or information-minimizing) principle is the most direct path to the canonical distribution and thus to the whole equilibrium theory. It is understood that the extremalizing is conditional, i.e., certain expected values, such as that of the Hamiltonian, are fixed. 

The measure 5 [P(co)] is called the entropy corresponding to the distribution P((o). According to information theory, if a certain set of moments of P(co) are known, that P(co) is optimum which maximizes 5 [P(co)] subject to the moment constraints. Suppose we know only... 

---