The Principle of Maximum Entropy

A major problem in Bayesian probability had been assigning the prior probability. Pioneering work by E. T. Jaynes has shown that there is a unique consistent way to assign prior probabilities, with this approach becoming known as the principle of maximum entropy. This principle states that the prior probabilities, pi, pi, , Pn, that we associate with the mutually exclusive hypotheses Hi, H2,Hn, should be those which maximize 

The concept of entropy is present in many disciplines. In statistical mechanics, Boltzmann introduced entropy as a measure of the number of microscopic ways that a given macroscopic state can be realized. A principle of nature is that it prefers systems that have maximum entropy. Shannon has also introduced entropy into communications theory, where entropy serves as a measure of information. The role of entropy in these fields is not disputed in the scientific community. The validity, however, of the Bayesian approach to probability theory and the principle of maximum entropy in this, remains controversial. 

It is fair to say that there is no universal agreement between those who champion either Bayesian or conventional probability theory. Indeed, as a bystander, one can find it interesting to watch the tussle. Conventional probability theory is very successful at modeling situations with large numbers of random events (e.g., the pressure in a vessel resulting fi om gas molecules colliding with the vessel walls). Indeed, statistical mechanics is based on conventional probability theory. On the other hand, Bayesian approaches seem to be more successful when the number of random events is smaller, and where there is insufficient information available. 

---

Equation (9.32) is a linear Fredholm integral equation of the first kind. It is also known as an unfolding or deconvolution equation. One can preanalyze the data and try to solve this first-kind integral equation. Besides the complexity of this equation, there is a paucity of numerical methods for determining the unknown function / (h) [208,379] with special emphasis on methods based on the principle of maximum entropy [207,380]. The so-obtained density function may be approximated by several models, gamma, Weibull, Erlang, etc., or by phase-type distributions. 

An extremum principle minimizes or maximizes a fundamental equation subject to certain constraints. For example, the principle of maximum entropy (dS)v = 0 and, (d2S)rj < 0, and the principle of minimum internal energy (dU)s = 0 and (d2U)s>0, are the fundamental principles of equilibrium, and can be associated with thermodynamic stability. The conditions of equilibrium can be established in terms of extensive parameters U and. S, or in terms of intensive parameters. Consider a composite system with two simple subsystems of A and B having a single species. Then the condition of equilibrium is... 

Shore, J. E. and Johnson, R. W., Axiomatic derivation of the principle of maximum-entropy and the principle of minimum cross-entropy. IEEE Trans. Inf. Theory 26, 26-37 (1980). 

Entropy Maximization and the Dishonest Die In this problem we will examine the way in which maximum entropy can be used to confront a problem in incomplete information. The basic idea is that we are told that the mean value that emerges after many rolls of a die is 4.5. On this information, we are urged to make a best guess as to the probability of rolling a 1, 2, 3,4, 5 and 6. Apply the principle of maximum entropy in order to deduce the probability p n) by applying the constraint 4.5 = np n). 

The intuition behind maximum entropy is that when we need to select among multiple possible distributions on a set of exclusive events, we choose the one that does not favor any of the events over the others. Hence, we choose the distribution that does not introduce new information that we did not have a priori. The principle of maximum entropy is widely used in other areas such as natural language processing. 

Particularly related to thermodynamics is the second law. The second law states that at constant energy the entropy tends to reach a maximum in equilibrium. The reciprocal statement is that at constant enfropy the energy tends to reach a minimum in equilibrium. Often in thermodynamics the principle of minimum energy is deduced from the principle of maximum entropy via a thermodynamic process that is not conclusive, in that as the initial assumption of maximum entropy is violated. These methods originate to Gibbs. Other proofs run via graphical illustrations. 

The principle of maximum entropy is widespread in other disciplines besides thermodynamics, for example, in information theory, economics. In fact, it implies to maximize the probability of a state of a system under certain constraints [5],... 

These equations indicate that a maximum of entropy is achieved at equilibrium. On the other hand, if we demand an isolated system that is even not transparent for entropy, then we must conclude that under such circumstances entropy will be generated, if such as system is moving toward equilibrium. We have used here the principle of maximum entropy under the constraints of constant energy and constant volume. 

This admissibihty is criticized by Rajagopal and his school using instead the assumption of maximum of entropy production rate, their further restrictions may be obtained, see, e.g., [37-39]. However, there are also reservations to the principle of maximum entropy production [40]. 

In the literature the terms entropy and information are frequently interchanged. Arih Ben-Naim, the author of Farewell to Entropy Statistical Thermodynamics Based on Information [52] insists on going one step further and motivates not only to use the principle of maximum entropy in predicting the probability distribution [which is used in statistical physics], but to replace altogether the concept of entropy with the more suitable information. In his opinion this would replace an essentially... 

---