The mathematics of entropy maximization

The maximum entropy method (MEM) is an information-theory-based technique that was first developed in the field of radioastronomy to enhance the information obtained from noisy data (Gull and Daniell 1978). The theory is based on the same equations that are the foundation of statistical thermodynamics. Both the statistical entropy and the information entropy deal with the most probable distribution. In the case of statistical thermodynamics, this is the distribution of the particles over position and momentum space ( phase space ), while in the case of information theory, the distribution of numerical quantities over the ensemble of pixels is considered. 

The probability of a distribution of N identical particles over m boxes, each populated by nf particles, is given by 

As in statistical thermodynamics, the entropy is defined as In P. Since the numerator is constant, the entropy is, apart from a constant, equal to 

In case there is a prior probability q for box i to contain n, particles, expression (5.45) becomes 

The maximum entropy method was first introduced into crystallography by Collins (1982), who, based on Eq. (5.47), expressed the information entropy of the electron density distribution as a sum over M grid points in the unit cell, using 

---