Shannon Entropy Concept

We will describe below the SE formalism in detail and explain how it can be used to estimate chemical information content based on histogram representations of feature value distributions. Examples from our work and studies by others will be used to illustrate key aspects of chemical information content analysis. Although we focus on the Shannon entropy concept, other measures of information content will also be discussed, albeit briefly. We will also explain why it has been useful to extend the Shannon entropy concept by introducing differential Shannon entropy (DSE) to facilitate large-scale analysis and comparison of chemical features. The DSE formalism has ultimately led to the introduction of the SE-DSE metric. 

Here p is the estimated probability, or frequency, of the occurrence of a specific channel of data. The pi corresponds to a particular histogram bin count normalized by the sum of all bin counts, c, being the bin count for a particular bin (Eq. [2])  

Note that the logarithm in Eq. [1] is taken to base 2. Although this amounts to a simple scaling factor, it is a convention adopted in information theory so that entropy can be considered equivalent to the number of bifurcating (binary) choices made in the distribution of the data. In other words, using base 2 allows us to address this question How many yes/no decisions do we need to make for data counts to fall into specific channels or bins in order to reproduce the observed data distribution The higher the information content, the more numerous are the decisions required to place each data point. 

Start with a histogram of the molecular weights of 100 compounds... 

The SSE has an absolute minimum of 0, associated with a data distribution where all the values fall into a single bin and a maximum of 1.0, where each bin is occupied by an equal number of data counts. As we shall see, SSE is not independent of boundary effects (described later) underlying the data and there is an asymptotic relationship associated with the number of bins, which can be ignored for most practical comparisons. 

---

There are other metrics of information content, and several of them are based on the Shannon entropyAbout 10 years after introduction of the Shannon entropy concept, Jaynes formulated the maximum entropy approach, which is often referred to as Jaynes entropy and is closely related to Shannon s work. Jaynes introduction of the notion of maximum entropy has become an important approach to any study of statistical inference where all or part of a model system s probability distribution remains unknown. Jaynes entropy, or relations, which guide the parameterization to achieve a model of minimum bias, are built on the Kullback-Leibler (KL) function, sometimes referred to as the cross-entropy or relative entropy function, which is often used and shown (in which p and q represent two probability distributions indexed by k), as... 

The linear response function [3], R(r, r ) = (hp(r)/hv(r ))N, is used to study the effect of varying v(r) at constant N. If the system is acted upon by a weak electric field, polarizability (a) may be used as a measure of the corresponding response. A minimum polarizability principle [17] may be stated as, the natural direction of evolution of any system is towards a state of minimum polarizability. Another important principle is that of maximum entropy [18] which states that, the most probable distribution is associated with the maximum value of the Shannon entropy of the information theory. Attempts have been made to provide formal proofs of these principles [19-21], The application of these concepts and related principles vis-a-vis their validity has been studied in the contexts of molecular vibrations and internal rotations [22], chemical reactions [23], hydrogen bonded complexes [24], electronic excitations [25], ion-atom collision [26], atom-field interaction [27], chaotic ionization [28], conservation of orbital symmetry [29], atomic shell structure [30], solvent effects [31], confined systems [32], electric field effects [33], and toxicity [34], In the present chapter, will restrict ourselves to mostly the work done by us. For an elegant review which showcases the contributions from active researchers in the field, see [4], Atomic units are used throughout this chapter unless otherwise specified. 

Godden, J.W. and Bajorath, J. (2000) Shannon entropy a novel concept in molecular descriptor and diversity analysis. J. Mol. Graph. Model., 18, 73-76. 

Euclidean distance is essentially a measure of positive, linear correlations however, other similarity measures may be used for clustering. For example, mutual information, an information theoretic measure, may be used to capture positive, negative, and non-linear correlations all at the same time. A pictorial explanation of the concept of mutual information along with instructions on doing calculations can be found in [18]. Mutual information is based on Shannon entropy (H = — Lpi log2 pp see above explanation of entropy) and is calculated as follows M(X, Y) = H(X) + H(Y) — Ff(X, Y),... 

Our paper is arranged as a sequence of successive step>s, definitions and derivations (4)-(12) and interpretations (82)-(86) especially, stating gradually, the relation between combinatorial definition of Shannon (information) entropy and Boltzman and Clausius (thermodynamic) entropy, and, finally, resulting in (87) and (88). Although we combined the known facts about heat transformations and the Shannon s concept of an information transfer chain, this combination presented has not been used yet by another else, as far as the author of the pap>er is informed. 

This article has been organized as follows. Section 2 summarizes the notation and formulation of the main concepts used in this work it also reports the formulation of the Shannon entropy indices in terms of the seniority numbers of the Slater determinants. In Sect. 3, we present numerical values of those indices for wave functions of selected atomic and molecular systems these values allow one to characterize the compactness of the wave function expansions. The calculation level and the computational details are also indicated in this section. An analysis and discussion of these results are reported in Sect. 4. Finally, in the last section, we highlight the main conclusions and perspectives of this work. 

Shannon [20] introduced the concept of entropy in 1948. First, for simplicity, let us consider an 8 bit gray image with 256 possible intensity values. The Shannon entropy is then defined as follows ... 

We begin with a short summary of selected IT concepts and techniques to be used in diagnosing the information content of electronic distributions in molecules and in probing their chemical bonds. The Shannon entropy [3, 4] in the (normalized) discrete probability vector p = p,, ... 

To a significant extent, the theoretical basis of modern communication theory arose from the work of Claude Shannon at Bell Labs. [80]. In these seminal works, the concept of the information entropy associated with an arbitrary signal arose. In 1981, Watanabe realised the close association between entropy minimization and pattern recognition [81]. An association between entropy minimization and the principle of simplicity is also recognized [82]. The basic mathematical form of signal... 

Several descriptors are based on the concepts of information content and entropy among these are the topological information indices, indices of neighborhood symmetry, Shannon... 

As we have previously seen (see Sections 2.5, 2.6, and 3.11), the maintenance of an ordered state requires energy expenditure, energy extracted from the environment. Entropy has previously been introduced (Section 2.5) as a concept of disorder. Thus, information storage (as an ordered state) and thermodynamic entropy (as a measure of disorder) are somehow related inversely. Shannon s definition of information is (Shannon and Weaver, 1949 Gatlin, 1972 Loewenstein, 1999) ... 

The most important concept in Information Theory is Shannon s Entropy, which measures the amount of information held in data. Entropy quantifies to what extent... 

Much of the current work on biomedical image registration utilises voxel similarity measures in particular, mutual information based on the Shannon definition of entropy. The mutual information (MI) concept comes from information theory, measuring the dependence between two variables or, in other words, the amount of information that one variable contains about the other. The mutual information measures the relationship between two random variables, i.e. intensity values in two images if the two variables are independent, MI is equal to 0. If one variable provides some information about the second one, the MI becomes > 0. The MI is related to the image entropy by ... 

As shown above, in biology information and thermodynamics meet most closely and the problem of Shannon s entropy becomes especially acute [233]. If we consider a nucleotide sequence, with the sample-description space X=/(A, T, C, G) indicating the available four-fold alphabet A, T, C and G. If the chance of any base appearing at a locus is %, the relation log 4 = 2 measures its before-the-fact uncertainty. This symbol-uncertainty constitutes the basis for assigning the nucleotide sequences entropy. Thus a DNA sequence 100 units long has, assuming symbol equiprobability . Shannon s entropy of 200 bits. If we happen to know what that sequence is, then that entropy becomes information and such reasoning compromises the concept of entropy at any front. 

The concept of entropy is present in many disciplines. In statistical mechanics, Boltzmann introduced entropy as a measure of the number of microscopic ways that a given macroscopic state can be realized. A principle of nature is that it prefers systems that have maximum entropy. Shannon has also introduced entropy into communications theory, where entropy serves as a measure of information. The role of entropy in these fields is not disputed in the scientific community. The validity, however, of the Bayesian approach to probability theory and the principle of maximum entropy in this, remains controversial. 

---