Kullback-Leibler divergence

The number of components necessary can usually be judged from the data, but the appropriateness of a particular value of n can be judged by comparing different values of n and calculating the entropy distance, or Kullback-Leibler divergence. 

A nonmetric measure of relative entropy is the Kullback-Leibler divergence (also called information divergence, information gain, or relative entropy) that is a measure of the difference between two probability distributions P and Q, defined as... 

Dragalin V, Fedorov V, Patterson S, Jones B (2003) Kullback - Leibler divergence for evaluating bioequivalence. Statistics in Medicine 22 913-930. 

Kullback Leibler style distances. The Kullback Leibler divergence is a measure of distance between two probability distributions. From this, we can derive an expression which calculates the distance between two spectral vectors [251], [472] ... 

Besides the lack of symmetry, the Kullback-Leibler functional has other formal limitations e.g., it is not bound, nor is it always well defined. In [60] the lack of these properties was addressed and the Jensen-Shannon divergence was introduced as a symmetrized version of Kullback-Leibler s functional. In [61] the Jensen-Shannon distribution was first proposed as a measure of distinguishability of two quantum states. Chatzisavvas et al. investigated the quantity for atomic density functions [62]. 

An important generalization of Shannon entropy, called relative (cross) entropy (also known as entropy deficiency, missing information, or directed divergence), has been proposed by Kullback and Leibler [5] and Kullback [6]. It measures the information distance between two (normalized) probability distributions for the same set of events ... 

---