Algorithmic information theory

Last, there is information theory and there is algorithmic information theory. The reader is encouraged to consult the classic text by Chaitin [15]. Wilf also presents a rigorous treatment of algorithms and information contexts [16]. Zurek has explored the subject in detail as well [17,18]. 

Chaitin, G. J. 19K7. Algorithmic Information Theory, Cambridge University Press, New York. 

Charles H. Bennett is an IBM fellow at IBM Research, where he has worked on various aspects of the relation between physics and information. He received his bachelor s degree from Brandeis University, majoring in chemistry, and his Ph.D. from Harvard in 1970 for molecular dynamics studies (computer simulation of molecular motion). His research has included work on quantum cryptography, algorithmic information theory, and quantum teleportation. He is an IBM fellow, a fellow of the American Physical Society, and a member of the National Academy of Sciences. 

Robertson, D. S. (1999). Algorithmic information theory, free will, and the Turing test. Complexity, 4(3), 25-34. Available from http //cires.colorado.edu/ doug / philosophy /i nfo8. pdf... 

Gregory Chaitin formalized this idea in his algorithmic information theory. He pointed out that while certain problems were unsolvable given a base set of axioms. 

Another way of looking at it is that Shannon information is a formal equivalent of thermodynamic entroi)y, or the degree of disorder in a physical system. As such it essentially measures how much information is missing about the individual constituents of a system. In contrast, a measure of complexity ought to (1) refer to individual states and not ensembles, and (2) reflect how mnc h is known about a system vice what is not. One approach that satisfies both of these requirements is algorithmic complexity theory. 

Coifman, R. R., and Wickerhauser, M. V., Entropy-based algorithms for best basis selection, IEEE Trans. Inform. Theory 38(2), 713-718 (1992). 

MacKay, D. J. C. Information theory, inference and learning algorithms Draft 3.1415, January 2003 www.inference.phy.cam.ac.uk/mackay/itprnn/book. html. 

PoHe78 Stephen C. Pohlig, Martin E. Heilman An Improved Algorithm for Computing Logarithms over GF(p) and its Cryptographic Significance IEEE Transactions on Information Theory 10/1 (1978) 106-110. 

R. Coifman and V. Wickerhauser, Entropy-based Algorithms for Best-basis Selection, IEEE Transactions on Information Theory, 38 (1992), 496-518. 

J. Ziv and A. Lempel, A universal algorithm for sequential data compression, IEEE Transactions on Information Theory, 23 (1977). 337-343. 

Another relevant concept within information theory, in some cases strongly related to the aforementioned measures, is the so-called complexity of a given system or process. There is not a unique and universal definition of complexity for arbitrary distributions, but it could be roughly understood as an indicator of pattern, structure, and correlation associated to the system the distribution describes. Nevertheless, many different mathematical quantifications exist under such an intuitive description. This is the case of the algorithmic [19, 20], Lempel-Ziv [21] and Grassberger... 

ViTERBi, A. J. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory 13, 2 (1967), 260-269. 

Discrete Systems Modeling Evolutionary Algorithms AND Metaheuristics Information Theory... 

D.J.C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge University Press, Cambridge, UK, 2003), http //www.inference.phy.cam.ac.uk/mackay/itila/... 

Shannon, C.E. 1948. A mathematical theory of communication. Bell Sys. Tech.. 27 379 423, 623 656. Ungerboeck, G. 1982. Channel coding with multilevel/phase signals. IEEE Trans. Inf Theory (Jan.). Viterbi, A.J. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory IT-13 260-269. 

---