Information and Expectation

The chemist does not labor without expectations. An experiment in which the state probabilities manifest as anticipated is valuable for the currency granted to a particular theory. Results that demonstrate otherwise are still of service and perhaps even more so. They motivate the search for a new theory or modification of the old one. 

Information offers a method of weighing expected versus contrary results. The method assigns a cost or penalty in bits for results that arrive in an unanticipated fashion. There is zero penalty charged for outcomes that conform to expectations. The penalties are substantial when the results fail wholesale to match expectations. Matters work as follows. 

Suppose that the chemist, by virtue of training and expertise, anticipated that a certain state i would rear its head 40% of the time, that is, prob t)= 0.4000. If indeed such an occurrence frequency were realized experimentally, there would be no penalty attached to the correct-in-advance expectations. The chemist foresaw the state contributing -0.4000 logj.(0.4000)/loge(2) 0.529 bits to the weighted surprisal sum, and that was indeed the case. As discussed in Section 2.1, it requires digital code—0/1, A/B, rediblack—to label states. Evidently the chemist was well prepared with the correct number of code units purchased and ready to go. 

If instead, the fractional occurrence (interpreted as probability) turned out to be 0.100, such a state would contribute only -0.1000 loge(0.1000)/loge(2) = 0.332 bits to the weighted surprisal sum. Under these circumstances, the chemist would have overestimated the bits needed for labeling the states. The chemist would have overpaid. 

One considers the reverse case. If the chemist had anticipated the fractional occurrence as 0.1000 with weighted surprisal of approximately 0.332 bits, and instead had observed 0.4000 and weighted surprisal 0.529 bits, a deficiency of code would have been encountered. The chemist would have allotted insufficient bits for the results. 

---