Shannon's measure of information is the well known formula (in the discrete case):(adsbygoogle = window.adsbygoogle || []).push({});

[tex]H(S) = {-}\sum_{i=1}^N P(s_i) \log_{2}P(s_i)[/tex]

Of course this can be written as:

[tex]H(S) = \sum_{i=1}^N \log_{2}P(s_i)^{-P(s_i)}[/tex]

It appears to me that the multiple occurence of the same quantity (the probability of a particular symbol) must have some profound meaning. Why is it apowerthat is used to formulate this 'entropy' measure? In the case of throwing a six with a dice, the result of [tex]1/6^{-1/6}=1.348[/tex] and [tex]5/6^{-5/6}=22.255[/tex]. Is there some intuitive alternative association with these numbers (without direct connection with binary strings)?

Until now I found only explanations that took the logarithm for granted. IMHO the logarithm is only rescaling the "chances to the power of chances" characteristic. So, I would like to have an explanation considering that characteristic.

Or is it more or less arbitrary, as taking quadratic error measures in stead of absolute errors (in e.g. the least mean square method)?

**Physics Forums | Science Articles, Homework Help, Discussion**

Dismiss Notice

Join Physics Forums Today!

The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

# Shannon's measure - any profound explanation?

**Physics Forums | Science Articles, Homework Help, Discussion**