What is Shannon entropy: Definition and 14 Discussions

In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent in the variable's possible outcomes. The concept of information entropy was introduced by Claude Shannon in his 1948 paper "A Mathematical Theory of Communication", and is sometimes called Shannon entropy in his honour. As an example, consider a biased coin with probability p of landing on heads and probability 1 − p of landing on tails. The maximum surprise is for p = 1/2, when there is no reason to expect one outcome over another, and in this case a coin flip has an entropy of one bit. The minimum surprise is when p = 0 or p = 1, when the event is known and the entropy is zero bits. Other values of p give different entropies between zero and one bits.
Given a discrete random variable



X


{\displaystyle X}
, with possible outcomes




x

1


,
.
.
.
,

x

n




{\displaystyle x_{1},...,x_{n}}
, which occur with probability




P

(

x

1


)
,
.
.
.
,

P

(

x

n


)
,


{\displaystyle \mathrm {P} (x_{1}),...,\mathrm {P} (x_{n}),}
the entropy of



X


{\displaystyle X}
is formally defined as:

where



Σ


{\displaystyle \Sigma }
denotes the sum over the variable's possible values and



log


{\displaystyle \log }
is the logarithm, the choice of base varying between different applications. Base 2 gives the unit of bits (or "shannons"), while base e gives the "natural units" nat, and base 10 gives a unit called "dits", "bans", or "hartleys". An equivalent definition of entropy is the expected value of the self-information of a variable.The entropy was originally created by Shannon as part of his theory of communication, in which a data communication system is composed of three elements: a source of data, a communication channel, and a receiver. In Shannon's theory, the "fundamental problem of communication" – as expressed by Shannon – is for the receiver to be able to identify what data was generated by the source, based on the signal it receives through the channel. Shannon considered various ways to encode, compress, and transmit messages from a data source, and proved in his famous source coding theorem that the entropy represents an absolute mathematical limit on how well data from the source can be losslessly compressed onto a perfectly noiseless channel. Shannon strengthened this result considerably for noisy channels in his noisy-channel coding theorem.
Entropy in information theory is directly analogous to the entropy in statistical thermodynamics. The analogy results when the values of the random variable designate energies of microstates, so Gibbs formula for the entropy is formally identical to Shannon's formula. Entropy has relevance to other areas of mathematics such as combinatorics. The definition can be derived from a set of axioms establishing that entropy should be a measure of how "surprising" the average outcome of a variable is. For a continuous random variable, differential entropy is analogous to entropy.

View More On Wikipedia.org
  1. G

    Calculating Shannon Entropy of DNA Sequences

    Unfortunately, I have problems with the following task For task 1, I proceeded as follows. Since the four bases have the same probability, this is ##P=\frac{1}{4}## I then simply used this probability in the formula for the Shannon entropy...
  2. C

    Can I improve the sinc interpolation?

    Hello everyone. I am working with mathematica, where I have developed a two-dimensional shannon interplation, just as can be seen in the slides 15 to 18 of this presentation. The code is as follows: savedX = Table[XposX = mat[[All, 1]]; YposX = mat[[All, 2]]; windXVal = mat[[All, i]]...
  3. I

    Maximal Shannon Entropy

    I have used the Lagrange multiplier way of answering. So I have set up the equation with the constraint that ## \sum_{x}^{} p(x) = 1## So I have: ##L(x,\lambda) = - \sum_{x}^{} p(x)log_{2}p(x) - \lambda(\sum_{x}^{} p(x) - 1) = 0## I am now supposed to take the partial derivatives with respect...
  4. Danny Boy

    A Von Neumann Entropy of a joint state

    Definition 1 The von Neumann entropy of a density matrix is given by $$S(\rho) := - Tr[\rho ln \rho] = H[\lambda (\rho)] $$ where ##H[\lambda (\rho)]## is the Shannon entropy of the set of probabilities ##\lambda (\rho)## (which are eigenvalues of the density operator ##\rho##). Definition 2 If...
  5. D

    Shannon Entropy vs Entropy in chemistry

    I'm wondering about the exact usage of the term entropy in programming vs. entropy in science.
  6. blackdranzer

    Entropy change : pure mixing of gases

    Consider three identical boxes of volume V. the first two boxes will contain particles of two different species 'N' and 'n'. The first box contains 'N' identical non interacting particles in a volume V. The second box contains 'n' non interacting particles. The third box is the result of mixing...
  7. V

    Help understanding Shannon entropy

    Hi I'm having some trouble understanding Shannon entropy and its relation to "computer" bits (zeros and ones). Shannon entropy of a random variable is (assume b=2 so that we work in bits) and everywhere I've read says it is "the number of bits on the average required to describe the random...
  8. T S Bailey

    Is Shannon Entropy subjective?

    If you have multiple possible states of a system then the Shannon entropy depends upon whether the outcomes have equal probability. A predictable outcome isn't very informative after all. But this seems to rely on the predictive ability of the system making the observation/measurement. This...
  9. D

    Shannon entropy of logic gate output.

    Homework Statement A particular logic gate takes two binary inputs A and B and has two binary outputs A' and B'. I won't reproduce the truth table. Suffice to say every combination of A and B is given. The output is produced by A' = \text{NOT} \ A and B' = \text{NOT} \ B . The input has...
  10. M

    Shannon Entropy and Origins of the Universe

    Hello Community, I have a question that I'm struggling to get clarification on and I would greatly appreciate your thoughts. Big bang theories describe an extremely low thermodynamic entropy (S) state of origin (very ordered). Question: Is the big bang considered to be a high or low shannon...
  11. O

    Shannon entropy - use to calculate the bit needed to encode a symbol

    To encode a symbol in binary form, I need 3 bits ,and I have 6 symbols. So I need 6*3=18 bits to encode "We are" into binary form. As shown in http://www.shannonentropy.netmark.pl/calculate My question: 3 bits to encode one then I have to use 16 bits, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. How to...
  12. K

    Shannon entropy of QXZ

    Shannon entropy of "QXZ" Hello everyone. I am trying to determine the Shannon entropy of the string of letters QXZ, taking into consideration those letters' frequency in English. I am using the formula: H(P) = –Ʃ pilog2pi What's puzzling me is that I am expecting to calculate a high...
  13. F

    Shannon entropy of wave function

    Is it ever useful to find the Shannon entropy or information content of a wave function? Thanks.
  14. S

    Shannon entropy problem

    consider a pack of 52 cards in a bridge game. a player try to convey 13 cards by nods of head or shake of heads to his partner. find the shannon entropy
Back
Top