1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Homework Help: Trying to understand binary entropy function

  1. Jul 20, 2010 #1
    Ok firstly I am new to statistics but have a layman's interest in entropy.

    On Wikipedia i came across this article on the binary entropy function (http://en.wikipedia.org/wiki/Binary_entropy_function).

    It says... If Pr(X = 1) = p, then Pr(X = 0) = 1 − p and the entropy of X is given by
    H(X) = -p log p - (1-p) log (1-p)

    For a a fair coin p = 1/2, so if I plug that into the equation I get

    H(X) = -0.5log0.5 - 0.5log0.5

    When I do this on my calculator I come up with H(X) = 0.301

    The actual answer should be H(X) = 1
    as this is the situation of maximum uncertainty (entropy in this case) as it is most difficult to predict the outcome of the next toss; the result of each toss of the coin delivers a full 1 bit of information.

    What am I doing wrong? Am I not putting it into my calculator correctly?
  2. jcsd
  3. Jul 20, 2010 #2
    According to the article you're supposed to use binary logarithms in the formula. You've entered plain old base 10 logarithms in your calculator
  4. Jul 20, 2010 #3
    oh ok. How do I enter binary logarithms on my calculator?

    Sorry about this, I've just realized that I should have posted this thread in the coursework forum.
  5. Jul 20, 2010 #4
    Oh - coursework: I'll get in trouble if I answer it. However, if you just think about the meaning of the binary logarithm, you should then see what the binary log of 0.5 immediately, and then see the answer to your equation without using a calculator.
  6. Jul 20, 2010 #5
    Ok I think I see what you are saying. 2 raised to the power of -1 = 0.5. So there is the 1 bit. This is not a problem I have to do for any paper or course (I study environmental planning). This is just personal study, I'm interested in getting a more thorough understanding of information entropy and how it relates to thermodynamic entropy.

    The sticky thread said that for any textbook style questions it had to go to the coursework forum. But I don't know how to shift it. I've emailed one of the moderators about it.

    On my calculator I have a log (to the base 10) button and a natural log button. The only way I know to put something in log to the base 2 is to go log(x)/log2.

    Is there a better a way to the get log to the base 2 of a number on the calculator?

    Cheers for your help though :)
    Last edited: Jul 20, 2010
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook