1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Logarithm base 2 and bits

  1. Jan 26, 2016 #1
    I know that if you have x states then you need log2(x) bits to encode them. For example a coin has 2 states and you need 1 bit which is log2(2). It also works for numbers between 0 and 1 for example if you halve the amount of states you need to add log2(1/2) bits which is -1.

    So what does log2(i) mean? How can you have i states and encode them in log2(i) bits?

    Also on a related note why does it require -infinite bits to encode 0 states?
  2. jcsd
  3. Jan 26, 2016 #2
    Actually, we can quantify the amount of information in an event (the bits we need) as ##I = -\log_2(p)## where ##p## is the probability of that event occurring. Notice that the less likely an event, the more information contained in it.

    Thus, the amount of information required to encode the result of a coin flip is ##-\log_2(1/2)## because the probability of an event (heads, for instance) is ##1/2##.

    It makes no sense to evaluate it at ##p = i##, because probabilities must be real numbers on ##[0,1]##.

    You get an infinite amount of information in an event if the probability of that event occurring is ##0##.
  4. Jan 26, 2016 #3
    Why can probabilities not be other numbers? For example if something was guaranteed to happen twice it's probability of happening once would be 2 so when it happens the information would be -log2(2) which means that when you see an event that you know is going to happen twice happening then you actually lose one bit of information.

    Here is another example of my question: If some event has probability of 1 but then the opposite happens then the event that occured had -1 probability (for example if I think that I am going to lose my bike but then instead I get an extra bike then I though my bikes would go -1 with 1 probibility but instead they went -1 with -1 probability and since -1*-1=1 I got an extra bike) so the information in it is -log2(-1) = - 4.53236014 i which means I actually lost an imaginary amount of bits.
  5. Jan 26, 2016 #4
    No, that isn't how probability works. A basic property of a probability function ##P## is that ##0\leq P(A) \leq 1## for any event ##A##.

    If something were guaranteed to happen twice, then the probability of it happening at least once is still 1.

    Also, an event cannot have probability 1 and not happen. By definition probability 1 means the event will happen.
    Last edited: Jan 26, 2016
  6. Jan 26, 2016 #5
    And what about probability 0? Can probability 0 happen so that one gets infinite information?
  7. Jan 26, 2016 #6
    Yes, probability 0 indicates that the event in question will never happen, so that there is an infinite amount of information contained in that event (though I doubt this ever shows up in the real world).

    An example of information in events:
    There's more information in the statement "it snowed in Miami on July 4" than there is in the statement "it snowed in New York City on December 25," because the former is much less likely, and the latter is practically a given.
  8. Jan 26, 2016 #7


    User Avatar
    Science Advisor

    There is a distinction that can be drawn between probability zero and "cannot happen". One article that discusses this is https://en.wikipedia.org/wiki/Almost_surely

    For instance, if you throw a dart at an ideal dartboard, the probability that it hits at any particular chosen point is zero. Yet it must strike at some point. If you were to write down the x and y coordinates at which it strikes in binary then you would get a pair of unending binary strings. Choose the point of impact just right and one of these could match a .pdf of Encylopedia Brittanica and the other could be an ASCII rendition of the complete works of William Shakespeare.

    In that sense, the impact point of an ideal dart conveys infinite information. [Real darts in the real world do not make that much information available]
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook