Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Question about Entropy/Shannon Information

  1. Jan 12, 2005 #1

    I would like second (and more) opinions on comments i received from a physicist in response to my observations on a certain experimental setup designed to make a particular point about information theory in the context of entropy.

    The experimental narrative is found on pages 9 & 10 of the following document :
    http://www.colorado.edu/philosophy/vstenger/Found/04Message.pdf [Broken]

    A diagram of what is discussed in given at : http://www.colorado.edu/philosophy/vstenger/Found/042Barmag.pdf [Broken]

    What follows is the discussion that ensued between "me" and "my respondent":

    ME :
    1) You started out with a more highly ordered state (magnets in contact N-S & S-N) .
    2) You opened the window
    3) Magnets scattered - so magnets in a more disordered state
    i.e. (entropy is increased =information is decreased)
    4) Result : yet you say "information in the system has increased by
    one bit"(p.10) i.e. (entropy is decreased= information is increased)

    How is this possible ? Why are the conclusions in #3 & #4
    different ?

    The bits at the end of the experiment tell me there is more
    information yet the system is also more disordered hence there is more
    entropy which by definition means there is less information. The
    contradiction in #3 & #4 below remains, doesn't it ? What am i missing
    here ?

    You are using an intuitive definition of entropy rather than a
    quantitative one, assuming the system is less orderly. Since entropy is
    negative information, the entropy has decreased and so the system is
    more orderly.

    My comments and questions for the Physics Help Forum :
    Why is my respondant calling my interpretation "intuitive" ? Does he mean "wrong" ?
    Is the system's final state really "more orderly" as he says ?
    Is it possible that his experiment is really not demonstrating what he thinks it demonstrates either in terms of entropy or information theory ?

    Many Thanks in advance for all comments received,

    Celal Berker
    London, England
    Last edited by a moderator: May 1, 2017
  2. jcsd
  3. Jan 12, 2005 #2

    I think your respondent is wrong when saying that the entropy has decreased. It has increased because it's the result of random action.

    But I doubt that the information has increased with the scattering. Wouldn't there be some way to retrieve the orientation of both magnets when they are on top of each other?

    And in general, it is true that entropy in general does decrease information. It may increase the quantity of data, but not necessarily information. Noise on the transmission cables, for example, usually will corrupt the bits. They may still remain as data, but the correct information has decreased as Shannon's law predicts. In your experiment, suppose the magnets fall on the opposite orientation. Isn't that a corruption of data, and hence a decrease in information?

    Last edited: Jan 12, 2005
  4. Jan 12, 2005 #3
  5. Jan 12, 2005 #4
    As I already said, I don't know if 4 bits of information result in the final system. They are random and don't make any sense. The original bits are lost by the event, so there is also a resulting loss from 2 bits to 0 bits of useful infromation.
    I'm not an expert in Information Theory and it's formulas, but if they relate entropy with any pattern of bits, I begin to doubt them.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook