Weight difference between an empty and a full memory stick

[Andy Resnick]

T<snip>

For illustration, let's get back to good old Maxwell's demon. Let's say he learns information about the molecules, but does not use that information just yet. Is he decreasing the entropy of the system? Undoubtedly - he can use the information to ultimately extract energy from the system, by working that little gate. (But he had to pay for that information by spending energy himself.) This is also in agreement with the definition of entropy - his uncertainty about the system decreases. But an outside, "classical" observer won't see any decrease in entropy. How could he? Mere learning information about the molecules, even if the demon had to interact with them, won't move the system out of equilibrium - how would such a non-equlibrium state look like, anyway? The demon's knowledge does not reduce the other observer's uncertainty about the system. It's only when the demon starts playing his trick that the entropy decreases for the other observer. But that doesn't absolve the demon from paying the price beforehand.

<snip>

Well, I understand the sentiment. Perhaps it wouldn't be a problem if one remained restricted to thermodynamics, but it seems to me that in more complicated cases (like those involving information theory), one has to accept entropy as a measure of the observer's uncertainty to avoid paradoxes.

I have to think more about the first paragraph, but I have to lodge another complaint against the second:

If entropy is observer-dependent, then chemical reactions (of which the entropy is a component) are also observer-dependent; as a specific example, let's discuss Na-K-ATPase, an enzyme that hydrolyzes ATP and generates a chemical gradient. So it's superficially related to the entropy of mixing. That enzyme has been working long before anyone knew about atoms, let alone the difference between Na and K, the existence of semipermeable membranes, and the Gibbs free energy.

Given that the function and efficiency of that chemical reaction is independent of our state of information, how can the energy content (the entropy of mixing) be dependent on the observer?

 

[Andy Resnick]

Interesting discussion.

Andy, I think what you’re suggesting is that in order to change the micro state (such as Maxwell’s demon does) there has to be an input of energy. If the system is isolated such that this energy is added to a perfectly insulated, closed system, then the amount of energy must obviously increase (conservation of energy).

<snip>

We can claim that entropy equates to information but there’s a problem with this. Consider the memory stick when it comes out of the manufacturer’s plant with no information on it. Let’s say this state equates to a string of 0’s. If we put random, meaningless data on it, we now have a string of 0’s and 1’s. Using the interpretation of entropy equating to data, the state of having random data on it is analogous to removing the partition from the container that separates a gas from a vacuum or 1 gas from a different one. Similarly, we could record a book such as War and Peace onto the memory stick, and we’d have another, slightly different string of 0’s and 1’s. But this physical state is not measurably different from the random string, so we can’t say anything about the entropy difference between the random string and the War and Peace string. The data added therefore, didn’t decrease the entropy. So the conclusion we have to accept is that entropy does not equate to data, and data is only observer relative.

You correctly summarize my point in the first paragraph.

As for the second, I agree the underlying assumption, which has not been discussed well, is what is meant by an 'empty' or 'full' memory stick? I assumed that 'empty' means a information-free state, while 'full' means 'maximal information'.

Note that the proper definition of information means that given a sequence which we read one bit at a time, zero information means we can exactly predict the next bit while maximal information mean we can never predict the value of the next bit- it has nothing to do with encoding 'war and peace' or a physics textbook. It's the difference between encoding a white noise signal and a simple harmonic signal- the white noise signal has maximal information!

 

[One128]

One128 – I think you’re actually arguing 2 different points here:

- The first assumption is that this system is not closed or otherwise insulated from the environment. In this case, energy input is dissipated to the environment as heat.
- The second assumption is that even if this is true, the micro states don’t correspond to “data”. I'll add that this interpretation of data is valid regardless of whether any of the writers in a string of letters knows what the previous person wrote or not. I’ll get back to this second issue in a moment.

If we equate data on a memory stick to decreased entropy, then the question of how much mass a memory stick contains can be answered by whether or not we consider the memory stick to be a closed or open physical system and how much energy is required to reduce the entropy of the memory stick. That’s an interesting, physical interpretation of this memory stick. But I think there’s a more fundamental issue here, and that is how we define a memory stick which contains data. Per the OP.

Well, let me try to clarify: the OP does ask about the difference in weight between an empty and a full memory stick, so yes, in order to truly answer that, one should first define what "empty" and "full" means, in terms of data. But the question is related to a more general one: does the mass of a memory device depend on what data it contains? One needn't define "full" and "empty" to answer that, and I believe it needs to be answered first in any case. (And the answer can still be yes, if the memory device uses different energy states to encode 0's and 1's.)

What I was addressing here was the point raised by Andy - i.e. that the memory's mass should depend, in principle, on the "entropy" of the data it contains, implying (since mass can be measured) that the entropy of the data can be objectively measured. Kote pointed out that this isn't the case and there's no objective measure of entropy of data, as that depends entirely on the observer - which is certainly true in information theory. So the problem here was the (apparent) discrepancy between entropy in thermodynamics (which is often taken as a sort of objective quality) and Shannon entropy. I tried to explain that the seeming incompatibility boils down to thermodynamics making certain assumptions about the observer, and the two concepts can be unified (as a measure of uncertainty of the observer), so thermodynamics should be able to address questions involving Shannon entropy, but that this becomes quite tricky - you are straying away to the scary land of Maxwell's demons, where equilibrium depends on the observer (indeed, with proper knowledge, you can extract energy where no-one else can). Yet, the path from thermodynamics to Shannon entropy goes through that territory, and unless you cross it, Shannon entropy will remain Shannon urgflx to you, and thermodynamics says nothing about urgflx.

The other thing I was trying to address was how to resolve the Landauer's principle; here's where the open and closed systems come to play. There are two different situations here: when the stick is lying on the table, it's not an isolated system; we assume it to be at the temperature of the environment etc. The initial question is apparently about the stick as it is lying on the table.

The other situation is what happens when you are writing to the stick - here we assume the stick and the writer to form a closed system, for the duration of the act. And here, second law prescribes that you must spend a certain amount of energy because you reduce the entropy of the memory. So entropy is reduced, and the question is whether this leads to a lasting change in energy that will persist even when the stick is lying on the table again.

With the example of a single-bit memory device, I showed that this can't be the case. Even though every writer decreases the entropy of the memory during write, the device can't keep increasing its energy. Now, this differs from the common experience; if you decrease the entropy of something by putting it out of equilibrium and fixing it there (like you fix the written bit), it remains that way. - The resolution is that the decrease of entropy when writing the memory is the "spooky" kind; the entropy decreases for you, but not necessarily for an unrelated observer.

One might wonder - why does the second law then require you to increase your entropy (by spending a certain amount of energy), if no-one else sees a decrease in entropy of the memory? And the answer is that while you don't necessarily decrease the entropy for others, with the knowledge you gained, you could in principle do that. As Maxwell's demon learns about the molecules, this mere act doesn't yet separate them, so no-one but the demon sees the decrease in entropy. But he can use the knowledge he gained to separate them with no further work. So the second law basically requires you to pay for a license to play second-law-tricks on others. Maybe you don't want to use that license, but you have it, and a promise not to use it is not enough - that's why you have to pay. - With the memory device, it's the same situation.

 

[One128]

If entropy is observer-dependent, then chemical reactions (of which the entropy is a component) are also observer-dependent; as a specific example, let's discuss Na-K-ATPase, an enzyme that hydrolyzes ATP and generates a chemical gradient. So it's superficially related to the entropy of mixing. That enzyme has been working long before anyone knew about atoms, let alone the difference between Na and K, the existence of semipermeable membranes, and the Gibbs free energy.

Given that the function and efficiency of that chemical reaction is independent of our state of information, how can the energy content (the entropy of mixing) be dependent on the observer?

Let me see if I can address that, and let me start with a somewhat different example, that may illustrate it better, but also say how your example fits into that.

Say you have a large bottle of hydrogen, separated into two chambers. In one, the gas is hot, in the other one, the gas is cold. You exploit the temperature difference to extract energy, until finally everything is at the same temperature. The entropy is maximized, no more energy can be extracted; the system is at equilibrium.

Or is it...? Just fuse some hydrogen nuclei. - Wow! Suddenly, a portion of your gas is very hot! The system is not at equilibrium at all and you can extract lots more energy. How can that be? Has the entropy decreased, has the second law been broken?

Well, what happened there? You had a model. You made assumptions. When determining entropy, you chose a specific set of macroscopic variables and degrees of freedom. But the experiment changed along a degree of freedom your model did not anticipate. You counted the microstates, but all of them consisted of hydrogen atoms in various configurations; none assumed that the hydrogen might change into something else. - You could of course fix everything by including the new degrees of freedom in your model - but then the entropy will be different, there will be many more microstates, and indeed, the equilibrium will be at a very different point.

Does that mean that one model is better than the other, in general? No. When you're running an experiment and you describe it in physics, you always make assumptions, you always have a model that fits the particular experiment. And when defining entropy, you choose the set of macroscopic variables and microstates appropriate for that model.

If you had the bottle of hydrogen, and didn't do the nuclear fusion (or even didn't know how to do it), and it didn't start on its own, the system would be at equilibrium. Your model wasn't wrong for that case.

So the fundamental question is - what do you allow to happen in an experiment? And here, you needn't even assume an intelligent experimenter; it can be a process that "does" the experiment. The capabilities of the particular process that extracts energy determine the amount of energy that can be extracted before equilibrium is reached. The example you mention utilizes a specific way to exploit energy, so you must include that in your model - otherwise your model won't describe the situation that happens. But if it didn't happen, and you assumed it did, your model also wouldn't describe the situation. (ETA: Let me expand, to answer more directly what you were asking: the question of whether the observer can tell the difference between substances is not about whether he's able to explain Na and K atoms, but whether he can observe an experiment with a result depending on the difference.)

Perhaps one might think of some absolute entropy and equilibrium - a state where no more energy can be extracted, no matter what you do, no matter what anyone can do. But let's be honest - nowhere in thermodynamics is such an equilibrium ever reached. If the substance you have isn't iron, then you haven't even begun to extract all the energy there is. But this is considered irrelevant; instead, we stick with a certain set of thermodynamic processes and that represents our model. But we must not forget that these processes don't describe all the natural processes that can happen in the universe.

Now, one of the less common ways to extract energy better than others is to have knowledge about the details of the system. If an observer - or even a natural process - has that information, then for him, the entropy is lower, so the system really isn't at equilibrium, and that can be exploited. It's counterintuitive to classical thermodynamics, but in statistical thermodynamics, it seems a valid and consistent concept (with consequences such as Landauer's principle) - and maybe, if you consider the earlier examples, it needn't be more scary than saying that for one person, the bottle of hydrogen is at equilibrium, and for another one with a fusion reactor, it's not. The analogy is not perfect, but perhaps it gives the idea.

 

[Andy Resnick]

I agree with part of your comment- we should not think about *global* minima (or maxima), but instead only *local* extrema. I also agree that 'equilibrium' is a limiting condition, nothing is actually at equilibrium. Also, there is no such thing as a completely isolated system- and the environment is allowed to perform measurements on the state of the system.

 

[Andy Resnick]

Ok, I read a bit about Gibbs' paradox of mixing. I wonder if the paradox really refers to the existence of an equilibrium state- either way, when the partition is removed, the two gases will diffuse into each other. The essential difference is that in one case, the total volume is already at an equilibrium state while in the other situation it is not.

Using your example of hydrogen and a putative fusion event, the same concept applies- is the total volume of hydrogen at equilibrium or not? Is a gas of radioactive elements ever at equilibrium?

Clearly, there is no *global, absolute* minimum of stability that can be reached in a finite time. As I tell my students, "how long are you willing to wait?" The corollary to this statement is that a thermodynamic temperature for any system is an approximation, since there is no equilibrium state.

This is not really a practical problem- we can assign timescales based on physical processes, and if the system does not change much over a timescale, we assign the system to a thermal equilibrium state. Onsager's relations are a linearization of the nonlinear dynamic problem.

And it's not even as bad as that- we can measure the change in free energy of mixing directly using a calorimeter, and we may even may try to *predict* a change based on the state of our knowledge about the system (which is slightly different than the information in a signal). We do the experiment, and the number does not agree with our prediction. That doesn't mean the system violates any laws of physics, it means that until we can reproduce the measured results, we did not account for some aspect of the experiment.

So, one portion of the energy of mixing was unaccounted for- the information content of the distribution of gas molecules. The activation energy for fusion/fission. In biology, another accounting of mixing was discovered- chemi-osmotic forces. The discovery of these new forms of energy *transmogrification* (?) didn't invalidate Joule's experiments, for example. Joule may have assigned these effects as an 'error' to be minimized by suitable design of the experiment.

There's another Gibbs paradox involving wetting... interestingly, it also involves equilibrium states.

 

[kote]

Note that the proper definition of information means that given a sequence which we read one bit at a time, zero information means we can exactly predict the next bit while maximal information mean we can never predict the value of the next bit- it has nothing to do with encoding 'war and peace' or a physics textbook. It's the difference between encoding a white noise signal and a simple harmonic signal- the white noise signal has maximal information!

Agreed... almost. We just have to remember that whether or not we can predict the next bit has nothing to do with the previous bits. It has to do with our own expectations. If you are sure that your bit-creating source is random, and you start seeing a simple harmonic signal, you still can't predict what your next bit will be. If you are sure your source is random, then no matter what string of bits you actually end up with, you have maximal information. A random source can produce literally any string of bits, so any string of bits has the potential to represent maximal information. The string itself is irrelevant to its information content.

 

[Nomdeplume]

I find these discussions of entropy quite interesting since I've struggled with the some aspects of the concept. I'm better versed in philosophy than physics however the learned opinions on this forum is a good way to change that. Ok now the platitudes and caveats are out of the way...
I tend to agree with the observer dependency of entropy, but then don’t we get into a quandary concerning open and closed systems? Given the 2nd law of therm. appears to be valid only in isolated systems, wouldn’t observers ad infinitum violate that state?
I think the example of the book is good to pursue. It appears language/observer dependent. Let’s expand the example to say I tear up all the individual letters in the book since it is written in an Navajo and I am literate only in English but I then rearrange the words into English. Has entropy increased or decreased? Szilárd tries to give us a way out of this since my very act of tearing up the pages means that I have increased the energy within the system.
However given that there are very few people that read Navajo haven't I decreased entropy if I've greatly increased the amount of people now able to read it (decrease in total amount of energy expended by observation) or is it only if many more people do read it, not simply are able to read [potentiality vs. actuality]. For someone who reads Navajo it has certainly increased. It an be argued that the thing in itself, the book, has had no change in entropy although it the information contained there-in might not even be close to the original.
I certainly appreciate any input and being pointed in the right the direction to better verse myself in concepts I may have misunderstood.

 

[Andy Resnick]

Agreed... almost. We just have to remember that whether or not we can predict the next bit has nothing to do with the previous bits. It has to do with our own expectations. If you are sure that your bit-creating source is random, and you start seeing a simple harmonic signal, you still can't predict what your next bit will be. If you are sure your source is random, then no matter what string of bits you actually end up with, you have maximal information. A random source can produce literally any string of bits, so any string of bits has the potential to represent maximal information. The string itself is irrelevant to its information content.

Well, we add Markov processes to allow for memory/history, that's not a big deal. Think about this: how many numbers (i.e. encoding) does it take to represent a sine wave? How about several sine waves? How about an image?

All of these require fewer numbers than a random noise signal.

Decoding the signal is not a measure of the information of the signal. And your last sentence is a good working definition of 'many thermal microstates corresponding to a single macrostate'.

 

[Andy Resnick]

<snip>
I tend to agree with the observer dependency of entropy, but then don’t we get into a quandary concerning open and closed systems? Given the 2nd law of therm. appears to be valid only in isolated systems, wouldn’t observers ad infinitum violate that state?
I think the example of the book is good to pursue. It appears language/observer dependent. Let’s expand the example to say I tear up all the individual letters in the book since it is written in an Navajo and I am literate only in English but I then rearrange the words into English. Has entropy increased or decreased? <snip>

Perhaps I should point out that the term 'information', as used in the context of physics, may be different that our everyday usage of the term. Another example of this is the term 'work'. Work is not the same thing as physical exertion.

So, instead of comparing languages, let's just discuss a book written in plain ol' english. Then, I take the text and replace every letter 'w' (since 'w' does not occur that often) with a randomized substitution. Chances are, you can still read the book- the information has only slightly changed. And there are several 'states' of the book that correspond to the same amount of information, since the substituted letters were chosen randomly.

Now do the same thing, but additionally, replace the letter 'e' with random substitutions. It may be harder to read the book. Again, there are many equivalent 'microstates' of the book, but they can more or less all be understood.

Hopefully you can see what happens as more and more letters (including spaces) get randomized. It is in this sense that the entropy of the book increases, and that the information (in the sense used in physics) also increases. Even though the book is less readable.

Since that's counterintuitive, sometimes people discuss the 'negentropy' of a signal, because that is how we think of information.

Now, in terms of not knowing how the information is encoded (i.e. using different languages, or even jargon), it's not clear to me how to quantify the amoiunt of information present. To some degree it doesn't matter- the idea of Dirac notation in quantum mechanics is great, because we don't *need* to know any detailed information about the system in order to describe it's dynamics.

 

[Q_Goest]

Sigh. If the entropy is different, the mass must also be different.

Let’s consider this more closely. At first I disagreed, but now I think Andy is correct.

“Consider a closed container in which a region R is marked off as special, say a bulb-shaped protuberence of one tenth of the container’s volume having access to the rest of the container through a small opening. Suppose that there is a gas in this container consiting of m molecules.” (Penrose) In other words, consider two situations. Consider a cylinder of volume 10V attached by a valve and short bit of tube to a second spherical container of volume 1V. Consider also there being a gas present inside these containers. Now consider these two separate situtions:
Situation 1: All the gas (total of m molecules) is in the very small, spherical container.
Situation 2: The gas (total of m molecules) is spread evenly throughout the pair of containers.

We’ll assume the set of containers has come to equilibrium at a given temperature T in both cases. So in both cases, there is a total of m molecules at an average temperature T.

Which one has more ‘mass’? Earlier, kote suggested a compressed spring has more mass than a spring that isn’t under stress. The compressed spring has more mass because:

It gets stored as chemical potential energy as the chemical bonds are stretched beyond where they are stable .

I agree. So what is the difference between situation 1 above and the situation where the spring is compressed? Similarly, what is the difference between situation 2 above and the situation where the spring isn’t under stress?

If we claim the spring has more mass because it is compressed, then the gas has more mass when it resides in the smaller spherical container and when entropy is lower. In fact, ANY time the gas has a lower entropy in the example provided by Penrose above, the system should have more mass. The lower the entropy the higher the mass.

Agreed?

That’s not the end of the problem though. I would still claim the memory disk has the same mass regardless of the micro state as long as there is no potential energy bound up by any part of the memory disk in order to store information. In fact, one could in principal, store more information (ie: you could store a sine wave) with a totally random string of 0's and 1's (by simply interpreting random information such that it equates to a sine wave). We could have a Batman decoder that took the random "information" of 0's and 1's and converted them into a sine wave. In this case, I’m using the term information the same way Andy is using it here:

As for the second, I agree the underlying assumption, which has not been discussed well, is what is meant by an 'empty' or 'full' memory stick? I assumed that 'empty' means a information-free state, while 'full' means 'maximal information'.

Note that the proper definition of information means that given a sequence which we read one bit at a time, zero information means we can exactly predict the next bit while maximal information mean we can never predict the value of the next bit- it has nothing to do with encoding 'war and peace' or a physics textbook. It's the difference between encoding a white noise signal and a simple harmonic signal- the white noise signal has maximal information!

If there is no energy associated with one information state A when compared to another information state B, then the two states have the same mass. If we compare a wooden stick in one of two positions or a car parked in garage A instead of garage B, there is no additional energy stored by those systems, so the two states are equivalent in terms of energy stored and thus their total mass.

We could put a spring on a wooden stick such that when depressed it stored energy. Or we could put a spring inside garage A or B such that the car stored energy in that particular garage. And we could have a very long string of wooden levers, or a very long string of cars that sat in garage A or B, but the total amount of stored energy would not depend on how we interpreted the "information" contained in the string. The amount of stored energy would only depend on how many cars were parked in garages with springs in them or how many wooden levers were depressing springs. And we could interpret these two states in any way whatsoever. We could make the springs correspond to a "full" memory disk or an "empty" one. So although entropy might influence mass, the information content that we take away from the system has nothing to do with mass, energy or entropy. The information content depends on how we interpret the physical state.

 

[Andy Resnick]

<snip>

That’s not the end of the problem though. I would still claim the memory disk has the same mass regardless of the micro state as long as there is no potential energy bound up by any part of the memory disk in order to store information. <snip>

If there is no energy associated with one information state A when compared to another information state B, then the two states have the same mass. If we compare a wooden stick in one of two positions or a car parked in garage A instead of garage B, there is no additional energy stored by those systems, so the two states are equivalent in terms of energy stored and thus their total mass.

<snip>

But that's the crux of the issue, isn't it? In fact, the wooden stick may have very different energies associated with it (if, for example, the height changed and gravity is present). And since energy is required to both read and write information in a memory device, leading to a change in the macrostate of the device (since the two configurations are distinguishable), the internal energy (alternatively, the configurational energy, the infomation content, the entropy...) of the memory device has been changed.

 

[One128]

The lower the entropy the higher the mass.

Agreed?

Yes, this is true, as long as certain assumptions are satisfied (namely the fundamental assumption of statistical mechanics, i.e. all microstates being equally likely) and the temperature is constant.

It can be also looked at in this way: increasing entropy means that some of the usable energy gets converted to heat. This leads to an increase in temperature while total energy remains the same (1st law). So in order to decrease the temperature to what it was, you need to take some heat away, lowering the energy.

So although entropy might influence mass, the information content that we take away from the system has nothing to do with mass, energy or entropy. The information content depends on how we interpret the physical state.

Well, this is one of the ways to approach the issue: insisting on classical thermodynamic interpretation of entropy, thus entirely separating entropy (in thermodynamics) from information (or Shannon entropy). It is partly satisfying in that it explains why the energy of the memory is not fundamentally required to change when the stored information gets changed.

But it is not fully satisfying, because ultimately, entropy in thermodynamics and entropy in information theory are related, and handling information can have real thermodynamic consequences on energy and entropy (as, once again, the Landauer's principle shows). I've tried to explain in detail how that works and why entropy and information can be unified as a single concept, while still reaching the same conclusion about the memory - that the energy needn't depend on the data.

But that's the crux of the issue, isn't it? In fact, the wooden stick may have very different energies associated with it (if, for example, the height changed and gravity is present). And since energy is required to both read and write information in a memory device, leading to a change in the macrostate of the device (since the two configurations are distinguishable), the internal energy (alternatively, the configurational energy, the infomation content, the entropy...) of the memory device has been changed.

We need to be very careful with how we define entropy, i.e. how we choose the macrostates and microstates.

1. If each distinct memory state represents a different macrostate, then the following are true:
- Reading data does not change the entropy of the system in any way (!), because it doesn't change the number of microstates associated with that particular macrostate or their probability distribution, thus by definition the entropy remains the same.
- Changing data is a change in macroscopic variables. It could cause an increase in entropy, decrease in entropy, or entropy could stay the same - again, all depends on the microstates associated with the particular macrostate. This is completely equivalent (!) to saying that different data can be encoded with different energy levels - higher, lower, or the same.
- The entropy of the memory device is a function of temperature and energy and tells us nothing about Shannon entropy of the data for some observer.
- The entropy (thus defined) of the data is zero - macrostates are by definition known, there is no uncertainty about them.

2. If a macrostate corresponds to a collection of memory states, each representing a microstate of such a macrostate, then the following are true:
- Reading or changing data can decrease the entropy (!) by eliminating uncertainty about the system (ruling out some microstates). Entropy can't increase in this way.
- Lowering the entropy of the system has thermodynamic consequences for the observer - his entropy must increase to compensate (energy kT ln 2 must be spent for each bit of uncertainty eliminated). However, it is in principle also possible to alter data without a decrease in entropy (reversible computing).
- The entropy is equivalent to Shannon entropy of the data for the same observer, and tells us nothing about temperature and energy of the memory device.

 

[Andy Resnick]

<snip>
We need to be very careful with how we define entropy, i.e. how we choose the macrostates and microstates.

1. If each distinct memory state represents a different macrostate, then the following are true:
- Reading data does not change the entropy of the system in any way (!), because it doesn't change the number of microstates associated with that particular macrostate or their probability distribution, thus by definition the entropy remains the same.
- Changing data is a change in macroscopic variables. It could cause an increase in entropy, decrease in entropy, or entropy could stay the same - again, all depends on the microstates associated with the particular macrostate. This is completely equivalent (!) to saying that different data can be encoded with different energy levels - higher, lower, or the same.
- The entropy of the memory device is a function of temperature and energy and tells us nothing about Shannon entropy of the data for some observer.
- The entropy (thus defined) of the data is zero - macrostates are by definition known, there is no uncertainty about them.

2. If a macrostate corresponds to a collection of memory states, each representing a microstate of such a macrostate, then the following are true:
- Reading or changing data can decrease the entropy (!) by eliminating uncertainty about the system (ruling out some microstates). Entropy can't increase in this way.
- Lowering the entropy of the system has thermodynamic consequences for the observer - his entropy must increase to compensate (energy kT ln 2 must be spent for each bit of uncertainty eliminated). However, it is in principle also possible to alter data without a decrease in entropy (reversible computing).
- The entropy is equivalent to Shannon entropy of the data for the same observer, and tells us nothing about temperature and energy of the memory device.

Maybe we need to more carefully distinguish between the amount of entropy and *changes* to the entropy. While I agree there is probably no way to unambiguously assign an absolute value of entropy to a system, the (lower bound to a) *change* of entropy when a system undergoes a process is possible to unambiguously assign.

 

[One128]

Maybe we need to more carefully distinguish between the amount of entropy and *changes* to the entropy. While I agree there is probably no way to unambiguously assign an absolute value of entropy to a system, the (lower bound to a) *change* of entropy when a system undergoes a process is possible to unambiguously assign.

Only for the same set of macroscopic variables and the same assumptions about the observer. The change of entropy does depend on how we specify it. As noted in the example I just wrote, in one instance reading the data does not decrease the entropy, in another instance it can.

Once again, you can try to avoid that by restricting entropy to its specific thermodynamic interpretation, but that means making assumptions that will a) make the term "entropy", restricted in this way, inapplicable to entropy of the data, b) in this interpretation the entropy of the memory device doesn't depend on the data in any fundamental way - i.e. the lower bound to a change of entropy is zero. This corresponds to scenario 1 in my last post.

 

[One128]

Q_Goest,

I apologize, I skimmed through your post too fast and I overlooked the details of your particular setup. Upon closer reading, I realize that it's actually not that straightforward to answer:

In other words, consider two situations. Consider a cylinder of volume 10V attached by a valve and short bit of tube to a second spherical container of volume 1V. Consider also there being a gas present inside these containers. Now consider these two separate situtions:
Situation 1: All the gas (total of m molecules) is in the very small, spherical container.
Situation 2: The gas (total of m molecules) is spread evenly throughout the pair of containers.

We’ll assume the set of containers has come to equilibrium at a given temperature T in both cases. So in both cases, there is a total of m molecules at an average temperature T.

Which one has more ‘mass’?

This can't be answered given only the details you give. If the gas was an ideal gas, 1 and 2 would have the same mass. If the gas was for example helium, 1 would have more mass, and if it was for example oxygen, 2 would have more mass.

First, let the gas in state 1 undergo free adiabatic expansion. The energy does not change during this process, as the system is isolated. The temperature of ideal gas will not change during free expansion, so the system will be directly in state 2 - more entropy, same temperature, same energy.

If the gas is helium, it will heat during free expansion due to Joule-Thomson effect. So to reach state 2 at temperature T, you must take away some heat, so energy (and mass) of state 2 will be lower. - Similarly, oxygen will cool during free expansion, so heat must be added, resulting in state 2 having more energy (and mass).

In case you're asking how this fits together with the seemingly contradictory statement "more entropy at same temperature means less energy", the answer is that that statement assumes there is no change in volume and pressure. But in your setup, volume and pressure are allowed to change. - See here for how the quantities are related.

 

[Andy Resnick]

Only for the same set of macroscopic variables and the same assumptions about the observer. The change of entropy does depend on how we specify it. As noted in the example I just wrote, in one instance reading the data does not decrease the entropy, in another instance it can.

Your second paragraph does not follow from the first. The first paragraph is equivalent to the Clausius-Duhem inequality, with the equality representing 'reversible' computing.

http://en.wikipedia.org/wiki/Clausius–Duhem_inequality

 

[One128]

Your second paragraph does not follow from the first. The first paragraph is equivalent to the Clausius-Duhem inequality, with the equality representing 'reversible' computing.

I don't understand what you're referring to (i.e. what doesn't follow from what and why). If by "first paragraph" you mean the one you quoted, then I don't see how the statement that change in entropy depends on how you define entropy has anything to do with Clausius-Duhem inequality.

 

[Pythagorean]

If all 00000000's has the same mass as all 11111111's, then any combination in between should have the same mass I am guessing.

Think about it this way: It takes less infromation to say "all 1" or "all 0" then it does to say "four 1's and four 0's alternating" If the combination gets much more complex (like 101101001 or something) then it takes even more information to describe the whole set, because the order becomes a more complex.

 

[Andy Resnick]

I don't understand what you're referring to (i.e. what doesn't follow from what and why). If by "first paragraph" you mean the one you quoted, then I don't see how the statement that change in entropy depends on how you define entropy has anything to do with Clausius-Duhem inequality.

The Clausius-Duhem inequality is a statement regarding allowable processes on physically realizable systems. In a way, it's similar to the second law of thermodynamics. Some people consider the inequality as a proper thermodynamic (as opposed to thermostatic) statement of the second law.

It's important to remember that the existence of state variables (or even the existence of a thermodynamic state) comes before the definition of reversible, cyclic processes, not the other way around. Also, it's important to realize that the flow of heat can have material effects other than a change in temperature.

So, yes- the actual, measured change in entropy acummulated after a process has occurred depends on the specific process that takes a system from state 1 to state 2. However, the lower bound (the equality in the equation) of the change in entropy is invariant to coordinate changes, satisfies the principle of material indifference-the response of a material is independent of observer- and also satisfies the principle of equipresence (all observers agree on the set of independent variables).

"entropy" is not restricted to use with reversible, slow cyclic processes; equilibrium states, or any of the usual thermostatic arguments. Neither is 'temperature'.

 

[One128]

I'm not sure what point you're trying the make here.

It's important to remember that the existence of state variables (or even the existence of a thermodynamic state) comes before the definition of reversible, cyclic processes, not the other way around. Also, it's important to realize that the flow of heat can have material effects other than a change in temperature.

So, yes- the actual, measured change in entropy acummulated after a process has occurred depends on the specific process that takes a system from state 1 to state 2. However, the lower bound (the equality in the equation) of the change in entropy is invariant to coordinate changes, satisfies the principle of material indifference-the response of a material is independent of observer- and also satisfies the principle of equipresence (all observers agree on the set of independent variables).

If I understand correctly, you seem to be essentially saying the participating in a process that involves flow of heat can have entropy effects on the material. Putting aside whether that is the case or not - how would that support any point of yours?

Clausius-Duhem inequality, like the 2nd law, only works in one direction - up, towards irreversibility, towards the increase of entropy. But your claim was that physical entropy decreases (and mass increases) with "information". It should therefore be obvious that you can't invoke Clausius-Duhem inequality to justify the alleged effect.

And even if we ignored that the direction is wrong, this wouldn't make anything depend on the data there is, but at best on what the memory has been through. But that has nothing to do with the subject of the thread.

 

[kote]

Andy, what do you propose happens to the mass of a system when a bit is changed from a 1 to a 0 and then back to a 1? You have been arguing that there is a lower bound to the change of entropy/energy when a system changes states. Is it not implied that the lower bound must be greater than zero?

A system in the exact same state must have the same mass, no? So no mass can be added to a system in an irreversible way simply by flipping bits. There can't be anything inherent to the process of flipping a bit that adds mass to the system. If both states 0 and 1 are at the same energy level, as is the case with objects that are simply moved horizontally from one place to another, then the content of the drive, the sequence of the 1s and/or 0s, would be totally irrelevant to the drive's mass.

I don't see where entropy even enters the discussion unless entropy is supposed to allow us to ignore conservation of energy. Ignoring inefficient losses, which are not stored within the system, no net work is done when moving an object horizontally or flipping bits between two equivalently energetic states.

Am I missing something here?

 

[Pythagorean]

A system in the exact same state must have the same mass, no? So no mass can be added to a system in an irreversible way simply by flipping bits. There can't be anything inherent to the process of flipping a bit that adds mass to the system. If both states 0 and 1 are at the same energy level, as is the case with objects that are simply moved horizontally from one place to another, then the content of the drive, the sequence of the 1s and/or 0s, would be totally irrelevant to the drive's mass.

Objects moved horizontally from one place to another are operating under the force of gravity, a conservative force. It just means there's no work done against gravity. There's plenty of other force involved in the real world so we do actually do work when we move an object from one place to another (we have to overcome it's inertia twice: once to start it in motion and once to stop it.)

In real hard drive, you have complicated solid state physics going on. I have no idea how they store 1's and 0's in a hard drive, but in order to make them distinguishable from each other, I assume they'd have to occupy different energy states.

When I built simple logic circuits, we used a voltage of 5 V to represent the 1 and some millivolts to represent the 0. Even if you assumed that we always had the "same" power through the relationship P = IV (the current goes up to compensate for the low voltage, so the power ideally stays the same) you'd have to realize that the dynamic process of switching from high voltage - low current to high current - low voltage are not equivalent once you consider the added problems of thermodynamics and entropy.

So you can't just consider the final state anyway, you also have to consider the dynamic process that allowed the states to change to where they are in the final state.

 

[kote]

So you can't just consider the final state anyway, you also have to consider the dynamic process that allowed the states to change to where they are in the final state.

The current mass of a system depends only on its current state, not on the processes that led to its current state. If the current state is the same as a previous state, then the mass is the same as it was in the previous state.

Also, the moving horizontally thing is hypothetical. What's more important is that when you flip a bit you can flip it back and end up in the same state you started in. Of course work is done in the real world to flip a bit and flip it back, but that work is all lost as heat to the environment and doesn't change the state of the system besides heating it temporarily.

 

[Pythagorean]

The current mass of a system depends only on its current state, not on the processes that led to its current state. If the current state is the same as a previous state, then the mass is the same as it was in the previous state.

Also, the moving horizontally thing is hypothetical. What's more important is that when you flip a bit you can flip it back and end up in the same state you started in. Of course work is done in the real world to flip a bit and flip it back, but that work is all lost as heat to the environment and doesn't change the state of the system besides heating it temporarily.

Actually that work is lost in addition to work lost through heat from solid state collisions in the conducting material. We're talking about a system of bits, not a single bit (but even with a single bit, the work to from 0 to 1 is not necessarily the same as to go from 1 to 0. I so no reason at all to assume only conservative forces are involved).

Anyway, the system is more ordered (has less entropy) for the empty disk drive. The system is less ordered (has more entropy) for the full disk drive. Everything else being constant, the difference in entropy implies a difference in energy (Gibb's).

A difference in energy means a difference in mass (E=mc^2)

 

[Andy Resnick]

I'm not sure what point you're trying the make here.

<snip>

This thread has developed into a long and winding road, so sure- a recap is in order.

My first post (#9), I answered the OP in the following way:

"since the empty (which I interpreted to mean devoid of information) memory device has less entropy than the full memory device- one bit of information is associated with kT ln(2) units of energy- the energies are different and so the mass is different"

This caused several objections to be raised, mostly along the lines of "Wait! Information content is observer-dependent, and besides, the 'information entropy' is different than the 'thermal entropy' and so the 'information entropy' does not correspond to a difference in energy."

Every post I have made since the original post has been addressing the various objections: first, the information content of a signal is different than the *encoding* of information in a signal (a subtle point), that 'information entropy' is no different than 'thermal entropy', and that the information content is not observer-dependent. Sometimes, my explanations are more clear than other times.

My line of reasoning follows that of rational continuum mechanics and rational thermodynamics, subjects that are sometimes unfamiliar. The advantage is that the arguments and conclusions are material- and process-independent

 

[Q_Goest]

Hi Andy, Thanks for the recap. I’d like to try a recap for myself and perhaps find out if I’m missing a point. Note that one axiom I think we’ve all taken for granted here is that this “memory card” is a hypothetical physical system. No one yet seems to have made the point of narrowing this down to an actual memory card that might be used on a conventional computer. I’m fine with that actually, though it would be nice to understand how a conventional memory card works.

Another axiom I think we’ve all been assuming is that ‘weight’ in this context is a summation of mass plus energy. I’ll keep using this axiom unless it’s challenged.

1. I’m sure we all agree that to decrease entropy, energy must be added. When that happens, the total mass plus energy for this system increases. To answer the OP in this case is to say that to decrease entropy, weight will increase for this closed system. (I don't know yet if this has anything to do with information or not yet.)

2. What I’m not sure about is that given we isolate this memory card from the environment and it undergoes a decay with no energy input, and assuming the physical entropy increases, the memory card might decrease in equivalent mass. In this case, mass plus energy is conserved, so I’m not sure one can claim that a simple increase in entropy of a closed system will necessarily lead to a decrease in that system’s total weight or mass. That problem needs to be addressed separately.

3. Another point I’ve seen suggests that if energy is added to a closed and isolated system, then it doesn’t matter if entropy increases or decreases for that system. The end result is that ‘weight’ must increase. This is obviously problematic if one wants to suggest that weight decreases when energy is added and entropy decreases. Given ‘weight’ being mass plus energy, the addition of energy requires an increase in weight regardless of whether entropy increases or decreases unless #2 above can somehow be proven.

4. Yet another point suggests that information entropy may or may not correspond to thermodynamic entropy. I suspect folks going into physics aren’t very familiar with “information” entropy. I don’t have any idea what information entropy is but I suspect it has nothing to do with this. I haven’t seen anyone yet quote a paper to defend this correlation. There have been many quotes of the literature, but I don’t see a single one that really brings this argument out and properly defends it one way or the other.

5. I’d like to add one more stick to the fire. Assuming we are considering this “memory stick” to be a hypothetical system as opposed to a real memory card, I’d like to resort back to the true, thermodynamic definition of entropy as was discussed earlier. Thermodynamics uses the concept of control volumes. This concept is a philosophical one really, as it has considerable unwritten assumptions. Those assumptions include that a control volume is unaffected by anything external to the control surface. This follows from nonlocalilty. Nothing can influence the going’s on inside a control volume without some causal influence passing the control surface. We can break up any given closed or open physical system into control volumes and show that the entropy within any given control volume is independent of what’s happening external to the control surface. Given this is true, the entropy of a switch in one of two possible positions is independent of anything external to the control surface. Since this is true, the entropy of a memory card is a simple summation of the system's individual control volumes. And if the system's total entropy is a simple summation of the entropy of it's individual control volumes, then any information we claim the system has is observer relative (or observer dependent).

There might be another argument that’s been posted, but I’m starting to fade…

 

[Andy Resnick]

Q_Goest,

Your points are well-taken. Memory sticks use something called "flash RAM', similar to an EEPROM. I looked up how they work, but I don't really understand it other than (1) power is not required to maintain the memory state, (2) an empty state is '1', and (3) the difference between '1' and '0' is the level of current. The only other revelant piece of information is that they can withstand a limited number of erase-write cycles, but possibly an infinite number of read cycles. That is a significant fact, and gives clues about the relevant thermodynamics.

A minor point- mass and energy are *equivalent*. 'Weight' is not mass plus energy, it's mass/energy plus gravity.

For point (2), it's not obvious what will happen. Clearly, writing or reading involves interaction with another object, but then the memory stick can be unplugged and stored in a drawer for years. The question is what is the fidelity of the encoded data over that time. I suspect that the contents of the unplugged memory stick can be pretty well considered isolated from the environment, but again, if the data becomes degraded, that indicates the system is not in perfect isolation.

For point (4), I'd recommend reading Shannon's original paper (1949?) as it is quite readable. It's crucial to understand how 'information' is quantified. White noise has a lot of information, while a pure tone does not. A randomly-generated string of ASCII has maximal information, but readable text does not. That is counterintuitive, but nonetheless, that's the definition.

Points (3, 5) require a bit of thought to answer. You are right that a control volume is a fundamental tool of thermodynamics, but it's no more mysterious than the control volumes used to prove Gauss's law. In order to change the state of a bit, something must flow through the boundary of the control surface.

'Entropy' is, unfortunately, as ill-defined as 'heat' and 'energy'. What *is* energy? I have no idea. I do know that the total energy of an isolated system is conserved, but the energy can be partitioned into a 'work' part (PV, electromagnetic, chemical, information, etc.) and a 'heat' part. I don't know what 'heat' is either, but I know that it can act on a body to either change the temperature (the specific heat) or effect some other change (latent heat- which is not limited to a simple change of phase), or some combination. What 'heat' cannot do (by itself) is perform work. 'Heat' is related to 'entropy' while 'work' is not. If I do work on a system to organize it, to structure it, or to somehow select out a specific state, then I have decreased the entropy of that system (at the expense of increasing it somewhere else).

An example I like to use in class is the folding of a protein: on order for a protein to be functional, it must assume a specific folded configuration. Proteins (like other living systems) operate at constant pressure and temperature- this is why they are good model systems for illustrating certain concepts. What is the difference of energy between a folded and unfolded protein? What about the entropy? How is this energy encoded?

Instead of thinking of the internal energy of a system as the potential energy, or the mass energy, it can be thought of as 'configuration energy': the configuration of a system has energy associated with it. Folded and unfolded proteins have different configurations, so they have different energies. Unfolded proteins are much more mobile (structurally), and so they have a higher entropy (larger number of equivalent states).

So it is with the state of a memory device. Since EEPROMS have a limited number of erase-write cycles, the entropy of the memory must be increasing each time a bit is written- I do not know how this entropy is manifested (changes in resistance, noisier signals, some other effect), but the overall entropy of the device must increase. Reading the data is interesting, and I'm not sure how to understand that process yet.

 

[Pythagorean]

Are you a physical chemist Andy? (Or a chemical physicist?)

 

[kote]

In the original Shannon paper (http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf ), "bits" are the unit used to measure information.

The choice of a logarithmic base corresponds to the choice of a unit for measuring information. If the base 2 is used the resulting units may be called binary digits, or more briefly bits, a word suggested by J. W. Tukey. A device with two stable positions, such as a relay or a flip-flop circuit, can store one bit of information.

Information is measured in bits. 8 gigabits of information is 8 gigabits of information.

Since a flash drive has a static number of bits, it is always storing the same amount of information. The amount of information, being constant, is irrelevant to any changes in the mass of the drive.

The rest of the article talks about compressing predictable patterns down to a minimum number of bits, and how many bits are required to represent different patterns. All of that has to do with random sources and partially predictable results, none of which seem to apply here.

 

[Stonebridge]

Anyway, the system is more ordered (has less entropy) for the empty disk drive. The system is less ordered (has more entropy) for the full disk drive.

To get back to the original question then; how exactly do you define empty and full in this context?
If I "filled" a disk drive with one large graphic file would it be full?
If I then "deleted" that file, would it be empty?
If the drive was empty before I stored the file, in what sense is it "empty" after file deletion.
Do we have two types of "empty"?

 

[kote]

To get back to the original question then; how exactly do you define empty and full in this context?
If I "filled" a disk drive with one large graphic file would it be full?
If I then "deleted" that file, would it be empty?
If the drive was empty before I stored the file, in what sense is it "empty" after file deletion.
Do we have two types of "empty"?

In practice, flash drives don't overwrite old data until they have to. They find an empty spot on the drive and put new data there. Once everything is full, they go back and wipe parts of the drive as needed before rewriting new data. When something is deleted, its bits are left as they are and some small portion of metadata changes to say "this junk here isn't a file anymore." When you delete a 10gb file you may only flip 3 bits in the drive.

Even after you delete a file, that portion of the drive won't be used again until the factory-fresh portions are all used up. Because of this, there are not portions of the drive that ever stay unused, and the drive never returns to having portions that are all 1s or all 0s.

So, in practice, deleting a file has very little impact on the overall sequence of bits in the drive, which, after an initial break in period, remains in an apparently chaotic/random state over the entire drive regardless of the actions of the user. Unless this is specifically played with by the user, after a break in period, flash drives will always have near 50/50 1s and 0s in an apparently random order.

They are designed to work this way because of the limited lifetime mentioned by Andy. The drive lasts longest when the data writing is spread evenly over the entire drive and when bits are only flipped as needed.

 

[Andy Resnick]

Are you a physical chemist Andy? (Or a chemical physicist?)

Heh... I stopped labeling myself a long time ago. My dissertation was straight-up physics; fluid mechanics and optics. I guess people call me a biophysicist now... I just tell people I'm a scientist.

 

[Andy Resnick]

<snip>

The rest of the article talks about compressing predictable patterns down to a minimum number of bits, and how many bits are required to represent different patterns. All of that has to do with random sources and partially predictable results, none of which seem to apply here.

Setting aside the choice of base (perhaps base 255 would be better for ASCII, for example, or base 10 for decimal numbers), the paragraph above is exactly why image compression (or data compression in general) is analyzed in terms of 'entropy', 'lossless', etc. Again, the information content of a signal is different than how that signal is encoded.

 

[Pythagorean]

To get back to the original question then; how exactly do you define empty and full in this context?
If I "filled" a disk drive with one large graphic file would it be full?
If I then "deleted" that file, would it be empty?
If the drive was empty before I stored the file, in what sense is it "empty" after file deletion.
Do we have two types of "empty"?

I believe we've laid out the assumption that an empty disk is either all 1's or all 0's. IIRC, this is how you format it "government style". (There may actually be a pattern, like 10101010101, but that's still a very ordered state.)

I'm not sure if USB sticks are like Hard Drive, where, after a lot of use, even if it's empty, it will still have remnants of the old files, because it never really deletes them unless you format it "government style". All it really does is flag that that part of the drive can be written over. There's software that you can recover files like this with that have been "deleted" but not really. In this case, it could be more difficult to find out which state is more ordered, because the drive isn't really empty, it's just set to overwrite files you don't want anymore. Not sure if Flash Memory does this or not, but it might be something to consider.

So here, we're assuming that it's truly empty as in, all 0's or all 1's or a very simple, repeated pattern of 1's and 0's. This is a very low information state. A file will have to store it's contents as 1's and 0's in a way more complicated pattern to be able to actually store all the information (like words or a map of where pixels go) that will result in a not-so-trivial pattern of 1's and 0's. The disorder of the system will have increased.

Heh... I stopped labeling myself a long time ago. My dissertation was straight-up physics; fluid mechanics and optics. I guess people call me a biophysicist now... I just tell people I'm a scientist.

I got my undergrad in physics and am going interdisciplinary myself. One of my potential interdisciplinary advisers is a biophysicist. I have no idea what I will do for a dissertation yet though.

 

[mugaliens]

Whether a thumb drive is written with actual data or random data is indifferent: same energy.

Same energy, same mass.

 

[Stonebridge]

Since a flash drive has a static number of bits, it is always storing the same amount of information. The amount of information, being constant, is irrelevant to any changes in the mass of the drive.

And thanks to Pythagoras for his reply.

So with regards to the original question, there is no difference between full and empty in terms of the amount of information stored. In fact, full and empty have no meaning here.
To go back to a question I posed earlier. Say we consider a byte made up of 8 bits.

Pythagoras, you claim that 00000000 or 11111111 would be empty? A more random configuration of bits would contain more information?

But if I say that the byte is just storing a number and that

00000000 = 0
11111111 = 255
11010010 = 210 (arguably more random that the other two)

then each state holds exactly the same amount of information. A number between 0 and 255. So the memory location is always "full".

Now add a few million more identical memory locations and call it a memory stick or a hard drive. The drive can never be "empty". Full and empty have no meaning.
So how does this impact on the original question? "Is a full memory stick "heavier" than an empty one?"
Are we saying that any mass/energy difference is not a result of any stored information?
After all, I can, on a whim, define the information content of that byte any way I want.

 

[Pythagorean]

And thanks to Pythagoras for his reply.

So with regards to the original question, there is no difference between full and empty in terms of the amount of information stored. In fact, full and empty have no meaning here.
To go back to a question I posed earlier. Say we consider a byte made up of 8 bits.

Pythagoras, you claim that 00000000 or 11111111 would be empty? A more random configuration of bits would contain more information?

But if I say that the byte is just storing a number and that

00000000 = 0
11111111 = 255
11010010 = 210 (arguably more random that the other two)

then each state holds exactly the same amount of information. A number between 0 and 255. So the memory location is always "full".

Now add a few million more identical memory locations and call it a memory stick or a hard drive. The drive can never be "empty". Full and empty have no meaning.
So how does this impact on the original question? "Is a full memory stick "heavier" than an empty one?"
Are we saying that any mass/energy difference is not a result of any stored information?
After all, I can, on a whim, define the information content of that byte any way I want.

But you're using a highly interpretive definition of information, useful to humans. This is not the definition of information we're using.

1) We are assuming that the state of the bits corresponds to the physical state of the system (you have to make physical changes in the hardware to represent the bits). We'd have to have a very specific kind of expert to answer that question (that knows how USB sticks work physically). I don't think it's very far out there though. In my experiences with simple gate logic circuits, it is definitely true: The 1 and the 0 correspond to different states (a higher voltage with a 1, a lower voltage with a 0). But I don't know about the micro-circuity of the usb stick.

2) Information, in this context, pertains to the variety of states in the system. We're talking about the physical system that represents the bits, operating under assumption 1) above. If the system has all of it's particles in the same exact state, it is a high-order, low-entropy system. If it's particles occupy many different states, it is a low-order system with high entropy. This is physics.

3) The Gibb's free energy is where my intuition breaks down. I've been assuming the only form of it I'm familiar with: G = H - TS, where T is temperature and S is entropy. So you see can at least see that entropy and energy are related. However, I don't know if this simple relationship works for dynamics situations, and further more, I don't know if H and T are really constant or if they somehow shift to make the energy ultimately the same.

my confidence:
1) fairly confident
2) fairly confident
3) no idea

By the way, I'm not saying the mass changes for sure. We've been answering the question "how would it work". We'd need a tech expert for 1), and a thermo expert for 2) and 3).

 

[Andy Resnick]

<snip> I haven’t seen anyone yet quote a paper to defend this correlation. There have been many quotes of the literature, but I don’t see a single one that really brings this argument out and properly defends it one way or the other.

<snip>

I dug out a book from the back corner, by John Pierce "An Introduction to Information theory". It's a Dover book, so it has all the usual good qualities. I can't think of a better book to start learning this material.

 

[One128]

2) Information, in this context, pertains to the variety of states in the system. We're talking about the physical system that represents the bits, operating under assumption 1) above. If the system has all of it's particles in the same exact state, it is a high-order, low-entropy system. If it's particles occupy many different states, it is a low-order system with high entropy. This is physics.

More accurately, entropy (in thermodynamic sense) does not quantify how many states the particles do occupy, but how many states they can occupy, in a given macrostate. In a system where 0's and 1's are encoded symmetrically (same energy level etc.), particles can occupy the same number of states whether any single bit is 0 or 1 - the particular set of possible states changes when a bit value changes, but its size remains the same. In this context, the bit configuration is a part of macrostate specification - it is fixed and known - and thus does not contribute to the entropy of the system.

In a somewhat different context, the exact bit configuration could be a part of the entropy, but what would matter is how many bits are known down to (any) value and how many are "undefined" (more accurately, how many different bit configurations there can be, under given constraints) - the particular pattern that the known 0's and 1's form would again be irrelevant with regard to entropy.

Even the latter context could be made thermodynamic - you could theoretically construct some weird machine that would operate on uncertainty of individual bit levels - but such a machine, while strictly conforming to basic thermodynamic laws, would be "Maxwell-demonian" - the relationship between entropy and energy would not be as straightforward as in classical thermodynamics (which makes some assumptions that wouldn't be satisfied here) and consequently, the total energy would still be the same, regardless of bit configuration. (If everything else was the same and construction details didn't entail inherent energy difference between configurations.)

 

[Andy Resnick]

I think you are focusing on the wrong thing. I encode some information on a memory stick and give it to you. Instead of focusing on a string of '1' and '0' numbers, let's pretend the memory stick encodes base 27: all lowercase letters and a space.

Two preliminaries: I have many more choices of 30-'bit' messages to send than I do 1-'bit' messages. In fact, there are only 2 1-bit messages possible: 'I' and 'a'. There are considerably more 30-bit long strings I can send. So you, the receiver, ascribe more entropy to the 30-bit long message than the 1-bit message. Also, there is differences in uncertainty if you are reading along and encounter (for example) a 'q' rather than a 'c': 'q' is almost always followed by 'u', while 'c' can be followed by many more letters.

Now, before you object that the information in the code is 'observer dependent' becasue I chose english text, the above argument can be brought back to 1's and 0's by me sending you a (binary) message which is the sequence of coin flips, 1 = heads, and 0= tails. There are many more possible results of 30 coin flips than 1 coin flip, although you lose the notion of 'predictability'.

The entropy is defined the exact same way it usually is in statistical mechanics: S = -Sum(p log p).

I can encode messages all kinds of ways; it is possible to use entropy to determine the most efficient way of coding by calculating Sum(p log p):

http://en.wikipedia.org/wiki/Huffman_coding

 

[Andy Resnick]

<snip>

3) The Gibb's free energy is where my intuition breaks down. I've been assuming the only form of it I'm familiar with: G = H - TS, where T is temperature and S is entropy. So you see can at least see that entropy and energy are related. However, I don't know if this simple relationship works for dynamics situations, and further more, I don't know if H and T are really constant or if they somehow shift to make the energy ultimately the same.

<snip>

The Gibbs free energy is generally used for systems at constant pressure and temperature, but the real utility of the free energy is that it can be used for open systems (which includes chemical reactions). If dG < 0, then the process can occur *spontaneously*, in slight contrast to dS > 0 (irreversible). The free energy change of a process dG is a measure of the driving forces present on a process.

Dill and Bromberg's book "Molecular Driving Forces" is pretty good, but I like Nicholls and Ferguson's "Bioenergetics" better. Bioenergetics is more focused on cellular respiration, but is more fun to read (IMO).

 

[Pythagorean]

More accurately, entropy (in thermodynamic sense) does not quantify how many states the particles do occupy, but how many states they can occupy, in a given macrostate.

But isn't that what we're doing to the stick by applying voltages? Changing the amount of available particle states. We're controlling the shape of the gradient of the energy potential that the particles are in, that's how we affect the particles. The particles do their thing after that, falling into particular states with particular probabilities given the new potential. Since we can predict the states, given a certain potential, we can utilize this information to make a physical register that holds the information of 1 or 0.

From an engineering standpoint, there's also no reason to believe that two different system of particles can't make the same 0, since there's so many particles involved. In laymen terms, it could sum up to "a voltage greater than 5 means 1, and voltage less than 5 means 0" but physically, we're talking a huge number of different systems of particles that will pass for a 1 (or 0).

(If everything else was the same and construction details didn't entail inherent energy difference between configurations.)

Well, this may be bordering on philosophy, but I believe all information is necessarily being represented by a physical system. I can't think of any information that's not.

 

[One128]

Now, before you object that the information in the code is 'observer dependent' becasue I chose english text, the above argument can be brought back to 1's and 0's by me sending you a (binary) message which is the sequence of coin flips, 1 = heads, and 0= tails. There are many more possible results of 30 coin flips than 1 coin flip, although you lose the notion of 'predictability'.

The entropy is defined the exact same way it usually is in statistical mechanics: S = -Sum(p log p).

Entropy is the measure of uncertainty, and as such it is ultimately defined the same way in information theory as in statistical mechanics: as a measure of uncertainty. I believe I may have mentioned that a few times. The crucial thing is: the uncertainty about what, given what. That makes all the difference. Entropy depends on what you know and what you allow to change.

What is the entropy of a binary message which you know nothing about, not even its length, not even an upper limit of its length? It's infinite. What is the entropy of a message which you know to be 30 bits long, but nothing else about it? It's 30 bits. What is the entropy of a 30 bit message that you know to contain an odd number of ones? It's 29 bits. What is the entropy of a 30 bit message that you know to have the same number of ones and zeroes? About 27.2 bits. What is the entropy of a 30 bit message that you know encodes an English word in some particular scheme? You could work that out, probably something between 10 and 20 bits. What is the entropy of a 30 bit message that you know someone produced by tossing a coin? It's 30 bits. What is the entropy of a 30 bit message the contents of which you know? It's zero.

All of the messages mentioned in the last paragraph can in fact be the very same message: 100100010100001100110111101110. Entropy of this message depends entirely of what you know and don't know about it. That's what entropy measures - your uncertainty, given what you know.

 

[Andy Resnick]

I don't understand what you are saying. For example, if I tell you I am sending you the results of 30 coin tosses, how can you, *a priori*, tell me bit #10 is a '1'? For that matter, how can you tell me the message has an odd number of '1's?

The missing part is the idea of *measurement*. The entropy changes once you perform a measurement.

 

[One128]

I don't understand what you are saying. For example, if I tell you I am sending you the results of 30 coin tosses, how can you, *a priori*, tell me bit #10 is a '1'? For that matter, how can you tell me the message has an odd number of '1's?

If I know you're sending me the result of 30 coin tosses, and nothing else, then the entropy of such a message - until I receive it - is 30 bits, and I can't tell the value of any bit. After I receive it, the entropy of the same message is zero, and I can tell the value of every bit. If I don't know beforehand that your message has an odd number of ones, then I can't tell you until I receive the message. If I know it beforehand, I can tell you beforehand.

Entropy depends on what you know. If you assume that you know nothing except the length of the message, then every 30-bit message will have an entropy of 30 bits. That doesn't change the fact that if you know more (for example if you know details about the process producing the message that makes some outcomes less likely - for example, if you know the process only produces texts in English), then the entropy of an identical message can be different.

The missing part is the idea of *measurement*. The entropy changes once you perform a measurement.

Entropy is a measure of your uncertainty. If something reduces that uncertainty, it reduces entropy. Measurement is a vague term; if by "measuring" you mean reading bits of the message, then sure, that eliminates uncertainty about the message and reduces entropy. If by "measuring" you mean determining statistical properties of the message source, then again, that can reduce uncertainty and entropy. - If you measure something that doesn't reduce your uncertainty, then it doesn't reduce entropy.

Anything that affects probability distribution affects entropy, and it's not limited to "measurement". For example, if you have gas filling a container, and the same amount of gas filling half a container (held in place by some barrier), the entropy of the latter is less. Not because you measured the location of individual particles, but because you restricted their possibilities. Although you still don't know their exact location, you know something about them - that they are not in that part of the container. That reduces your uncertainty and entropy. - Similarly, if the contents of a binary message is somehow restricted, the entropy of the message will be less assuming that restriction.

 

[Andy Resnick]

I am not disagreeing with a single word you said here- when you receive the message (or read the memory stick), the entropy (of you + the memory stick, since the two of you are isolated from the rest of the universe) changes.

 

[Pythagorean]

Is there an established physical basis for information theory?

Anytime we have two physical systems (of particles) interacting that we can frame thermodynamically, could we use all the techniques of information theory to discover more about their system?

Could we, in a reductionist way, reduce all applications of information theory to physical processes? If we did, would the two theories produce the same entropy?

 

[Andy Resnick]

There is an established basis, and it is quite developed:

http://www.itsoc.org/
http://www.inderscience.com/browse/index.php?journalCODE=ijicot

Parenthetically, I am opposed to the reductionist approach- this thread is a good example of an integrationist (?) approach.

 

[dmtr]

Wouldn't http://en.wikipedia.org/wiki/Black_hole_information_paradox set the minimum on the increase of the mass of the alleged USB stick upon increasing the entropy on one bit?