Is there a deeper connection between thermal and information entropy?

[Andrew Mason]

It seems that he is retreating. All he had to say in response was this:

"The 2nd law states that ΔS ≥ 0. None of those examples violates the second law."

What is your response to this?

He is quite correct. But I was not suggesting that the second law can be violated. It can't. I was merely attempting to explain why entropy is NOT a measure of the energy that is unavailable for useful work.

His reply: "Entropy is a measure of the energy not available for useful work" is another way of looking at entropy. There is not just one way to look at entropy."

What say you?

There are many incorrect ways and a few correct ways to look at entropy. "Entropy is a measure of the energy not available for useful work" is an incorrect interpretation of entropy, for the reasons I have given.

AM

 

[revo74]

There are many incorrect ways and a few correct ways to look at entropy. "Entropy is a measure of the energy not available for useful work" is an incorrect interpretation of entropy, for the reasons I have given.

AM

His response to this was:

"I gave you a link to a video in which Bousso used the concept of entropy as information. Yet, you pursue this insane line that entropy is what you think it is. I'm wrong, wikipedia is wrong, Bousso, an outstanding physicist, is wrong."

I have come to realize that entropy in information theory and entropy in thermodynamics are different. Isn't the first paragraph of the entropy Wiki page making reference to thermodynamics? It doesn't specify. What is your response to his statement?

 

[Rap]

I have come to realize that entropy in information theory and entropy in thermodynamics are different. Isn't the first paragraph of the entropy Wiki page making reference to thermodynamics? It doesn't specify. What is your response to his statement?

Don't you agree that it is a matter of words, not physics? If you believe it is physics, then suggest an experiment whose outcome we will disagree on as a result of our disagreement on the definitions of information and thermodynamic entropy.

If there is none, then our disagreement is semantic, and is only resolvable by our mutual agreement on the meaning of language. I'm interested in physics, not language, which means I will not defend to the death my use of language, or attack yours. If there is no experiment to decide, what is your basic objection to the idea that entropy is a concept in the realm of pure probability which finds application in both information theory and thermodynamics?

 

[revo74]

Don't you agree that it is a matter of words, not physics? If you believe it is physics, then suggest an experiment whose outcome we will disagree on as a result of our disagreement on the definitions of information and thermodynamic entropy.

If there is none, then our disagreement is semantic, and is only resolvable by our mutual agreement on the meaning of language. I'm interested in physics, not language, which means I will not defend to the death my use of language, or attack yours. If there is no experiment to decide, what is your basic objection to the idea that entropy is a concept in the realm of pure probability which finds application in both information theory and thermodynamics?

I understand what you are saying. I have read all of your posts. My question to then is, is AM incorrect when he says Wikipedia is wrong. Are the examples he put forward wrong?

 

[Rap]

I understand what you are saying. I have read all of your posts. My question to then is, is AM incorrect when he says Wikipedia is wrong. Are the examples he put forward wrong?

I think Wikipedia is, at worst, wrong on the subject of entropy as a measure of energy unavailable for work. I think AM's exercise shows that interpreting the statement simplistically makes that statement nonsense. I think that any defense of the statement will make use of so many extra unmentioned assumptions and conditions as to render the statement useless as a quick and easy way to understand entropy.

On the separate subject of the difference between thermodynamic and information entropy, you state:

I have come to realize that entropy in information theory and entropy in thermodynamics are different.

Again - this is a question - What is your basic objection to the idea that entropy is a concept in the realm of pure probability which finds application in both information theory and thermodynamics?

 

[Rap]

It says that entropy is a measure of the energy that is not available for useful work. That is not correct. These examples will show what I mean:

While I agree with AM, I found this reference which supposedly defends the idea that entropy is a measure of energy unavailable for work:

T. V. Marcella, ‘‘Entropy production and the second law of thermodynamics:
An introduction to second law analysis,’’ Am. J. Phys. 60, 888–895, 1992

I cannot access it, if anyone can, I would be interested in what it says.

 

[revo74]

I think Wikipedia is, at worst, wrong on the subject of entropy as a measure of energy unavailable for work.

Why is the definition of entropy as a measure of energy unavailable for work so common if it's wrong. Some people argue that this is just a different way of looking at entropy. Other than the exercise AM presented, what other reasons or examples can you provide that demonstrates that this explanation is in-fact a bad one?

 

[Rap]

Why is the definition of entropy as a measure of energy unavailable for work so common if it's wrong. Some people argue that this is just a different way of looking at entropy. Other than the exercise AM presented, what other reasons or examples can you provide that demonstrates that this explanation is in-fact a bad one?

I think what is needed is a quantitative explanation of why it is true. If the explanation works, then fine, if not, then that's another indication that it is flawed.

 

[Andrew Mason]

Why is the definition of entropy as a measure of energy unavailable for work so common if it's wrong. Some people argue that this is just a different way of looking at entropy. Other than the exercise AM presented, what other reasons or examples can you provide that demonstrates that this explanation is in-fact a bad one?

You will find there are many misleading explanations or definitions of entropy. Repeating them over and over does not make them more accurate. Entropy = disorder is a misleading unless you have a very specific definition of disorder. "Entropy is a measure of the energy unavailable for useful work" is also inaccurate and misleading.

An increase in entropy (of the system + surroundings) is certainly related to an increase in the amount of heat flow into a system that is not turned into useful work in a thermodynamic process. The greater the increase in total entropy the less efficient the thermodynamic process in terms of the ratio of useful work produced to input heat flow energy.

But you cannot conclude from this that entropy is a measure of the heat flow that is not available to do useful work. The change in entropy relates to only a part of the heat flow that does not produce useful work. Much of that heat flow is not available to do useful work even if there is no increase in entropy (reversible process).

AM

 

[lowerlowerhk]

You will find there are many misleading explanations or definitions of entropy. Repeating them over and over does not make them more accurate. Entropy = disorder is a misleading unless you have a very specific definition of disorder. "Entropy is a measure of the energy unavailable for useful work" is also inaccurate and misleading.

AM

Ok I am convinced information entropy and thermodynamic entropy doesn't mix well. However, even for the macroscopic, thermodynamic meaning of entropy, I still don't get what it is. I currently imagine it is a measure of energy that goes "elsewhere", to a place where it couldn't be changed to other forms. A system with high entropy means it has few "flow-able" energy, whether it flows within or outside; and a low entropy system means much of the energy could (potentially) flow. An analog to my imagination would be a glass of water with fixed number of molecules. If more water is in the form of ice (entropy), less water (energy) could flow inside the cup or be poured away.

Does that match what macroscopic thermodynamic entropy is?

 

[Rap]

Ok I am convinced information entropy and thermodynamic entropy doesn't mix well.

Why are you convinced of this?

 

[lowerlowerhk]

Why are you convinced of this?

Because when the term entropy was first invented, people still don't know the molecular nature of thermodynamics and the original meaning has nothing to do with disorder. Before the concept was expanded to other fields, the purpose of inventing the notion of entropy was just to understand the action of steam engines. So should be able to understand what (thermodynamic) entropy in macroscopic scale is without bothering disorder or information.

 

[Rap]

Because when the term entropy was first invented, people still don't know the molecular nature of thermodynamics and the original meaning has nothing to do with disorder. Before the concept was expanded to other fields, the purpose of inventing the notion of entropy was just to understand the action of steam engines. So should be able to understand what (thermodynamic) entropy in macroscopic scale is without bothering disorder or information.

But then that would mean that we can understand temperature without understanding the energy of a molecule, we can understand pressure without understanding the impact of molecules on a surface. In other words, you are saying we don't need statistical mechanics or atomic theory to understand classical thermodynamics. Actually, that is true, but there is so much more understanding to be gained by understanding all of these things in terms of statistical mechanics. Once you try to gain more understanding of entropy by using statistical mechanics, then you get the connection to information entropy.

 

[Studiot]

It is important to understand the difference between molecules and classical particles.

 

[lowerlowerhk]

But then that would mean that we can understand temperature without understanding the energy of a molecule, we can understand pressure without understanding the impact of molecules on a surface. In other words, you are saying we don't need statistical mechanics or atomic theory to understand classical thermodynamics. Actually, that is true, but there is so much more understanding to be gained by understanding all of these things in terms of statistical mechanics. Once you try to gain more understanding of entropy by using statistical mechanics, then you get the connection to information entropy.

I understand how useful it is to understand phenomena in microscopic terms. But as a noob in thermodynamics, I don't even have a concept to begin my connection (referring to your last sentence). It is like when talking about pressure, people just think of particle colliding with walls without realizing the corresponding macroscopic phenomena is the walls being "pushed" by a force. When talking about thermodynamic entropy, people just talk about disorder without linking the microscopic phenomena to the macroscopic one. The lack of macro-micro linking is the main reason why the notion of entropy is difficult to grasp for new comers. Although it may be turned out the microscopic aspect is more generalized hence more applicable, as a stepping stone it is still useful to know the macro aspect of entropy.

Anyway, here is what I found in a site:
http://www.science20.com/train_thought/blog/entropy_not_disorder-75081"

Historically, there is a problem of how to tell thermal energy of a system by its temperature. Entropy was invented originally to state the relationship between temperature and thermal energy. A body with the same temperature could have different thermal energy. In common sense higher temperature must means more thermal energy, but it is not always true. In fact there could be body with high thermal energy but low temperature (due to high entropy), and vice versa.This is the macroscopic phenomena of entropy.

To explain this common-sense-violating weirdo, the microscopic explanation comes into play. If the gas molecules have more ways to move, to vibrate, to oscillate, then they could gain more energy without flying faster (having higher KE). That's why bodies with same temperature (same average KE) could have different thermal energy.

And then somebody proposed the more ways to move a system has, the more messy (disorder) it is. It was not until then entropy is related to disorder, then even later, a randomness in information. And finally the term entropy has extended to other fields which has nothing to do with thermal energy.

To end with a analogy, we learn atoms first as tiny balls, then as a micro-solar system with electrons orbiting nucleus, then as electrons cloud orbiting nucleus in a quantum-mechanic way. Although the quantum-mechanic model has the strongest explanatory power, we still learn its former, less predictive model because it is too un-intuitive to grasp for new comers. So similarly, it is a bad idea to just talk about the deepest meaning of entropy and hoping a few genius could retro-deduce it back to the macroscopic aspect.

 

[Rap]

I understand how useful it is to understand phenomena in microscopic terms. But as a noob in thermodynamics, I don't even have a concept to begin my connection (referring to your last sentence). It is like when talking about pressure, people just think of particle colliding with walls without realizing the corresponding macroscopic phenomena is the walls being "pushed" by a force. When talking about thermodynamic entropy, people just talk about disorder without linking the microscopic phenomena to the macroscopic one. The lack of macro-micro linking is the main reason why the notion of entropy is difficult to grasp for new comers. Although it may be turned out the microscopic aspect is more generalized hence more applicable, as a stepping stone it is still useful to know the macro aspect of entropy.

Entropy is not only difficult for newcomers, it is difficult for almost everyone. Forget about "disorder", that only gives you a "feel" for entropy. "disorder" is not rigorously defined, and the idea gets squishy if you push it too far. The link between thermodynamic entropy and information entropy is the beginning of the "macro-micro linking" you are looking for. The thermodynamic entropy is equal to the Boltzmann constant times the information entropy, and the information entropy is the minimum number of yes/no questions you have to ask to determine the microstate, given that you know the macrostate (temperature, pressure, etc.). Sort of like the game of twenty questions. This doesn't give you the complete link, but it goes a LONG way towards it.

Anyway, here is what I found in a site:
http://www.science20.com/train_thought/blog/entropy_not_disorder-75081"

This guy uses "diversity" as giving a better feel for entropy. I think it is better. But to be exact, what is "diversity"? If I flip four fair coins, but I don't look at them yet, the info entropy is 4 bits. I have to ask at least 4 questions about them to know how they came out. With 8 coins, its 8 questions. If you say that the 8 coin result is exactly twice as diverse as the 4 coin result, then yes, diversity is a good word because it is exactly proportional to the information entropy. Call it whatever you want, and as long as what you call it is proportional to the information entropy, you will be correct. The thermo entropy is proportional to the info entropy by the Boltzmann constant, so either one is good.

Historically, there is a problem of how to tell thermal energy of a system by its temperature. Entropy was invented originally to state the relationship between temperature and thermal energy. A body with the same temperature could have different thermal energy. In common sense higher temperature must means more thermal energy, but it is not always true. In fact there could be body with high thermal energy but low temperature (due to high entropy), and vice versa.This is the macroscopic phenomena of entropy.

To explain this common-sense-violating weirdo, the microscopic explanation comes into play. If the gas molecules have more ways to move, to vibrate, to oscillate, then they could gain more energy without flying faster (having higher KE). That's why bodies with same temperature (same average KE) could have different thermal energy.

And then somebody proposed the more ways to move a system has, the more messy (disorder) it is. It was not until then entropy is related to disorder, then even later, a randomness in information. And finally the term entropy has extended to other fields which has nothing to do with thermal energy.

To end with a analogy, we learn atoms first as tiny balls, then as a micro-solar system with electrons orbiting nucleus, then as electrons cloud orbiting nucleus in a quantum-mechanic way. Although the quantum-mechanic model has the strongest explanatory power, we still learn its former, less predictive model because it is too un-intuitive to grasp for new comers. So similarly, it is a bad idea to just talk about the deepest meaning of entropy and hoping a few genius could retro-deduce it back to the macroscopic aspect.

Thats the part of thermodynamic entropy that information entropy does not give much help on - the connection to the rest of thermodynamics. Thermodynamic entropy is Boltzmann's constant times the information entropy. The internal energy is temperature times Boltzmann's constant times the information entropy. Boltzmann's constant is historically connected to entropy rather than temperature, but its just as good to define a new temperature T'=kT and say that internal energy is the new temperature times the information entropy. Now the real problem is to understand temperature. I think that once information entropy is understood, the real problem lies in understanding temperature. People don't worry about temperature, because its something we intuitively "understand" - hot and cold and all that. But the real problem is the bridge from information entropy to internal energy, and that bridge is the temperature. Anyone who claims to understand thermodynamic entropy MUST understand temperature, and not in the "hot and cold" sense, but in the sense of being the bridge between internal energy and information entropy, two things we can more easily understand.

 

[lowerlowerhk]

Thats the part of thermodynamic entropy that information entropy does not give much help on - the connection to the rest of thermodynamics. Thermodynamic entropy is Boltzmann's constant times the information entropy. The internal energy is temperature times Boltzmann's constant times the information entropy. Boltzmann's constant is historically connected to entropy rather than temperature, but its just as good to define a new temperature T'=kT and say that internal energy is the new temperature times the information entropy. Now the real problem is to understand temperature. I think that once information entropy is understood, the real problem lies in understanding temperature. People don't worry about temperature, because its something we intuitively "understand" - hot and cold and all that. But the real problem is the bridge from information entropy to internal energy, and that bridge is the temperature. Anyone who claims to understand thermodynamic entropy MUST understand temperature, and not in the "hot and cold" sense, but in the sense of being the bridge between internal energy and information entropy, two things we can more easily understand.

Temperature is relatively easy to understand. Both the macro meaning (ease of conducting heat away) and micro meaning (average KE in molecules) are intuitive enough to understand and linked quickly.

Entropy, macrosocpically is a (not directly proportional) measure of thermal energy that doesn't rise temperature of a body. Microscopically entropy is a measure of how many ways could the molecules oscillate (more oscillations available means less KE and lower temperature), which is directly proportional to the number of ways to arrange molecules of a body. This is how thermodynamic entropy and information entropy related.

This missing piece of logic that makes thermodynamic entropy and information entropy so unrelated is the hidden relationship between molecule arrangements and available oscillations, which is expressed mathematically be Boltzmann's constant.

 

[Studiot]

Temperature is relatively easy to understand. Both the macro meaning (ease of conducting heat away) and micro meaning (average KE in molecules) are intuitive enough to understand and linked quickly.

Entropy, macrosocpically is a (not directly proportional) measure of thermal energy that doesn't rise temperature of a body. Microscopically entropy is a measure of how many ways could the molecules oscillate (more oscillations available means less KE and lower temperature), which is directly proportional to the number of ways to arrange molecules of a body. This is how thermodynamic entropy and information entropy related.

This missing piece of logic that makes thermodynamic entropy and information entropy so unrelated is the hidden relationship between molecule arrangements and available oscillations, which is expressed mathematically be Boltzmann's constant.

So was all this a statement that a cathode ray does or does not possesses a defineable and measurable temperature?

 

[lowerlowerhk]

So was all this a statement that a cathode ray does or does not possesses a defineable and measurable temperature?

I don't see your logic. Cathode ray, ie electrons, are moving, so they have kinetic energy. So you can calculate the average kinetic energy as temperature. Though when talking about beams we usually use power to describe its heat transfer instead of temperature, because you can't put a travel beam of electrons next to an object like a block.

And I don't understand what you want to demonstrate by throwing out the question.

 

[Studiot]

I notice you avoided my comment, just as you did for post #44 and this thread

https://www.physicsforums.com/showthread.php?t=517005

When my comments were designed to help you tighten up on some loose terminology, which I think is hindering your understanding.

In particular specific heat is a property of the system/material itself, not of the process.

You can only add or subtract heat from a material/system at the (not time) rate of specific heat.

Entropy is a process property. You can add more or less heat to a system and the process equations (first and second laws etc) lead to different entropy (and other variable) solutions.

go well

 

[Pythagorean]

Thermodynamic entropy is the logarithm of the number of microstates of the molecules of a system in phase space (momentum, position in 3D) multiplied by the Boltzmann constant. It seems to me that information entropy, dealing with probability starts with a very different concept - the number of 1s and 0s required to convey an intelligible message.
AM

Wait, what?

1s and 0s ARE microstates. The map of all possible trajectories in the system's phasespace IS the information about the system. If you only have one bit, you only have two microstates (yes, they call them 1 and 0, but names aren't important, we can call them state a and state b).

A system that is in steady state has 0 bits (it has no states to change to so you don't even need a 1 or 0 to express it's state).

"intelligible" has nothing to do with it. The "message" could be a billiard ball impacting another billiard ball. It's the energy, the wave, the propagation of a disturbance, not the matter itself.

Maxwell's demon requires information to do work.

If you can direct me to some authority who says that there is a real and substantial connection between the two concepts I'll be happy to reconsider.

What do you think of what Kolmogorov has done with information theory? (metric entropy, Kolmogorov complexity)

 

[Rap]

This missing piece of logic that makes thermodynamic entropy and information entropy so unrelated is the hidden relationship between molecule arrangements and available oscillations, which is expressed mathematically be Boltzmann's constant.

Come to think of it, you are right. I was thinking of Boltzmann's constant, as something that could be set to one, by the right definition of temperature, and so not that important. But that is an "information-centric" view. How do you come by that "right definition"? You cannot use a well calibrated thermometer based on the second law, measure Boltzmann's constant, then multiply the temperature by that constant to get your new temperature, because that begs the question. From an information theory point of view, it doesn't matter what the constant is, but from a thermodynamic+information point of view, the value of Boltzmann's constant is the key to understanding thermodynamic entropy.

 

[lowerlowerhk]

In particular specific heat is a property of the system/material itself, not of the process.

You can only add or subtract heat from a material/system at the (not time) rate of specific heat.

Yes, we both agree on that.

Entropy is a process property.

go well

Shouldn't entropy S like internal energy U is a state, while dS and dU process?

What you said contradicts what I see in wikipedia and various sources that entropy is a state.

http://en.wikipedia.org/wiki/List_of_thermodynamic_properties"

 

[lowerlowerhk]

From an information theory point of view, it doesn't matter what the constant is, but from a thermodynamic point of view, its the key to understanding thermodynamic entropy.

There is something that amuses me here. It is easier to start from thermodynamic entropy to search for a interpretation that links the two kinds of entropy together. But if you begin with information entropy it is much harder to associate information with thermal energy. I tried the second approach and ended up with an instrumentalist conclusion of "it's just a dummy variable that make things complete and there is no point to think what that represent as long as things could work". (To me) it looks like some point view is easier to generalize and extend than other.

 

[Studiot]

I notice you have also joined this thread.

https://www.physicsforums.com/showthread.php?t=487647&highlight=wikipedia&page=2

There is a long discussion noting the shortcomings of Wikipedia in thermodynamics.
In particular have you read post #23?

Strictly, entropy is a state function, yes.

But consider the thermodynamic implications of this variation on a once popular auto fuel advert.

Take two different makes of car.

Put 1 litre of identical fuel in each.

Car A achieves 7.8 miles on its litre of fuel.

Car B achieves 13.4 miles on its litre of fuel.

Is it the difference due to the system or the fuel?

 

[Andrew Mason]

I don't see your logic. Cathode ray, ie electrons, are moving, so they have kinetic energy. So you can calculate the average kinetic energy as temperature.

Temperature is only defined for a substance in thermal equilibrium. In thermal equilibrium the translational kinetic energies of the molecules follow a Maxwell-Bolztmann distribution. The kinetic energies are not all the same. In a cathode ray the moving electrons are all in one direction and all with the same energy. So its temperature cannot be defined.

AM

 

[Andrew Mason]

Wait, what?

1s and 0s ARE microstates. The map of all possible trajectories in the system's phasespace IS the information about the system. If you only have one bit, you only have two microstates (yes, they call them 1 and 0, but names aren't important, we can call them state a and state b).

A system that is in steady state has 0 bits (it has no states to change to so you don't even need a 1 or 0 to express it's state).

"intelligible" has nothing to do with it. The "message" could be a billiard ball impacting another billiard ball. It's the energy, the wave, the propagation of a disturbance, not the matter itself.

Maxwell's demon requires information to do work.



What do you think of what Kolmogorov has done with information theory? (metric entropy, Kolmogorov complexity)

There are some analogies between the entropy in information theory and entropy in thermodynamics, but the underlying concepts are very different. They have different origins and apply to different things having vastly different numbers of particles/events. Thermodynamic entropy is inextricably tied to a concept of temperature with an underlying assumption that all microstates are equally probable. Information entropy has nothing even analogous to temperature, as far as I can see, and assumes that the probabilities of individual microstates are not equal. So, as far as I can see the only real connection is that they have same name.

AM

 

[Studiot]

Good afternnon, Andrew.

Your last two posts could do with some amplification, I feel, else some get the wrong impression.

So in post#56

(absolute) temperature may be only defined for a system in equilibrium, but temperature difference is OK for for non equilibrium situations.

and in post#57

A distinction should be made between the statistics of the average action of a large number of particles, leading to classical thermodynamic relationships in the physical world (which again is OK) and information states in the nonphysical world which can be very different depending upon your system of logic and information.

Further comment on my cathode ray example.

My main aim in producing this was to offer a physical example not involving molecules to reinforce my effort to stress the difference between molecules and particles.

However if the beam impacts upon another physical object energy will be transferred, in accordance with the laws of thermodynamics.
In this case a temperature difference may be established.

go well

 

[Rap]

There are some analogies between the entropy in information theory and entropy in thermodynamics, but the underlying concepts are very different. They have different origins and apply to different things having vastly different numbers of particles/events. Thermodynamic entropy is inextricably tied to a concept of temperature with an underlying assumption that all microstates are equally probable. Information entropy has nothing even analogous to temperature, as far as I can see, and assumes that the probabilities of individual microstates are not equal. So, as far as I can see the only real connection is that they have same name.

AM

Nobody argues that thermodynamic entropy is not tied to temperature, but the thermodynamic entropy is equal to Boltzmann's constant times the information entropy. The information entropy is proportional to the minimum number of yes/no questions you have to ask to determine the microstate, given the macrostate. As mentioned above, the Boltzmann constant is the "bridge" between thermodynamic and information entropy, and the full meaning of thermodynamic entropy is to be found in

#1: The understanding of information entropy, and

#2: The understanding of why the Boltzmann constant has the value it has.

You can understand #1 without reference to temperature. Its #2 that brings the temperature in. You cannot understand #2 without understanding temperature. This is the way that temperature is "unlinked" from information entropy. The question of whether thermodynamic entropy and information entropy both deserve to have the word "entropy" in their name is semantic, not very interesting. The link between the two is unavoidably there, whatever you want to call them. #2 is where the fun is, not in a semantic argument over the naming of things, and not in refusing to divide up the understanding of thermodynamic entropy into #1 and #2.

Also, they do not have "vastly different numbers of particles/events". Information entropy puts no limit on the number of "events" that it will deal with. Nor does it necessarily assume that microstates have different probabilities.

 

[Studiot]

but the thermodynamic entropy is equal to Boltzmann's constant times the information entropy.

Such a statemen by itself is not proof of linkage any more than the following proves a linakge between a certain stone in my garden an the USS Forrestal.

The weight of the USS Forrestal is exactly 1.2345658763209 * 10⁹ times the weight of a certain stone in my garden.