Musing on the Foundations of Statistical Mechanics

[Matterwave]

Hello~

I've been reading about the Foundations of Statistical Mechanics recently and found the topic of foundations Entropy quite interesting (although I haven't actually gotten to the section on Entropy yet in the book I'm reading -- A. I. Khinchin Foundations of SM). I was wondering if this would be a good place to have some discussion about it? I don't have a concrete technical question per-say, and some of my thoughts and ponderings borders on philosophy (e.g. ontological categorization of Entropy). So I'm not sure if it violates some forum rule. If it's not great to do the discussion here, lmk and I'm ok if this thread gets closed.

If it's ok, then I'll post some musings and maybe some folks with be interested in having a discussion? :)

 

[Matterwave]

Ok then~

I have a lot of random thoughts so I'll just first post something rough otherwise I'll never post anything if I'm trying to be precise haha.

There's a lot in foundations of SM that interests me, but the concept of entropy has always been a bit of a mystery. Mostly, I think, for a few reasons:

1. There's several definitions of entropy, the Claussius $dS=\frac{\delta Q}{T}$, Boltzmann $S=k\text{ln}\Omega$, Gibbs $S=-k\sum \rho\text{ln}\rho$, Von Neumann $S=\text{Tr}(\rho\text{ln}\rho)$, and Shannon entropy to name a few. I was basically taught (all those many years ago) these are all the same thing or maybe they're just for slightly different circumstances (but that they measure the same "physical thing" in different circumstances). But they really seem not, despite formal similarities with each other. Especially the Shannon one seems very different than the rest.
2. So many definitions "under different circumstances" it is hard to keep them straight in my head, and I really can't see how they define "the same thing".
3. If they're not all the same thing then it would seem the second law must be shown to be true in each of their cases. And if they are all the same thing then the equivalence would have to be shown case by case.

Furthermore, the ontological status of entropy is unclear and seems to differ for each definition. What I mean by "ontological status" is vague, but roughly speaking it's the question "is entropy a physical quantity like mass/distance/time/energy (it certainly has units Joule/Kelvin) or is it a consequence of our ignorance of the microphysics like not knowing how a dice will land after being thrown?" I.e. if we "knew everything there is to know" would entropy disappear?

Even within one definition, the ontological status seems murky. Take the Boltzmann entropy as an example. We could partition the phase space up into separate chunks based on "our knowledge of the macro states", course grain it (this concept is also fuzzy to me, but I see it very roughly as some way to define a "unit volume" in phase space, correct me if I'm wrong) and count $\Omega$. The entropy then seems to be a property of the partition (essentially, the log of the quantized phase space volume) and not the actual trajectory (in Classical mechanics this is simply one point in phase space). Each time I think on this I come to a different conclusion on whether entropy is a physical thing (my trajectory is in this particular partition, so at least it makes some certain sense to attribute all physical configurations in here to have some certain value of entropy) or not (the partitions seem arbitrary based on what integrals of the motion I have knowledge of).

Other views might also bolster a physical interpretation -- e.g. that the arrow of time is somehow defined using the second law. But then you get things like the Poincare recurrence which seems to just massively violate the second law given long enough time scales.

Anyways I'll stop here before I ramble on too long. I'll just say I am heavily influenced by the thinking of David Albert and Tim Maudlin, but even their expositions here are not entirely convincing. I can't seem to form a concrete opinion on the matter.

 

[Matterwave]

I'm quite interested to see what Khinchin has to say about this topic because he has so far been very mathematically rigorous. But I'm still working through the ergodicity foundations (since his book is from 1940s, he doesn't know that KAM theory will kind of ruin the dream of strict Ergodicity in the 1960s). And I took a very rough skim on his statements on entropy and they seem to just be mathematical statements (i.e. proving the Claussius definition arises in some sense from the statistical definition and that entropy is a state variable iirc).

 

[Dale]

But they really seem not, despite formal similarities with each other.

Well, this sort of thing is pretty common. Like for “work”. In Newtonian mechanics it is force times displacement. But in thermo it is energy transfer by any means other than heat. Those are also not the same thing.

 

[Matterwave]

Well, this sort of thing is pretty common. Like for “work”. In Newtonian mechanics it is force times displacement. But in thermo it is energy transfer by any means other than heat. Those are also not the same thing.

Yes indeed, but the situation for Entropy somehow feels more pronounced. For in your example, at least when we consider piston-like objects doing work, the PdV work is at least very similar formally to Fdx work. And for me, this seems intuitive.

But in the case of Entropy, I am aware of various "proofs" that Boltzmann Entropy "implies" (I'm not even sure what the right word is here) Claussius or is used as a foundation for the Thermodynamic entropy, but they all rely on some extra assumptions (e.g. course graining, or some very specific set ups).

Maybe if someone has a super "clean" proof/derivation? This is what I was hoping Khinchin would provide.

Unfortunately, because I am no expert here and my conceptual understanding is limited, I am not able to make very intelligent comments. I can only give vague senses of "something feels off".

 

[Dale]

when we consider piston-like objects doing work, the PdV work is at least very similar formally to Fdx work. And for me, this seems intuitive.

But in the case of Entropy, I am aware of various "proofs" that Boltzmann Entropy "implies" (I'm not even sure what the right word is here) Claussius or is used as a foundation for the Thermodynamic entropy, but they all rely on some extra assumptions

That seems closely analogous to me. In the case of work if you assume a piston then you can show the thermodynamic and mechanical work are the same. In the case of entropy if you make “some extra assumptions” then they are also the same.

In both cases you have one word (work or entropy) that is used to refer to multiple different concepts. Those concepts are different and cover different scenarios. But there is some overlap (pistons or some extra assumptions). Where they overlap they agree. So using the same word to refer to the different concepts is justified. And where you need to identify one specific concept you can do that with additional descriptive terms.

I can only give vague senses of "something feels off"

That is why I bring in the analogy. It may help resolve the feeling to recognize that this is the same thing that happens elsewhere. The issue here is not entropy, but your feeling that something is off.

 

[Matterwave]

Well it might be true for Boltzmann Entropy and Claussius. Though, I still think the connection is much more tenuous than the work example and especially the "ontological status" is quite different in the entropy sense (is it purely phenomenological? Is it fundamental?) but not so different in the work sense (where I see it as just different ways to label how we get energy into/out of the system).

But Shannon Entropy for example I see no connection to a physical entropy. Maybe someone can enlighten me on that one.

But still, I'm ok with using the same word "entropy". It's why I am just musing and not making arguments haha.

 

[Dale]

especially the "ontological status" is quite different

I cannot help you with philosophical questions like ontological status. Usually, you can adopt any ontology you like and simply adjust your epistemology accordingly.

But Shannon Entropy for example I see no connection to a physical entropy. Maybe someone can enlighten me on that one.

The Wikipedia page on that topic is pretty good: https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory

You are definitely not the only person to have had that concern. The main connection is simply the mathematical form. Both are a sum of products of probabilities and log probabilities. That's it.

The Shannon entropy can be calculated for any probability distribution. The physical entropy can only be calculated for physical probability distributions. But for physical probability distributions, where they overlap, they give the same value (up to a factor of $k_B$).

 

[Matterwave]

Even within one definition, the ontological status seems murky. Take the Boltzmann entropy as an example. We could partition...

Beyond the "are these different definitions the same", this part of just the Boltzmann entropy also bothers me a decent amount.

 

[Dale]

Beyond the "are these different definitions the same", this part of just the Boltzmann entropy also bothers me a decent amount.

Because of the ontological questions? If so, just pick the ontology you prefer. You are always free to change it later if your preferences change. It is just philosophy.

 

[Paul Colby]

One comment,

$$dS = \frac{\delta Q}{T} $$

was know from thermodynamics. Unclear that atoms were even a thing when this form of entropy was coined.

 

[Matterwave]

Because of the ontological questions? If so, just pick the ontology you prefer. You are always free to change it later if your preferences change. It is just philosophy.

Yeah, I don't really have many calculational questions arising from the musings as I am no longer being asked by homework assignments to calculate something.

 

[Matterwave]

Maybe the closest way I know how to steer this towards physics and not philosophy:

If I knew everything there is to know about a microcanonical ensemble (i.e. the trajectories of every particle) does that ensemble have 0 entropy or the same entropy as it has if I knew only $N,V,E$? If the former, does this violate the second law? If the latter, what am I supposed to do with $p_i$ in the Gibbs definition of entropy (which is still supposed to be for physical systems)? Pretend I don't know?

If this is still philosophy, then.. my apologies.

 

[GiorgioPastore]

...

The Shannon entropy can be calculated for any probability distribution. The physical entropy can only be calculated for physical probability distributions. But for physical probability distributions, where they overlap, they give the same value (up to a factor of $k_B$).

Not exactly. The statistical mechanics entropy is explicitly defined for finite-size systems. The properties of thermodynamic entropy are recovered from statistical-mechanics formulae only in the thermodynamic limit. Without the thermodynamic limit, there is no overlap. For example, the nice convexity properties of classical thermodynamics do not generally hold for interacting finite-size systems in Statistical Mechanics.

Not recognizing the need for and the consequences of the limit to an infinite number of degrees of freedom is the root of many historical misunderstandings about Statistical Mechanics.

 

[Matterwave]

Not recognizing the need for and the consequences of the limit to an infinite number of degrees of freedom is the root of many historical misunderstandings about Statistical Mechanics.

I would like to hear more about this :)

I'm just now getting to the section in Khinchin where he makes the formal analogy/conversion between phase space measures and probability measure. I think he will set up the CLT soon to handle the N->inf limit explicitly without invoking general ergodicity.

 

[Demystifier]

Gibbs and Boltzmann entropy are not the same, but in thermal equilibrium they have the same value.

For some foundational, philosophical and ontological aspect of statistical mechanics take a look at my https://arxiv.org/abs/2308.10500
and references therein.

 

[Andrew Mason]

There are other threads that discuss the similarities and differences between thermodynamic entropy Boltzmann entropy and Clausius entropy, and Shannon entropy:

M

Graduate Thread 'Shannon Entropy and Origins of the Universe'

Jan 20, 2014

Hello Community,

I have a question that I'm struggling to get clarification on and I would greatly appreciate your thoughts.

Big bang theories describe an extremely low thermodynamic entropy (S) state of origin (very ordered).
Question: Is the big bang considered to be a high or low shannon entropy (H -information) state?

Here's why I question:

S and H are related: S = the amount of H needed to define the detailed microscopic state of the system, given its macroscopic description.
Thus: Gain in entropy always means loss of information.

2nd Law of Thermodynamics: S always tends...

• mr_whisk
• Entropy Shannon entropy Universe
• Replies: 3
• Forum: Astronomy, Astrophysics, Cosmology

Graduate Post in thread 'Is there a generalized second law of thermodynamics?'

May 14, 2022

As I understand the principle, it says that regardless of how small the physical means of carrying information, changing the smallest amount of information will still require a non-zero expenditure of energy.

Erasing, not necessarily changing. Reversible logical gates don't necessarily need to produce thermodynamic entropy.

This will result in some increase in thermodynamic entropy. But this is more about thermodynamics than information theory. I really don't see how it relates Shannon entropy.

It's more about thermodynamic entropy, I agree, but it doesn't mean that...

• Demystifier

S

Graduate Post in thread 'Why do we need a quantum correction for black hole entropy?'

Jun 6, 2022

Rovelli argues that entropy can be much larger than the Bekenstein bound
https://arxiv.org/abs/1710.00218 . I myself made similar arguments in https://arxiv.org/abs/1507.00591 .

from this article...
""
This conclusion is not in contrast with the the various arguments leading to identify Bekenstein-Hawking
entropy with a counting of states. "
""

For example, Page says that Casini's proof corresponds to one particular definition of S and RE https://arxiv.org/abs/1804.10623 , but it's not clear whether this definition is the right one.

...

• somok

AM

 

[Matterwave]

I finally "finished" reading Khinchin (at least up through chapter 7). I also read through https://plato.stanford.edu/entries/statphys-statmech/ which I found very helpful. Since this thread began with my musings, I suppose I will make another post here~

Still just my musings, but I appreciate all the discussion that this thread has so far. Some thoughts here have been helpful to my readings.

I think my biggest take-away after reading these resources is actually to gain a deeper interest and appreciation for the subject. Thermodynamics and Stat-mech were always weak subjects for me when I was studying physics. I can't exactly put my finger on why, but they just didn't inspire my interest. Maybe because I kept trying to attribute fundamental/foundational status to thermodynamics when it is clearly a phenomenological science. Perhaps my overall mental model of how things piece together was wrong.

Continuing my musings:

40,000ft level it seems like Statistical Mechanics should provide the microphysical foundation for the phenomenological science of Thermodynamics. To do this, SM mostly has to explain the "Second Law of thermodynamics" since the first law is generally pretty easy to show is satisfied by the microphysics (it's just conservation of energy). Simply the expression of the second law of thermodynamics, though, seems quite varied.

Khinchin, for example:

Thus we see that the quantity $\vartheta(dE-\bar{\delta A})$ is the total differential of a certain thermodynamic function. The above result really contains the second law of thermodynamics.

Here $\bar{\delta A}$ is the average (infinitesimal) work done on the system, and I read this statement to basically be saying that the Clausius entropy's integrand $\delta Q/T$ is total differential so its integral is a valid state function.

Goodstein in "State of Matter" says:

According to the celebrated Second Law of thermodynamics, the entropy of a system out of equilibrium will tend to increase.

And a bit later he quotes Omar Khayyam (translated by Edward Fitzgerald):

The Moving Finger writes; and, having writ,
Moves on: nor all thy Piety nor Wit
Shall lure it back to cancel half a line,
Nor all thy Tears Wash out a Word of it.

Which he says is the most elegant statement of the Second Law.

In any scenario, it seems clear that "maximize entropy" seems to be the general statement for how Equilibrium is or should be reached. And once equilibrium *is* reached, the two main problems in the foundations literature seems to point at irreversibility (shouldn't you be able to get out of equilibrium once you get there?) and (Poincare) recurrence (Hamiltonian flow inside a finite phase volume will visit infinitely near any given point an infinite number of times).

Anyways, to tie the microphysics to the macrophysics, it seems a large chunk of effort was spent in studying Ergodic theory. At the time of Khinchin, such study was nascent and Von Neumann and Birkhoff had just proved the initial "ergodic theorems" which just says the infinite time-average integral of phase functions (for a trajectory in phase space) exists (is bounded) for initial conditions almost everywhere (given a measure on the phase space) and then that this time average would equal the phase average in the case that the region of phase space under consideration is "metrically indecomposable" (which, just says very roughly that the trajectories inside that region of phase space "explores everywhere" and don't get trapped). I put the words "ergodic theorems" in quotes because Khinchin calls these theorems "ergodic theorems" but I think in modern language they would just be more like a set-up (that such integrals make sense at all) and not at all proofs of ergodicity (read: time average == phase average) of any specific systems. It seems to me that "the ergodic path" towards founding thermodynamics on SM is a hard one indeed. Proving some system is ergodic seems really hard (and for some systems, I think in fact non-ergodicity is proven), and even after you do prove it, it's not clear if solves all your problems. Khinchin largely bypasses ergodicity because he focuses on sum functions of phase variables (ones that are approximately constant on a surface of constant energy surface in the thermodynamic limit) and uses the central limit theorem on defined probability distributions instead. This lets him get away with not proving ergodicity, but it does make his results very narrowly defined (I think). It seems that he narrows his focus in favor of mathematical rigor.

Last musing -- I am much more interested in the topic now that I see these foundational issues which are no longer being glossed over by me. Although I still don't think I quite understand all the pieces well enough, the shape of what the foundations actually are seem slightly more clear to me now than 3 months ago.