Understanding the Uniform Probability Distribution in Statistical Ensembles

[A. Neumaier]

OK, what is the minimal amount of information that would trigger you to determine probabilities? How many coin flips is the minimum?

Are you so ignorant about statistical practice? It depends on the accuracy you want.

 

[stevendaryl]

Respectable scientists are no fools that would determine probabilities from the information you gave.

If they have the luxury of performing more tests, then they can put off making any kind of decision until they have more data. But at some point, you have to make a decision based on the data that you have.

 

[Demystifier]

Are you so ignorant about statistical practice? It depends on the accuracy you want.

You are smart, but I am smart too.
I want the minimal possible accuracy that will trigger you to assign some definite numbers as probabilities.

 

[stevendaryl]

Are you so ignorant about statistical practice? It depends on the accuracy you want.

There is never a point when you know that your probability estimate is accurate. There is never a point when you can say with certainty: "The probability of heads is between 45\% and 55\%."

 

[Demystifier]

There is never a point when you know that your probability estimate is accurate. There is never a point when you can say with certainty: "The probability of heads is between 45\% and 55\%."

In other words, all you have is the probability of probability of probability of probability of ...

 

[A. Neumaier]

sending someone to Mars [...] What do we do?

We developed uncertainty software for ESA - see Robust and automated space system design for a related publication.

The way of arriving at a design that is acceptable to all parties that have a say is a far more complex process than assigning of faulty probabilities to epistemic uncertainties. Idealized textbook decision procedures are as irrelevant there as are the textbook state reduction recipes when doing a real measurement of a complex process.

 

[A. Neumaier]

There is never a point when you can say with certainty

Physics is never about certainty but about the art of valid approximation.

We treat photons as massless not because we know it for certain but because the mass is known to be extremely small.

Every argument in physics that demands the exact knowledge of the numbers involved is extremely suspicious.

 

[stevendaryl]

We developed uncertainty software for ESA - see Robust and automated space system design for a related publication.

The way of arriving at a design that is acceptable to all parties that have a say is a far more complex process than assigning of faulty probabilities to epistemic uncertainties.

That doesn't mean it is more accurate or more rational. The actual process is as much about psychology (reassuring all interested parties) as it is dealing with uncertainty.

 

[stevendaryl]

Physics is never about certainty but about the art of valid approximation.

But there is no point where you know that your approximation is valid.

 

[A. Neumaier]

You are smart, but I am smart too.
I want the minimal possible accuracy that will trigger you to assign some definite numbers as probabilities.

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

 

[A. Neumaier]

But there is no point where you know that your approximation is valid.

This is why physics is slightly in flux, and why these are sometimes controversies.

 

[stevendaryl]

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

It doesn't make any difference if you include uncertainty in the probability, because the assignment of uncertainty is itself uncertain.

 

[A. Neumaier]

That doesn't mean it is more accurate or more rational. The actual process is as much about psychology (reassuring all interested parties) as it is dealing with uncertainty.

Yes, as everywhere in life. Including in science.

You need to convince your peers that you did your study according to scientific standards. Claiming a probability based on two coin tosses will convince no one.

 

[Demystifier]

I'll never assign definite probabilities. probabilities are always estimates, with an intrinsic uncertainty.

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

 

[A. Neumaier]

the assignment of uncertainty is itself uncertain.

Indeed. There is no certainty about real numbers obtained from experiment. It is never like $\pi$ or $e$ or $\sqrt{2}$.

 

[A. Neumaier]

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

Play this game by yourself, using a book on elementary statistics!

 

[stevendaryl]

OK, then just estimate probabilities, with the lowest possible accuracy, for the lowest possible number of coin flips. Can you perform such a lowest possible task?

You know how it's really done. People pick a cutoff C-- 5% or 1% or 0.1% or whatever. Then they flip the coin often enough so that they can say:

If the probability of heads were outside of the range p \pm \delta p, then the probability of getting these results would be less than C.​

So, relative to the cutoff choice C, they conclude that the probability is in the range p \pm \delta p. That's not actually what you want to know, but it's the best you can get, using frequentist methods.

 

[Demystifier]

Play this game by yourself, using a book on elementary statistics!

Are you a politician? (Never giving a direct answer to a tricky question.)

 

[A. Neumaier]

to a tricky question

To a physically meaningless question.

 

[Demystifier]

That's not actually what you want to know, but it's the best you can get, using frequentist methods.

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

 

[stevendaryl]

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

A way to win any argument is to declare any question that you don't have a good answer for "meaningless". I'm not saying that's what Neumaier is doing, but sometimes it seems that way.

 

[Demystifier]

To a physically meaningless question.

It's meaningful for practical purposes.

 

[A. Neumaier]

It's meaningful for practical purposes.

Only for this practical purpose:

to force Neumaier to admit that sometimes Bayesian methods make more sense

I prefer to spend my time on more interesting topics.

 

[Demystifier]

Only for this practical purpose:

I prefer to spend my time on more interesting topics.

Trying to trick you is fun and challenging, but not practical.
I head some really practical purposes in mind. For instance, assigning p(head)=1/2 with N=0 coin flips may be practical in many actual situations. For instance, I used that once for playing a strip game with a girl, but that's another story. (In short she used intuition while I used Bayesian probability, and the final result was ... well, more fun than practical. )

 

[stevendaryl]

The Bayesian criticism of frequentist methods is that what you want to know is: "What is the probability that my hypothesis is true, given these experimental results?" but all that frequentism can tell you is "What is the probability of getting these experimental results, under the assumption that my hypothesis is true?" The two are of course related through Bayes' formula, but only if you allow for there to be prior probabilities for non-statistical facts.

 

[vanhees71]

You know how it's really done. People pick a cutoff C-- 5% or 1% or 0.1% or whatever. Then they flip the coin often enough so that they can say:

If the probability of heads were outside of the range p \pm \delta p, then the probability of getting these results would be less than C.​

So, relative to the cutoff choice C, they conclude that the probability is in the range p \pm \delta p. That's not actually what you want to know, but it's the best you can get, using frequentist methods.

Well, what else can you do to test a hypothesis than to just use statistics to do this test? Which other way to test a probabilistic statement do you think you can do, if not repeating the measurement on a sufficiently large ensemble? I've never understood these "Bayesian" statements that the probabilities of the Kolomogorov axioms have another meaning to real-world phenomena than a statistical analysis in the spirit of the frequentist interpretation.

There are ways to "guess" probabilities given some limited information about the phenomena at hand like the maximum-entropy method using the Shannen-Jaynes (or for QT von Neumann) entropy as a measure of relative missing information given a probability distribution against a prior distribution. Nevertheless, to be sure to estimate the probabilities successfully you have to check it, and this is possible only in the frequentist spirit. Then you gain more information about the system.

On top of all these purely statistical analyses you have to keep in mind that the real art of experimental science is not the statistical analysis but the analysis of systematic uncertainties too! This is not solvable with pure mathematics but needs further tests and reference measurements etc. etc. Science is much more complicated and plagued by much more uncertainties than pure math!

 

[vanhees71]

A way to win any argument is to declare any question that you don't have a good answer for "meaningless". I'm not saying that's what Neumaier is doing, but sometimes it seems that way.

That's then called "philosophy", right? SCNR

 

[stevendaryl]

Well, what else can you do to test a hypothesis than to just use statistics to do this test?

I'm not saying that you can do anything else, but in terms of conditional probabilities, what you're doing is: (letting H be your hypothesis, and letting R be your results)

If P(R | H) < C, then the hypothesis is unlikely to be true.

That is just not logically valid reasoning. What you want is something like

If P(H | R) < C then the hypothesis is unlikely to be true.

The difference between these is illustrated with tests for cancer.

Suppose that you have a test for some rare type of cancer (affecting 1 in a million people). The test is 99% accurate. If a patient tests positive, is it likely that he has cancer? No, it's actually very unlikely. The probability that the test is wrong is much more likely than the probability that he has cancer.

Now, in this case, we actually already have a known prior probability of the hypothesis (that the patient has cancer) being true: 0.0001% But using P(R|H) as a stand-in for P(H|R) is just as fallacious if we don't have a prior probability of the hypothesis.

 

[atyy]

Regarding the Jaynes maximum entropy or whatever - in what sense is the entropy unique? For example, why should we use the Shannon entropy? Why not some other Renyi entropy? If there is no uniqueness, can the objective Bayes position be sustained?

 

[atyy]

Sure, but I want to force Neumaier to admit that sometimes Bayesian methods make more sense. But he's smart, he understands what I am trying to do, so that's why he doesn't answer my questions.

I thought I already did that. Neumaier wants to be coherent. Only subjective Bayesians can be coherent. Which is why I'm a frequentist, so I can be incoherent.

 

[stevendaryl]

That's then called "philosophy", right? SCNR

Well, there is a distinction between "A. Neumaier does not have a good answer to the question" and "It is impossible to answer the question"

 

[N88]

Well, there is a distinction between "A. Neumaier does not have a good answer to the question" and "It is impossible to answer the question"

Let A = A. Neumaier does not have a good - as judged by stevendaryl - answer to the question.
Let B = It is impossible to answer the question.
In Bayesian terms: P(AB) = P(A).P(A|B) = 1?

 

[Demystifier]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

 

[atyy]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

Did you take into account that at the operational level, the Frequentist results can be derived from Bayesian probability via the assumption of exchangeability?

http://www.stats.ox.ac.uk/~steffen/teaching/grad/definetti.pdf

 

[Demystifier]

Did you take into account that at the operational level, the Frequentist results can be derived from Bayesian probability via the assumption of exchangeability?

Isn't that just a special case of the idea that results of measurements can be derived from the theory?

 

[atyy]

Isn't that just a special case of the idea that results of measurements can be derived from the theory?

No, I don't think so. Exchangeability allows one to derive operations which in Frequentist practice assume iid. Here is another explanation http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture1.pdf

 

[Demystifier]

I think I have a simple way to resolve the frequentist vs Bayesian probability dilemma in physics. The rule is this:
1) To measure probability, you need frequentist probability.
2) To predict probability by theoretical methods, you need Bayesian probability.

Of course, true physics is always an interplay between experiment and theory, so true probability in physics is an interplay between frequentist and Bayesian. With such a view, I don't see any reason for further fight between the two philosophies. This is like fighting on whether physics is experimental or theoretical science, while it should be obvious that it is both.

With that insight, I think I can give a better answer to the initial question of this thread. The question was the meaning of the ensemble in statistical physics. Well, the concept of an ensemble is an experimental concept. On the other hand, statistical physics is a branch of theoretical physics. This suggests that one should not use ensemble in statistical physics. But somebody used to think in experimental terms may feel uneasy without ensembles, so he may want to restore ensembles in theoretical statistical physics. But then he must proclaim that the ensemble is a virtual imagined ensemble, not a real ensemble, and this is what creates confusion. What a virtual ensemble really means? How is it related to a real ensemble? Should I replace virtual ensemble average with a time average, so that the ensemble becomes more real? Do I need ergodic theorems for that? Or perhaps only quasi-ergodic ones? What is the relevant time scale that makes (quasi)ergodic theorems applicable?

Yes, it is possible to answer such questions, but it's not that simple. For that reason I think that statistical physics, as a branch of theoretical physics, is easier to formulate without ensembles, using only probability in the Bayesian form as primitive. Frequentist probability is needed too, but only as a derived concept, relevant for making measurable predictions.

 

[Demystifier]

No, I don't think so. Exchangeability allows one to derive operations which in Frequentist practice assume iid. Here is another explanation http://www.cs.berkeley.edu/~jordan/courses/260-spring10/lectures/lecture1.pdf

So do you think that this influences my conclusions in #133?

 

[vanhees71]

Regarding the Jaynes maximum entropy or whatever - in what sense is the entropy unique? For example, why should we use the Shannon entropy? Why not some other Renyi entropy? If there is no uniqueness, can the objective Bayes position be sustained?

Exactly that's my point. Whatever you do to make a better educated guess about probabilities than just making up something, to be sure you have to do experiments with sufficiently large ensembles. Of course, you use Bayes's formula for conditional probabilities, but that doesn't mean that you are a Bayesianist denying the frequentist interpretation of probabilities.

The Shannon entropy is so much dominant, because it's the right entropy for the usual kinetics with short-range interactions, where the entropy is additive, i.e., you don't have higher correlations. It's valid for gases of neutral particles, because there you have van der Waals interactions with potentials falling much faster then the inverse distance of the particles. It's even valid for plasmas, where the long-ranged Coulomb potential between the particles is screened (Debye screening), and you can use the Vlasov equation to push the long-range correlations in the mean field and keep the collision term with short-ranged interactions of an effective screened potential. This is not valid anymore, e.g., when simulating the structure formation in the universe, because the gravitational interaction is not screened, and there Shannon entropy is the wrong measure. So it depends on the underlying microscopic dynamics which kind of statistics leads to good descriptions. The more important are experimental validations of such models, and again you can only achieve this by statistics. In the universe you assume the cosmological principle and then average over large enough distances, because here of course you cannot "prepare" an ensemble of universes.

 

[vanhees71]

With that insight, I think I can give a better answer to the initial question of this thread. The question was the meaning of the ensemble in statistical physics. Well, the concept of an ensemble is an experimental concept. On the other hand, statistical physics is a branch of theoretical physics. This suggests that one should not use ensemble in statistical physics. But somebody used to think in experimental terms may feel uneasy without ensembles, so he may want to restore ensembles in theoretical statistical physics. But then he must proclaim that the ensemble is a virtual imagined ensemble, not a real ensemble, and this is what creates confusion. What a virtual ensemble really means? How is it related to a real ensemble? Should I replace virtual ensemble average with a time average, so that the ensemble becomes more real? Do I need ergodic theorems for that? Or perhaps only quasi-ergodic ones? What is the relevant time scale that makes (quasi)ergodic theorems applicable?

Yes, it is possible to answer such questions, but it's not that simple. For that reason I think that statistical physics, as a branch of theoretical physics, is easier to formulate without ensembles, using only probability in the Bayesian form as primitive. Frequentist probability is needed too, but only as a derived concept, relevant for making measurable predictions.

What's the "probablity in the Bayesian form"? In statistical mechanics you always "coarse grain" over many microscopic details to describe the relevant classical observables. E.g., you can take fluid mechanics, where you start with (quantum) many-body theory and derive transport equations, given a separation of spatial and temporal scales, i.e., microscopic high-frequent and short-ranged fluctuations you average over by your observations (measurement apparati) to get sufficiently detailed description for the macroscopic low-frequent and long-ranged changes of macroscopic variables. Here the phase-space cells must be large enough to lead to a sufficient averaging and "smoothing" of the highly fluctuating microscopic distributions to macroscopic phase-space distributions, and you have to truncate the practically infinite tower of $N$-body phase-space distribution functions (BBGKY Hierachy). On the level of transport equations it's usually the single-particle phase-space distribution function you are after, and this quantity makes sense only in this coarse-grained definition to begin with. Formally it can be achieved by gradient expanding the Wigner function, which is not a phase-space distribution but becomes one if appropriately coarse grained. Another step of simplification is to switch to hydrodynamics rather than transport descriptions, where you go in the limit that the mean-free path is much smaller than the typical variations of the usual macroscopic quantities like density, pressure, flow-velocity field, etc. Then you can assume local thermal equilibrium (leading to ideal fluid dynamics) and expansions (in terms of moments or Chapman-Enskog, etc.) (leading to various levels of viscous fluid dynamics, in the next linear order of deviations from local thermal equilibrium the Navier-Stokes equation).

All this uses more or less explicitly the frequentist interpretation too, and the "ensembles" are the states of a partial system, i.e., "microscopically large" but "macroscopically small" large region of space (and sometimes also time), "smearing" over the microscopic fluctuations and details to get macroscopic quantities in terms of such averages. These averages in practice are done by the measurement apparati themselves, and that's what statistical physics also does in the theoretical description/modeling. So there is no dichotomy between experiment and theory in statistical physics!

 

[atyy]

Exactly that's my point. Whatever you do to make a better educated guess about probabilities than just making up something, to be sure you have to do experiments with sufficiently large ensembles. Of course, you use Bayes's formula for conditional probabilities, but that doesn't mean that you are a Bayesianist denying the frequentist interpretation of probabilities.

I'm a Frequentist, so I agree with you.

However, my attack (based on the non-uniqueness of the entropy) is only against objective Bayesians like Jaynes.

Since subjective Bayesians freely admit their subjectivity, they are immune to our criticisms. Furthermore, they can show derive Frequentist behaviour. And they can show that their position is coherent. So while I am not a subjective Bayesian, I don't believe their position can be criticized, except for being impractical.

 

[stevendaryl]

Since subjective Bayesians freely admit their subjectivity, they are immune to our criticisms. Furthermore, they can show derive Frequentist behaviour. And they can show that their position is coherent. So while I am not a subjective Bayesian, I don't believe their position can be criticized, except for being impractical.

Yes, that's what I think--from the point of view of (subjective) Bayesianism, frequentism is a heuristic, or shortcut, to simplify reasoning. It's way too much work to assign probabilities to every possible hypothesis, and compute how all those probabilities change in light of new information.

 

[Demystifier]

What's the "probablity in the Bayesian form"?

In the context of statistical mechanics, it is the assumption that probability of a given state is proportional to the phase-space volume of that state. In equilibrium it is assumed to be true even for a single particle at a single instant of time, implying that the frequentist probability cannot be applied. Since frequentist probability cannot be applied, what remains is the Bayesian probability.

 

[A. Neumaier]

the concept of an ensemble is an experimental concept

No; it is primarily a theoretical concept. It is properly introduced the first time in statistical mechanics - in the theory of equilibrium states. The experimental part of this theory is called thermodynamics and has no notion of an ensemble.

The experimental version of the ensemble concept is ''repetition of an experiment under comparable conditions''. Experimenters do not talk about ensembles but about statistical procedures.

that the ensemble is a virtual imagined ensemble, not a real ensemble, and this is what creates confusion.

This is how Gibbs, who coined the term, defined it. It is perfectly reasonable; why should there be confusion?
The ensemble describes a single glass of water (or any other chemical substance or mixture), and the ensemble is a fiction employed to justify intuitively the statistical machinery to the single glass of water. It was necessary in Gibbs' days where the idea that mathematics provides only models and that the same math can describe very different subject matters was not yet developed. it was clear to Gibbs that no actual repetition was relevant for the interpretation of the results.

Today we know that probability and statistics can be applied to very deterministic subject matters such as the (fully objective!) distribution of prime numbers or rounding errors in computer calculations. Therefore no notion of ensemble or repetition under similar condition must be in the background - so the name is just a historical leftover (just as the particle notion in quantum field theory). Just as the abstract notion of a vector is typically unrelated to arrows in 2- or 3-dimensional space but just means elements of anything satisfying the vector space axioms, so the abstract notion of an ensemble is often unrelated to betting games or repetition of experiments (these are just some specific applications) but can be anything that gives rise to expectations with the usual properties.

 

[Demystifier]

No; it is primarily a theoretical concept. It is properly introduced the first time in statistical mechanics - in the theory of equilibrium states. The experimental part of this theory is called thermodynamics and has no notion of an ensemble.

You and me are starting from different points. Your starting experimental point is thermodynamics, which you want to explain statistically. My starting experimental point is coin flipping, which I want to explain statistically. My starting point is much more basic then yours, so I think that my perspective is more fundamental then yours. Of course, you think that probability cannot be applied to a single flipping of a coin, while I think it can. Yesterday we established that we cannot agree on that basic thing, so we also cannot agree on all other things that follow from this basic one.

 

[A. Neumaier]

My starting point is much more basic then yours, so I think that my perspective is more fundamental then yours.

I completely disagree. Coin flipping is not even a subject of physics, except if you are interested in the motion of a coin rather than in head or tail. Thus it is strictly speaking off-topic here.

Coins and measurements are human artifacts, hence cannot be basic for an understanding of physics (the only concern in this forum). The objects that already existed in Nature before humans appeared (water, light, etc.) are far more basic.

 

[Demystifier]

I completely disagree. Coin flipping is not even a subject of physics, except if you are interested in the motion of a coin rather than in head or tail. Thus it is strictly speaking off-topic here.

Coins and measurements are human artifacts, hence cannot be basic for an understanding of physics (the only concern in this forum). The objects that already existed in Nature before humans appeared (water, light, etc.) are far more basic.

Then let as agree that we disagree.

 

[vanhees71]

I completely disagree. Coin flipping is not even a subject of physics, except if you are interested in the motion of a coin rather than in head or tail. Thus it is strictly speaking off-topic here.

Coins and measurements are human artifacts, hence cannot be basic for an understanding of physics (the only concern in this forum). The objects that already existed in Nature before humans appeared (water, light, etc.) are far more basic.

Measurements are at the heart of physics. Physics is all about measurements and what we can learn about how nature behaves in terms of quantitative observation of objective (i.e., repeatedly confirmable) phenomena. In this sense physics is a complete human artifact. Nature couldn't care less about our understanding of her in terms of the natural sciences!

 

[Demystifier]

Measurements are at the heart of physics. Physics is all about measurements and what we can learn about how nature behaves in terms of quantitative observation of objective (i.e., repeatedly confirmable) phenomena. In this sense physics is a complete human artifact. Nature couldn't care less about our understanding of her in terms of the natural sciences!

If you said that in the context of quantum foundations, I might not agree. But here, in the context of foundations of statistical mechanics, I agree. Even though I like to think that physics, in general, might be something more then a human tool for a description of nature, I don't have problems with admitting that statistical physics is not much more than that.

 

[A. Neumaier]

Measurements are at the heart of physics. Physics is all about measurements and what we can learn about how nature behaves in terms of quantitative observation of objective (i.e., repeatedly confirmable) phenomena. In this sense physics is a complete human artifact. Nature couldn't care less about our understanding of her in terms of the natural sciences!

Measurements are human artifacts used to check and perhaps to arrive at physical theories. But they are nothing fundamental - they don't figure in Newton's laws, Einstein's general relativity, or the standard model. Moreover, how things are measured changes considerable with time, while the fundamental physics is supposed to be time-invariant (though less completely known at earlier times). Otherwise we couldn't apply physics to the past or the far distance (where we only see radiation from the far past).