How do the candidates for Dark Energy 'work'?

[RogerWaters]

TL;DR Summary

What is the nature of the the candidates for Dark Energy: (1) Einstein's cosmological constant; (2) vacuum energy/virtual particles; (3) quintessence

When it comes to explaining some phenomena, I think in terms of mechanisms whereas physics explains through mathematics. With this in mind, I'm trying to compare and contrast the differences between the candidate explanations for dark energy, as nicely summarised on the NASA website: https://science.nasa.gov/astrophysics/focus-areas/what-is-dark-energy/

(1) Einstein's Cosmological Constant

"Albert Einstein was the first person to realize that empty space is not nothing. Space has amazing properties, many of which are just beginning to be understood. The first property that Einstein discovered is that it is possible for more space to come into existence. Then one version of Einstein's gravity theory, the version that contains a cosmological constant, makes a second prediction: "empty space" can possesses its own energy. Because this energy is a property of space itself, it would not be diluted as space expands. As more space comes into existence, more of this energy-of-space would appear. As a result, this form of energy would cause the universe to expand faster and faster."

Questions:

a) Why does more space come into existence, according to relativity?
b) Why does empty space possesses its own energy, according to relativity?

2. Vacuum Energy/Virtual Particles

"Another explanation for how space acquires energy comes from the quantum theory of matter. In this theory, "empty space" is actually full of temporary ("virtual") particles that continually form and then disappear. But when physicists tried to calculate how much energy this would give empty space, the answer came out wrong - wrong by a lot. The number came out 10^120 times too big. That's a 1 with 120 zeros after it. It's hard to get an answer that bad. So the mystery continues."

Question:

c) If virtual particles continually form in particle-antiparticle pairs, in most cases annihilating each other and disappearing, wouldn't the net energy be zero?

3. Quintessence

"Another explanation for dark energy is that it is a new kind of dynamical energy fluid or field, something that fills all of space but something whose effect on the expansion of the universe is the opposite of that of matter and normal energy. Some theorists have named this "quintessence," after the fifth element of the Greek philosophers. But, if quintessence is the answer, we still don't know what it is like, what it interacts with, or why it exists. So the mystery continues."

Questions:

d) How, broadly, is this the quintessence hypothesis of a scalar field causing space to expand different from the vacuum energy hypothesis?
e) Why does a field expand space? Other fields, such as the fields associated with electromagnetism, the Higgs field, etc. don't.

Thanks in advance. Apologies for the massive questions, as I'm sure people spend entire careers on each hypothesis and so I would chuffed with even the most general of answers, or being directed towards reading materials/videos etc.

 

[PeterDonis]

Our state of knowledge in this area is so poor that we don't even know for sure that the three "candidates" you describe are actually different possibilities, as opposed to just different viewpoints on the same possibility.

For example, these two questions...

a) Why does more space come into existence, according to relativity?
b) Why does empty space possesses its own energy, according to relativity?

...apply to anything that looks like dark energy, i.e., anything that causes accelerated expansion. So they apply to all three "candidates" as you describe them, not just the first. (And this is one indication that maybe the three are actually all the same thing, just looked at in different ways.)

Also bear in mind that the article you linked to is not a peer-reviewed paper, and should not necessarily be taken as a good description of what is going on. "Empty space being created" is one way of looking at accelerated expansion (or indeed expansion in general--see further comments below), but it has some serious limitations. (One obvious one is that our universe would be expanding even without dark energy, and that would also "create empty space" in the sense that that term is being used here, so your question a) above is really a question that applies to any expanding universe model, and the best way to answer it is to learn more details about the model so you understand what "empty space being created" means and what the limitations of that viewpoint are.)

c) If virtual particles continually form in particle-antiparticle pairs, in most cases annihilating each other and disappearing, wouldn't the net energy be zero?

This is a question that has to be answered by quantum field theory, not relativity. Unfortunately, quantum field theory's best current answer gives a value for "the energy density of empty space" that is about 120 orders of magnitude larger than what we observe. So clearly we still have a lot to learn in this area.

Note also that if we ever do figure out how to explain the energy density of empty space that we actually observe using QFT, that would just mean that candidates 1 and 2 on your list would be combined into one. As far as relativity is concerned, anything that looks like "energy density of empty space" will have the properties of a cosmological constant, as far as the expansion of the universe is concerned.

d) How, broadly, is this the quintessence hypothesis of a scalar field causing space to expand different from the vacuum energy hypothesis?

We would not expect "vacuum energy" to vary from point to point, because we would not expect "vacuum" itself to vary from point to point. Heuristically, in QFT terms, anything that does vary from point to point, by its very nature, cannot be "vacuum".

However, there is also no reason that a nonzero "vacuum energy" and a scalar field, that adds some energy density, possibly varying from point to point, to the vacuum energy, could not both be present. In which case we would just have a combination of candidate 2 (which, as above, is really just candidate 1 combined with a QFT explanation for why the vacuum energy is what it is) together with candidate 3.

Or, just to throw in more confusion, it could be that all of the accelerated expansion we currently observe is due to a scalar field, whose variation from point to point just happens to be so small that we do not (yet) detect it. In which case we would have candidate 3, but looking to us like candidate 1/2.

e) Why does a field expand space?

As above, space would be expanding even without any vacuum energy or scalar field. The question is not why a scalar field expands space. It's why a scalar field causes accelerated expansion. The answer is that the effective stress-energy tensor of a scalar field has the same property that a cosmological constant (aka "vacuum energy", see above) has: it has a pressure equal to minus its energy density. And that property is what causes accelerated expansion (just plug $p = - \rho$ into the second Friedmann equation).

Other fields, such as the fields associated with electromagnetism, the Higgs field, etc. don't.

The Higgs field actually does, since it's a scalar field; but its vacuum expectation value (basically another QFT term for "vacuum energy density") is much too small to explain the accelerated expansion we currently observe.

Other fields don't because they aren't scalar (spin 0) fields; the other Standard Model fields are all spin 1.

 

[PeroK]

With this in mind, I'm trying to compare and contrast the differences between the candidate explanations for dark energy, as nicely summarised on the NASA website: https://science.nasa.gov/astrophysics/focus-areas/what-is-dark-energy/

You have to be careful with these sites. For example, that page says:

Albert Einstein was the first person to realize that empty space is not nothing.

What?! What about the ether that everyone believed in pre-SR?

Who writes this stuff?

 

[RogerWaters]

You have to be careful with these sites. For example, that page says:

Albert Einstein was the first person to realize that empty space is not nothing.

What?! What about the ether that everyone believed in pre-SR?

Who writes this stuff?

For sure, I just found the NASA page to be a nice inventory of a bunch of different hypotheses I've been reading about in books but which were overlapping in my head.

 

[ohwilleke]

A lot of the recent work on dark energy is tied to the Hubble tension, in which the Hubble Constant shows a lower value (in a parameter that in theory should be constant) at high redshift (i.e. closer to the Big Bang) than at low redshift (i.e. when it is measured at comparatively recent time depths and closer distances), and to tensions in another more obscure cosmological parameter that relates to the amplitude of inhomogeneities in cosmology observations (σ₈).

The paradigm, the ΛCDM model of cosmology, embraces a fixed cosmological constant, Lambda (Λ), which is equivalent to a physical constant scalar field in every volume of space-time in the Universe, but can also be described, for example, as an intrinsic curvature of space-time in our universe. The ΛCDM model doesn't concern itself with the mechanism by which a cosmological constant arises.

Efforts to reconcile the Hubble tension have focused on a variety of ways that dark energy evolved (see, e.g. "Early Dark Energy" discussed in Herold 2021), or for example, a varying Newton's constant G (see e.g., Sakr 2021 testing this hypothesis and finding it wanting), or some other factor that matters (see e.g., the consequences of inflation theories as discussed in Belgacem 2021). Any of these might give rise to a reality other than a cosmological constant, and might instead cause the universe to evolve over time in a way that alleviates the tensions in the current observations.

A parallel effort argues that the Hubble tension is due not to real physical phenomena (or is only in part due to real physical phenomena) and is instead (in full or in part) due to systemic errors in measurements used to determine the Hubble constant at different redshifts (see e.g.,http://iv.org/abs/2111.07872, Goicoechea 2022. Greene 2021 and Spallicci 2021). Some of those criticisms are well founded, others less so.

One has to sort out both the methodological concerns, and the nature of the physical reality that one reaches after doing measurement that address all pertinent systemic errors, before one can distinguish between which set of mathematical behavior of the universe fits reality. And, one can only really make progress in considering possible mechanisms for this mathematical behavior once that is pinned down.

The good news is that there are a great many independent astronomy and theory groups working on gathering the information needed to answer these questions and analyzing it properly. This is being done with multiple different methodologies, so the end result is likely to be robust. Thus, there is good reason to hope that we will have data driven narrowing of the possibilities in our lifetimes and maybe even over the next several years to a decade or two.