Negative variance of an observable quantity

[Mandragonia]

Quantum mechanics has a well-known procedure for evaluating the expectation value of an observable quantity in a given quantum state. First one must obtain the quantum operator O that is associated with the observable quantity. Then the rule for computing the expectation value is: Apply O to the wave function psi, multiply the result with psi*, and finally integrate over spatial variables.

The procedure described above can be extended to evaluate the variance of an observable quantity. Now the rule is: apply the operator O twice in succession to the wave function psi, multiply with psi* and integrate over space. From this result, subtract the square of the expectation value.

All this is in clear analogy with the standard rules of statistics. However, in QM things are less clear-cut. For example, it can be demonstrated that the variance may acquire a negative value!

My question is: How should one interpret a negative value for the variance of an observable quantity?

 

[Bill_K]

No, this is false. In QM as well, the variance of an observable is nonnegative.

<A²> = <ψ|A²|ψ> = <ψ|A(Aψ)> = <Aψ|Aψ> ≥ 0

 

[atyy]

Maybe the OP was thinking about negative entropy?
http://arxiv.org/abs/quant-ph/9512022
http://arxiv.org/abs/quant-ph/0702225

 

[Mandragonia]

No, this is false. In QM as well, the variance of an observable is nonnegative.

Okay, thank you! I think I have a counter-example, but I could be wrong of course.

<A²> = <ψ|A²|ψ> = <ψ|A(Aψ)> = <Aψ|Aψ> ≥ 0

Sorry, but I don't understand your third equality. Applying the operator A twice to the wave function on the right may well result in something that correlates negatively with the wave function on the left. However, if you are allowed to shift one operator to the left, then the result becomes indeed positive (or better non-negative) as you conclude.

For example, let psi(x) = sin(x) and operator A is the first derivative with respect to x.
Then your third term equals -sin(x)*sin(x) whereas your fourth term equals cos(x)*cos(x).

 

[The_Duck]

All observables are Hermitian; the definition of a Hermitian operator is that it satisfies

$\langle \phi | A \psi \rangle = \langle A \phi | \psi \rangle$

for any states $| \phi \rangle$ and $| \psi \rangle$. As Bill_K showed, this immediately implies that $A^2$ is "positive semidefinite," i.e., that its expectation value is always nonnegative. This is intuitive if you know that Hermitian operators have only real eigenvalues. Squaring the operator squares the eigenvalues, so the squared operator has only real nonnegative eigenvalues.

 

[Mandragonia]

Thank you. I will have to check whether my example is in accordance with the Hermitian rules.

 

[strangerep]

I will have to check whether my example is in accordance with the Hermitian rules.

You might also want to check whether your example is indeed a normalizable wave function.

If it's not, then it's not in the usual Hilbert space and everything becomes far more challenging...

 

[Mandragonia]

I spent a considerable amount of time checking my calculations. There are things I can't get right.

Observable quantity: Kinetic energy
Associated operator: constant * Laplace operator (L)
Wave function: ground state of Hydrogen atom in spherical coordinates (psi)

A simple example, everything is real. Now according to the Hermitian postulate (psi, LL psi) = (L psi, L psi). But this can not possibly be true! Here is just of many reasons: L takes first and second derivative with respect to r, LL third and fourth derivative. So if psi is a polynomial of degree 1 or 2, then the left hand side is zero, whereas the right hand is some positive function.

 

[strangerep]

A simple example, everything is real. Now according to the Hermitian postulate (psi, LL psi) = (L psi, L psi). But this can not possibly be true! Here is just of many reasons: L takes first and second derivative with respect to r, LL third and fourth derivative. So if psi is a polynomial of degree 1 or 2, then the left hand side is zero, whereas the right hand is some positive function.

You're hitting well-known problems associated with unbounded operators and continuous spectra.

If you want more explicit help, you'll have to show your work...

Firstly, write down your example $\psi$ explicitly (here) as such a polynomial. Then calculate $(\psi,\psi)$ explicitly (here). I presume your inner product is just an integral? What do you get?

 

[Mandragonia]

Okay, thank you very much for your offer! Here is my calculation. For clarity, I will omit all physical parameters and constants such as Planck's constant, electron mass, Bohr radius and pi.

psi(r) = exp(-r) ....... [wave function; the n=1 orbital of the Hydrogen atom]
L = (2/r)*(d/dr) + d^2/dr^2 ..... .. [radial part of the Laplace operator in spherical coordinates]
LL = (4/r)*(d^3/dr^3) + d^4/dr^4 ... [radial part of the Laplace operator applied twice in succession]

I calculate the expectation value of the Kinetic energy squared in two ways, which should be identical [if the Hermitian criteria are met]. First I give the result of the projection of the wave functions (f and g); next the result of multiplying by (r^2)*dr and integrating over the spatial variables, yielding F and G. Note that a * means multiplication (not complex conjugate!).

[1] f(r) = psi * (L(L psi)) = (-4/r + 1) * exp(-2*r)
F = integral (0 to inf) f(r) * r^2 * dr = -3/4

[2] g(r) = (L psi) * (L psi) = (4/r^2 - 4/r + 1) * exp(-2*r)
G = integral (from 0 to inf) g(r) * r^2 * dr = +5/4

I chose method [1], but this leads to a negative value. Why? Method [2] should be identical, but in fact leads to a different result (positive, and therefore perhaps the correct result?).

 

[Bill_K]

L = (2/r)*(d/dr) + d^2/dr^2 ..... .. [radial part of the Laplace operator in spherical coordinates]
LL = (4/r)*(d^3/dr^3) + d^4/dr^4 ... [radial part of the Laplace operator applied twice in succession]

No, when you calculate LL, there's a couple of terms you forgot. (... d²/dr² )(2/r d/dr + ...) when expanded produces d/dr and d²/dr² terms in addition to the ones you have.

 

[Mandragonia]

Did I make a mistake? I believe I performed the calculation with considerable care. The result for LL is what I got after expanding all the terms; in the final result most of the terms (those with the lower order derivatives) cancel. I also performed the calculation by first computing L(psi) explicitly and only then applying the second operator L.

 

[Bill_K]

Sorry, your expression is right.

 

[Mandragonia]

Thank you! The mystery continues...

In my calculations I obtained the general expression for applying N consecutive (radial) Laplace operators:

L^N = (2*N/r) * d^(2*N-1)/dr^(2*N-1) + d^(2*N)/dr^(2*N).

 

[strangerep]

Mandragonia,

Well, congratulations! Your calculations of those integrals above look correct to me.
So then it took me most of today to figure out WTF is going on here...

Here's an exercise to help you resolve the puzzle:

Exercise: Determine (carefully!) whether $L$ is/isn't self-adjoint. I.e., complete the following:
$$
(\phi, L\psi) ~\equiv~ \int_0^\infty\!\! dr\; r^2 \phi^* (2r^{-1}\partial_r
~+~ \partial^2_r ) \psi ~=~ \cdots ~,
$$where $\phi,\psi$ are functions of $r$. (Hint: use integration by parts twice.) What conditions on the wavefunctions must be satisfied for $L$ to be self-adjoint? I.e., under what conditions is it true that:
$$(\phi, L\psi) ~=~ (L\phi, \psi) ~~?$$
And btw, PLEASE learn how to use Latex on this forum.

 

[Mandragonia]

Thank you very much for your time and assistance!

I followed your advice and performed partial integration. The result I get is that the Laplace operator indeed satisfies the Hermitian criterium. However it is necessary that there is no contribution from the boundary terms. These terms can be combined into a single function f.

Define: f(r) = (r^2) (phi* psi' - psi phi*').

[* = complex conjugate; ' = derivative with respect to r]

The condition is that f(inf) - f(0) must be zero.

 

[stevendaryl]

I'm not sure if it is relevant to the original post, but Feynman wrote a little note about the use of negative probabilities in quantum mechanics here:
http://cds.cern.ch/record/154856/files/pre-27827.pdf

 

[Mandragonia]

I will continue the calculation started in my previous post. We are ready to apply the Hermitian rule derived in my previous post to the case discussed in this thread. So we set phi = exp(-r) and psi = L(phi) = (1 - 2/r)*exp(-r). It turns out that the
r -> inf boundary term vanishes, but the r=0 term doesn't. It picks up a contribution equal to -2.

Therefore, in this particular case, the Hermitian rule adjusts to:

(phi, L psi) = (L phi, psi) - 2 or equivalently (phi , LL phi) = (L phi, L phi) - 2.

This explains the difference between the two integrals, evaluated as -3/4 and +5/4 respectively!
An elegant line of reasoning, leading to a surprising and thought provoking result!

 

[strangerep]

Correct so far, afaict. For wavefunctions which vanish at infinity like $e^{-r}$, i.e., faster than any power of $r$, the condition is indeed:
$$
\lim_{r\to 0} \Big( r^2 (\phi^* \psi_r - \phi^*_r \psi) \Big) ~=~ 0 ~,~~~ \forall \phi, \psi ~,
$$ (where my subscripts denote a partial derivatives). However, this must hold for arbitrary $\phi,\psi$, so probably a stronger condition that each term vanish separately is justified.

[...] This explains the difference between the two integrals, evaluated as -3/4 and +5/4 respectively!
An elegant line of reasoning, leading to a surprising and thought provoking result!

Indeed. Here's some more food for thought...

Back in post #9, I made the cryptic remark:

You're hitting well-known problems associated with unbounded operators and continuous spectra.

The problems that arise in this case can broadly be expressed in the Hellinger--Toeplitz theorem:
en.wikipedia.org/wiki/Hellinger-Toeplitz_theorem

Wiki's explanation is rather brief, so a Functional Analysis textbook is needed to delve deeper. Kreyszig's textbook on FA is one that's reasonably friendly to physicsts and engineers, though still quite difficult. Typically, most QM textbooks don't explore such arcane results of Functional Analysis in much detail. The better texts might make a brief remark about how one must be careful about "domains of definition" and "domains of self-adjointness" when working with such operators.

But this still doesn't explain thoroughly why your kinetic energy operator behaves in such a recalcitrant manner for this (supposedly basic) problem of the nonrelativistic H-atom. Naively, one might complain: "well, I solved the Schrodinger equation and normalized the resulting wavefunctions, so what's gone wrong??"

If you'd like to explore further, try this:

Exercise: Repeat the earlier self-adjointness exercise, but this time for:
(i) the potential energy operator $V$ associated with this problem,
and then,
(ii) the full Hamiltonian $H := L+V$ .

 

[Hypersphere]

I'm not sure if it is relevant to the original post, but Feynman wrote a little note about the use of negative probabilities in quantum mechanics here:
http://cds.cern.ch/record/154856/files/pre-27827.pdf

I think it is slightly off the topic Mandragonia eventually wanted, but you might find the Wigner function interesting. It is basically an attempt to provide a probability distribution in phase space, but in some regions associated with unphysical states, the function takes negative values.

 

[bohm2]

I think it is slightly off the topic Mandragonia eventually wanted, but you might find the Wigner function interesting. It is basically an attempt to provide a probability distribution in phase space, but in some regions associated with unphysical states, the function takes negative values.

There is also Khrennikov's interpretation of negative probabilities:

...negative probabilities are absurd objects in the framework of the standard Kolmogorov theory of probability. We present a large class of non-Kolmogorovean probability models where negative probabilities are well defined on the frequency basis.These are models with probabilities which belong to the so-called field of p-adic numbers.

.
p-adic probability prediction of correlations between particles in the two-slit and neutron interferometry experiments
http://arxiv.org/pdf/0906.0509v1.pdf

 

[Mandragonia]

If you'd like to explore further, try this:

Exercise: Repeat the earlier self-adjointness exercise, but this time for:
(i) the potential energy operator $V$ associated with this problem,
and then,
(ii) the full Hamiltonian $H := L+V$ .

Ha! I had already completed all these calculations some time ago...! The peculiar result for the variance of the kinetic energy motivated me to start this thread. Well, I am glad this issue has been resolved.

It turns out the +5/8 result is satisfactory, because now the variance of the kinetic energy is equal to the variance of the potential energy. This makes physical sense. Psi is an eigen state of H; hence for every value of r the deviation from the mean for L(psi) equals minus the deviation from the mean for V(psi). Integrating the squares of these deviations over r, the two variances must be identical.

It is interesting to point out that the standard deviation in the kinetic energy is equal to twice the average value of the kinetic energy. This result suggests that there is a fair probability that the kinetic energy of the electron is negative! In classical physics this make little sense. However in QM the probability distribution stretches well beyond the boundary where the kinetic energy becomes negative (r > 2 * Bohr radius). To get a crude estimate of this probability, I suppose one may use a Gaussian model. The result is 31 %.

 

[George Jones]

The problems that arise in this case can broadly be expressed in the Hellinger--Toeplitz theorem:
en.wikipedia.org/wiki/Hellinger-Toeplitz_theorem

Wiki's explanation is rather brief, so a Functional Analysis textbook is needed to delve deeper. Kreyszig's textbook on FA is one that's reasonably friendly to physicsts and engineers, though still quite difficult. Typically, most QM textbooks don't explore such arcane results of Functional Analysis in much detail. The better texts might make a brief remark about how one must be careful about "domains of definition" and "domains of self-adjointness" when working with such operators.

There is a new book that, at a glance, seems to have nice treatments of all this, "Quantum Theory for Mathematicians" by Brian Hall.

https://www.amazon.com/dp/146147115X/?tag=pfamazon01-20

This looks like a really wonderful book. It has "the basics of L^2 spaces and Hilbert spaces" as prerequisites. Most of the functional analysis presented in the book is in chapters 6 - 10, through which the author gives several paths "I have tried to design this section of the book in such a way that a reader can take in as much or as little of the mathematical details as desired."

The Hellinger-Toeplitz theorem is not named, but it is presented and proved as Corollary 9.9.

 

[stevendaryl]

I think it is slightly off the topic Mandragonia eventually wanted, but you might find the Wigner function interesting. It is basically an attempt to provide a probability distribution in phase space, but in some regions associated with unphysical states, the function takes negative values.

A place where negative probabilities pop up is in the attempt to come up with a hidden-variables model for spin-1/2 twin particle EPR. To simplify the analysis, let's confine spin measurements to being along three axes:

A = the x-axis
B = the direction in the xy plane at 120 degrees clockwise from A.
C = the direction in the xy plane at 240 degrees clockwise from A.

So we assume that there is a hidden variable, \lambda that has 8 possible values:
\lambda_{ABC}, \lambda_{AB\overline{C}}, \lambda_{A\overline{B}C}, \lambda_{A\overline{B}\overline{C}}, \lambda_{\overline{A}BC}, \lambda_{\overline{A}B\overline{C}}, \lambda_{\overline{A}\overline{B}C}, \lambda_{\overline{A}\overline{B}\overline{C}}

where \lambda_{ABC} means that a spin measurement along any of the axes will produce spin-up, \lambda_{AB\overline{C}} means that a spin measurement along A or B will produce spin-up, while a measurement along axis C will produce spin-down, etc. (We assume, because of the perfect anti-correlation, that if one particle has spin-up along one axis, then the twin ha spin-down along that axis).

You can show that this hidden variables theory can only reproduce the quantum mechanical predictions if we assume the following probabilities for the various values of \lambda:

P(\lambda = \lambda_{AB\overline{C}}) <br /> = P(\lambda = \lambda_{A \overline{B}C})<br /> = P(\lambda = \lambda_{\overline{A} B C})<br /> =P(\lambda = \lambda_{\overline{A}B\overline{C}}) <br /> = P(\lambda = \lambda_{\overline{A} \overline{B}C})<br /> = P(\lambda = \lambda_{A \overline{B} \overline{C}}<br /> = \frac{3}{16})

P(\lambda = \lambda_{ABC}) <br /> = P(\lambda = \lambda_{\overline{A} \overline{B}\overline{C}})<br /> = -\frac{1}{16}

The impossibility of negative probabilities shows that no such hidden variables theory can exist.

 

[rlduncan]

A place where negative probabilities pop up is in the attempt to come up with a hidden-variables model for spin-1/2 twin particle EPR. To simplify the analysis, let's confine spin measurements to being along three axes:

A = the x-axis
B = the direction in the xy plane at 120 degrees clockwise from A.
C = the direction in the xy plane at 240 degrees clockwise from A.

So we assume that there is a hidden variable, \lambda that has 8 possible values:
\lambda_{ABC}, \lambda_{AB\overline{C}}, \lambda_{A\overline{B}C}, \lambda_{A\overline{B}\overline{C}}, \lambda_{\overline{A}BC}, \lambda_{\overline{A}B\overline{C}}, \lambda_{\overline{A}\overline{B}C}, \lambda_{\overline{A}\overline{B}\overline{C}}

where \lambda_{ABC} means that a spin measurement along any of the axes will produce spin-up, \lambda_{AB\overline{C}} means that a spin measurement along A or B will produce spin-up, while a measurement along axis C will produce spin-down, etc. (We assume, because of the perfect anti-correlation, that if one particle has spin-up along one axis, then the twin ha spin-down along that axis).

You can show that this hidden variables theory can only reproduce the quantum mechanical predictions if we assume the following probabilities for the various values of \lambda:

P(\lambda = \lambda_{AB\overline{C}}) <br /> = P(\lambda = \lambda_{A \overline{B}C})<br /> = P(\lambda = \lambda_{\overline{A} B C})<br /> =P(\lambda = \lambda_{\overline{A}B\overline{C}}) <br /> = P(\lambda = \lambda_{\overline{A} \overline{B}C})<br /> = P(\lambda = \lambda_{A \overline{B} \overline{C}}<br /> = \frac{3}{16})

P(\lambda = \lambda_{ABC}) <br /> = P(\lambda = \lambda_{\overline{A} \overline{B}\overline{C}})<br /> = -\frac{1}{16}

The impossibility of negative probabilities shows that no such hidden variables theory can exist.

Actually what you have shown (in a concise and eloquent manner using probabilities) is that the last two conditions are impossible. At least one of the three must be + or -, but all three simultaneously cannot have the same sign at the specified angles.

 

[dauto]

This hydrogen wavefunction is a solution to the Schrodinger equation everywhere except at the origin where the solution forms a cusp. The cusp means that the function is not differentiable at the origin, so it can't be a solution to a differential equation. That may, as you found out, cause boundary problems at the origin. The source of that problem is, off course, the fact that the Coulomb potential diverges at the origin. One possible way to attempt to solve that problem would be to replace the point charge with a Gaussian-like distribution of charge at the origin.

 

[strangerep]

Ha! I had already completed all these calculations some time ago...!

So... what results did you get for $(\psi, H^2\psi)$ and $(H\psi, H\psi)$ ?

The peculiar result for the variance of the kinetic energy motivated me to start this thread. Well, I am glad this issue has been resolved.

Hmm, so you think it has been fully resolved?

It turns out the +5/8 result is satisfactory,

What "5/8" result? Previously, you only quoted "5/4".

because now the variance of the kinetic energy is equal to the variance of the potential energy.

Please expand your reasoning here. I don't follow.

This makes physical sense. Psi is an eigen state of H; hence for every value of r the deviation from the mean for L(psi) equals minus the deviation from the mean for V(psi). Integrating the squares of these deviations over r, the two variances must be identical.

I don't follow this either. Please translate your words into math so I can see what you're really talking about.

 

[Mandragonia]

Time-independent Schroedinger equation: H psi = L psi + V psi = E psi ... with E constant
Multiply by psi and integrate: (psi, L psi) + (psi, V psi) = (psi, E psi)
This can be written alternatively as: <L> + <V> = E ... (where the brackets denote averages)
Substitute this into the Schroedinger eq.: (L-<L>) psi + (V-<V>) psi = 0
Define the difference operator D as L-<L>, which is therefore also equal to -(V-<V>).
Then (D psi, D psi) = <D^2> = variance of L = variance of V.

 

[stevendaryl]

Actually what you have shown (in a concise and eloquent manner using probabilities) is that the last two conditions are impossible. At least one of the three must be + or -, but all three simultaneously cannot have the same sign at the specified angles.

That's the same as saying that

P(\lambda = \lambda_{ABC}) = P(\lambda = \lambda_{\overline{A}\overline{B}\overline{C}}) = 0

But that assumption can't possibly reproduce the predictions of QM for this experiment.

Okay, here's a little more of the argument. By symmetry, let's assume that 6 of the probabilities are the same, and the other two are the same:
P(\lambda = \lambda_{\overline{A} B C})<br /> = P(\lambda = \lambda_{A \overline{B} C})<br /> = P(\lambda = \lambda_{A B \overline{C}})<br /> = P(\lambda_{A \overline{B} \overline{C}})<br /> = P(\lambda_{\overline{A} B \overline{C}})<br /> = P(\lambda_{\overline{A} \overline{B} C})

P(\lambda = \lambda_{A B C}) = P(\lambda = \lambda_{\overline{A} \overline{B} \overline{C}})

So let \alpha be P(\lambda = \lambda_{\overline{A} B C}) and let \beta be P(\lambda = \lambda_{A B C})

The QM probability that one particle is detected to have spin-up along axis A while the other particle is detected to have spin-down along axis B is:

\frac{1}{2}sin^2(\frac{120}{2}) = \frac{1}{8}

In terms of this hidden-variables model, there are two ways to get this result:

1. \lambda = \lambda_{A B C}
2. \lambda = \lambda_{A B \overline{C}}

So for the probabilities to work out, we would need:
P(\lambda = \lambda_{A B C}) + P(\lambda = \lambda_{A B \overline{C}}) = \frac{1}{8}

So in terms of \alpha and \beta, we have:
\beta + \alpha = \frac{1}{8}

The QM probability that one particle is detected to have spin-up along axis A while the other particle is detected to have spin-up along axis B is:

\frac{1}{2}cos^2(\frac{120}{2}) = \frac{3}{8}

In terms of our hidden-variables model, this implies:

P(\lambda = \lambda_{A \overline{B} C}) + P(\lambda = \lambda_{A \overline{B} \overline{C}}) = \frac{3}{8}

So in terms of \alpha and \beta, this means:
2 \alpha = \frac{3}{8}

So we have \alpha =\frac{3}{16} and \beta=-\frac{1}{16}

 

[Mandragonia]

Strangrep -- Thank you for showing an interest in my calculations. Apart from the <LL> problem, I encountered one other oddity. Namely that the operators VL and LV seemed to have different expectation values. I thought this effect could presumably be explained by the non-commutativity of the two operators. But I had no real clue how to resolve the issue.

Thanks to the discussions in this thread, I also found a satisfactory answer to this problem. Using Hermitian properties the expectation values should be rewritten in the form: (V psi, L psi) and (L psi, V psi). Now for the VL case the procedure is trivial (since V is multiplicative). For the LV case however, the Hermitian procedure leads to a boundary term. Taking this term into account, the LV and VL operators have the same expectation value. Nice!

It is still a bit mysterious to me that different representations co-exist, with sometimes different results. This raises the question how one can know a priori what the correct representation is. Suppose one is interested in three observables with operators A, B and C. First of all there are 6 permutations ABC, ACB etc. Secondly, each of these can be represented in four forms: (psi, ABC psi); (A psi, BC psi); (BA psi, C psi) and (CBA psi, psi). A messy situation. How should one proceed?