Sample Variance: Why Divide by n-1?

[jackiefrost]

If I have a sample consisting of n measurements why is the sample variance the result of dividing by n-1 instead of n?

jf

 

[quadraphonics]

If I have a sample consisting of n measurements why is the sample variance the result of dividing by n-1 instead of n?

jf

Well, some texts/people use n, but the reason for using n-1 is to make the estimate unbiased. I.e., you want the expected value of your estimate to equal the true population variance, and this requires using n-1. I'll leave the details to you...

 

[jackiefrost]

I've been looking at my textbooks explanation and Wikipedia and Wolframs webpages. I'm still confused what they mean by "unbiased estimators" and what subtracting 1 in the denomenator has to do with much of anything in helping out with whatever seems to be the isssue here...

jf

 

[maverick280857]

Well, if you have (n-1) then the expectation of the so defined sample variance exactly equals the population variance. That is

E(S^2) = \sigma^2

the expectation is thus independent of n. But if you define it instead as

S'^2 = \frac{1}{n}\sum_{i=1}^{n}(X_{i}-\mu)^2

then since S'^2 = \frac{n-1}{n}S^2, you have

E(S'^2) = \frac{n-1}{n}E(S^2) = \frac{n-1}{n}\sigma^2

which goes to \sigma^2 in the limit n\rightarrow \infty but does not equal \sigma^2 for finite values of n. Thats why the sample variance defined with (n-1) in the denominator is called an unbiased estimator (of the population variance).

By the way, the (n-1) factor is a 'correction' for finite n. As you can see, in the asymptotic limit (only), both these definitions are equivalent.

The idea behind having estimators is to be able to find the population variance (and mean) which in an actual statistical problem, you don't know. So you construct functions (called estimators) which estimate the value of the population statistics from the sample statistics. The closer your estimator takes you to the population statistics, the better it is for you. In the large n limit, the properties of the sample approach the properties of the underlying distribution exactly.

 

[Tedjn]

I have had the same problem understanding this issue. Frequently, textbooks and online websites gloss over the issue with a pithy and unsatisfactory statement about degrees of freedom, leaving me to wonder whether the real explanation has anything to do with degrees of freedom at all.

maverick, your explanation cleared up a few things for me, but one problem remains:

Well, if you have (n-1) then the expectation of the so defined sample variance exactly equals the population variance.

Why is this, or is division by n-1 just a better estimator than division by n in the finite case. If so, why?

 

[quadraphonics]

I have had the same problem understanding this issue. Frequently, textbooks and online websites gloss over the issue with a pithy and unsatisfactory statement about degrees of freedom, leaving me to wonder whether the real explanation has anything to do with degrees of freedom at all.

It is degrees of freedom. Specifically, it's because you're already using the same data to estimate the mean; if you were to know the population mean ahead of time, and were only interested in the variance on its own, then the unbiased estimator would indeed use a denominator of n. What's more common, however, is that you need to first estimate the mean, and then use that estimate in your estimate of the variance. It's this cascaded method of estimation that throws off the variance estimator, and requires the n-1 denominator. Intuitively speaking, introducing the mean estimate into the variance estimator eliminates one degree of freedom because the mean estimate (which is just the population average), together with any n-1 of the samples, uniquely determines the other sample.

Why is this, or is division by n-1 just a better estimator than division by n in the finite case. If so, why?

Let's crank through it:

Assume we have n i.i.d. samples \left\{x_1,\ldots,x_n \right\} with mean \mu and variance \sigma^2. First, let's consider what would happen if we knew the true mean \mu and only wanted to estimate the variance:

E\left[ \sum_{i=1}^n (x_i - \mu)^2 \right] = \sum_{i=1}^n E\left[ (x_i - \mu)^2 \right] = n\sigma^2.

Which is to say that we'd use an estimator with denominator n to get an unbiased estimate. So far, so good, right? Now, let's examine what happens if we don't know \mu and instead need to estimate it, using the usual sample average estimator (which is unbiased):

E\left [ \sum_{i=1}^n\left( x_i - \frac{1}{n}\sum_{k=1}^n x_k \right)^2 \right ] = \sum_{i=1}^nE\left[ \left( x_i - \frac{1}{n}\sum_{k=1}^n x_k \right)^2 \right]

= \sum_{i=1}^n E \left[ x_i^2 - \frac{2}{n}x_i\sum_{k=1}^nx_k + \frac{1}{n^2}\left( \sum_{k=1}^n x_k \right)^2 \right]

= n\left( \sigma^2 + \mu^2 - \frac{2}{n}(\sigma^2 + n\mu^2) + \frac{1}{n^2}(n\sigma^2 + n^2\mu^2) \right) = (n-1)\sigma^2

So, we see that the terms arising from the mean estimator (which is a random variable) had the net effect of subtracting \sigma^2 from the sum, requiring a denominator of (n-1) for unbiasedness. I.e., it's like you're estimating the variance with a known mean, but only only (n-1) data points.

A more explicit way to demonstrate this is to write the mean estimate in terms of the true mean: \frac{1}{n}\sum_{i=1}^n x_i = \mu + \epsilon where E(\epsilon) = 0, Var(\epsilon) = \frac{\sigma^2}{n} and E(x_i \epsilon) = \frac{\sigma^2}{n}\,,\,\forall i. Then, the variance estimator looks like:

\sum_{i=1}^nE\left[ (x_i - \mu - \epsilon)^2 \right] = \sum_{i=1}^n E\left[ (x_i - \mu)^2 - 2(x_i-\mu)\epsilon + \epsilon^2 \right] = n\left( \sigma^2 - 2\frac{\sigma^2}{n} + \frac{\sigma^2}{n} \right) = (n-1)\sigma^2.

Comparing that derivation with the fist one (using the true mean, that is), it should be evident that the introduction of the "error term" (\epsilon) has had an effect equivalent to the elimination of a degree of freedom.

 

[Tedjn]

Thanks quadraphonics,

Your explanation looks very detailed, and I will take my time reading through it. In particular, it will take me a little time to wrap my head around algebra with the expectation value.

I'll probably respond again in awhile if I run into any problems, but thank you very much.

 

[jackiefrost]

Good explanations maverick and Quad! That's what I wanted. It makes sense. [Also - now I understand the n-1 from the "degrees of freedom" approach that some texts mention]

Thanks again.

 

[tioguerra]

Hi all,

First of all I apologize for reviving such an old thread.

I am currently trying to find a way to proof the same thing, but without relying on the definition of expected value, but I am not sure if this is possible at all. Intuitively it seems it should be possible, but I simply cannot find a way around it. Here is what I tried.

First of all I define \mu as the true population mean, and
\bar{x} = \frac{1}{n}\sum^n_{i=1}x_i
as the sample mean.

Then from that I define
\sigma^2 = \frac{1}{n}\sum^n_{i=1}(x_i - \mu)^2
as the sample variance estimated using the true mean and
s^2_{n-1} = \frac{1}{n-1}\sum^n_{i=1}(x_i - \bar{x})^2
as the unbiased sample variance estimated using the sample mean.

Similarly I also define
s^2_{n} = \frac{1}{n}\sum^n_{i=1}(x_i - \bar{x})^2
as the biased sample variance.

What I could do already is to proof that s^2_{n} has a bias, and also how much that bias is. But I could not find a way to proof that s^2_{n-1} doesn't have a bias.

Here is how I did to proof that s^2_{n} has a bias:

If I take the term (x_i-\bar{x})^2 and both add and subtract \mu inside the parenthesis, then the expression expands this way:
(x_i-\bar{x} - \mu + \mu)^2=(x_i-\mu)^2-2\bar{x}(x_i-\mu)+2\mu(x_i-\mu)+(\bar{x}-\mu)^2
Well, since
ns^2_{n-1} = \sum^n_{i=1}(x_i - \bar{x})^2
Then from the expression before I can get the following sums:
ns^2_n = \sum^n_{i=1}(x_i-\mu)^2-2\bar{x}\sum^n_{i=1}(x_i - \mu) + 2\mu\sum^n_{i=1}(x_i - \mu)+\sum^n_{i=1}(\bar{x} - \mu)^2
Here I can use the fact that
\sum^n_{i=1}c = (c+c+\cdots+c)=n c
and that
\sum^n_{i=1}x_i = n \bar{x}
and then simplify a lot the above expression this way (sorry if I am too verbose):
ns^2_n=\sum^n_{x=1}(x_i-\mu)^2-2\bar{x}(n\bar{x}-n\mu)+2\mu(n\bar{x}-n\mu) + n(\bar{x}-\mu)^2=\sum^n_{x=1}(x_i-\mu)^2-n(\bar{x}-\mu)^2

Therefore:
s^2_n = \sigma^2-(\bar{x}-\mu)^2
Here the second term of the right hand side is an always negative bias that is present as long as the sample mean differs from the population mean.

My real problem is the second part. I cannot find a way to show that s^2_{n-1} is unbiased... it seems I always end up going in circles... if I try to make s^2_{n-1} = \sigma^2 then from the definitions above I end up with s^2_{n-1} = \sigma^2 = n(\bar{x} - \mu)^2 which simply does not make sense to me!

Does anyone have any idea how I could solve this problem?