Calculating the standard deviation of the standard deviation

[Chain]

I was wondering if anyone could help me with calculating the standard deviation of the standard deviation. What I mean by this is say for example I roll a dice 100 times and then calculate the mean and standard deviation from the results I collected. The results are not going to be exact because I took a finite sample size $N$. I could calculate the standard deviation in the result for the mean which would be:

$\sigma/\sqrt{N}$

Where $\sigma$ is the true standard deviation not the measured one. I was wondering since it's possible to calculate the standard deviation in the mean whether it's also possible to do it for the standard deviation. Essentially I want to calculate the error in the standard deviation calculated from a finite sample size.

 

[Stephen Tashi]

You need to state your goal using more precise language. It isn't clear whether you are asking a question about "estimation" or about theoretical calculations, or whether you are merely asking a question about convention.

In your post you made the distinction between "the true standard deviation" and "the measured one". This is an important distiction. It applies to all parameters of the population being sampled - for example, the "true mean" ( which is the population mean) is different concept than the "measured mean", which is the sample mean. A common goal in statistics is to estimate a population parameter by doing computations on a sample. For a population parameter such as the standard deviation, there are several different formulas that can be used to estimate the population parameter. Which formula is "best" depends on how you define the precise the meaning of "best".

On the other hand there are conventional meanings to terms like , "the sample mean" and "the sample standard deviation". Unfortunately, different textbooks define "the sample standard deviation" in different ways. But once you select a definite meaning for that term, you can compute the sample standard deviation from a given sample of data. It doesn't matter what the data represents. Each single value in the sample might be a sample standard deviation computed from sample values of a different random variable.

The term "sample standard deviation" is often used to indicate a single number such as when we say "the sample standard deviation was 23.8". This is technically not correct. The sample standard deviation is a formula applied to values in the sample. The values in the sample are random variables. Hence the "sample standard deviation" is a random variable. What we should say is that "The realization of the sample standard deviation was 23.8" since this refers to one observation of a random variable. Since the sample standard deviation is a random variable, this random variable has a probability distribution and the distribution has parameters that specify its own population mean and population standard deviation. (This is what makes statistics complicated and where students in introductory courses get confused. It seems to be a snake swallowing its tail.)

If X is a given random variable, the standard deviation of the "population standard deviation" of X would have to be defined as zero because its population standard deviation of X is a constant. It doesn't depend on samples.

If we define the random variable Y to be the sample standard deviation of a set of N independent measurements of X then we can do theoretical calculations to compute the standard deviation of Y as a function of the population parameters of the distribution of X. These calculations don't involve using specific numbers from sample data.

If we don't assume the distribution of X is known then we can ask how to best estimate the parameters of the distributions of X and Y from data in a sample. However, this is not a precise question. The problem must be fleshed out by specifying what we do know about the distribution of X and how we intend to define "best". (The common definitions of "best" involve the technical definitons of "unbiased", "maximum liklihood", and "minimum expected square error". Different goals can lead to different formulas.)

 

[Chain]

Okay so if a roll a dice 100 times I would predict the mean value of the sum of all the rolls to be:

$\langle x \rangle = \sum_{n=1}^{100} \sum_{i=1}^6 P_i i = \sum_{n=1}^{100} \sum_{i=1}^6 i/6 = 350$

and we would expect the mean squared deviation to be:

$\langle x^2 \rangle - \langle x \rangle^2 = \sum_{n=1}^{100} (\sum_{i=1}^6 P_i i^2 - (\sum_{i=1}^6 P_i i)^2) = 100(15.17-3.5^2) = 292$

Which gives a standard deviation of:

$\sqrt{292}=17$

So I would expect the sum of all the dice to be 350$\pm$17. Now this has all been on pen and paper so let's say I actually roll 100 dice and calculate the average and standard deviation using the data I collected. I may end up with a result like 360$\pm$15. The mean and standard deviation values I get from actually rolling 100 dice are also random variables.

I want to know to what certainty I can claim my result for the standard deviation is correct. Obviously this will be a function of $N$ which is the sample size since if I rolled 1000 dice I would get a much more accurate value for the mean and standard deviation of the sample.

 

[Office_Shredder]

The sample standard deviation is typically defined by
$$\sigma = \sqrt{ \frac{1}{n-1} \sum_{k=1}^{n} (X_n - \overline{X}) }$$
Where $\overline{X}$ is the sample mean

To find the standard deviation of sigma you just want to calculate
$$\sqrt{ E(\sigma^2) - E(\sigma)^2}$$
Interestingly $E(\sigma^2)$ is just the true variance of the random variable (because the definition of $\sigma$ is that it's an unbiased estimator for the variance). Calculating $E(\sigma)^2$ looks challenging, which is interesting because usually this is the easier term to deal with. It would be easier to deal with the sample variance,
$$\sigma^2 = \frac{1}{n-1} \sum_{k=1}^{n} (X_n - \overline{X})$$
And the variance of the sample variance
$$E(\sigma^4) - E(\sigma^2)^2$$
Everything is now just a polynomial in the X_k so you should be able to calculate it in terms of the moments of your random variable.Alternatively you can start with an unbiased estimator of the sample standard deviation
http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation

This is highly dependent on which random variable you start with, and hard to calculate, but an asymptotically correct formula is given. Once you have that, $E(\sigma)^2$ is known and $E(\sigma^2)$ is something you can calculate because it will be some constant times the true variance.

I guess the moral of the story is that variance is a lot nicer than standard deviation

 

[mathman]

The sample standard deviation is typically defined by
$$\sigma = \sqrt{ \frac{1}{n-1} \sum_{k=1}^{n} (X_n - \overline{X}) }$$
Where $\overline{X}$ is the sample mean

To find the standard deviation of sigma you just want to calculate
$$\sqrt{ E(\sigma^2) - E(\sigma)^2}$$
Interestingly $E(\sigma^2)$ is just the true variance of the random variable (because the definition of $\sigma$ is that it's an unbiased estimator for the variance). Calculating $E(\sigma)^2$ looks challenging, which is interesting because usually this is the easier term to deal with. It would be easier to deal with the sample variance,
$$\sigma^2 = \frac{1}{n-1} \sum_{k=1}^{n} (X_n - \overline{X})$$
And the variance of the sample variance
$$E(\sigma^4) - E(\sigma^2)^2$$
Everything is now just a polynomial in the X_k so you should be able to calculate it in terms of the moments of your random variable.


Alternatively you can start with an unbiased estimator of the sample standard deviation
http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation

This is highly dependent on which random variable you start with, and hard to calculate, but an asymptotically correct formula is given. Once you have that, $E(\sigma)^2$ is known and $E(\sigma^2)$ is something you can calculate because it will be some constant times the true variance.

I guess the moral of the story is that variance is a lot nicer than standard deviation

You forgot to square the terms in the summation.

 

[Chain]

Thanks for the replies guys that was helpful :)

 

[Chain]

Actually sorry I think I need a bit more help, I have no idea how to go about evaluating

$E(\sigma^2)$ and $E(\sigma^4)$

The expectation value is usually defined as

$\frac{1}{N}\sum_{n=1}^N X_n$

Where $X_n$ is the $n^{th}$ value of the sample data. However for a given sample you only get one value of the variance so would I just use that one value of all $X_n$??

If that was the case then I would find the standard deviation in my measured value of the sample variance to be always be zero so I'm kinda confused.

 

[Chain]

Okay I just did some computer simulations and I found that the standard deviation of the measured variance of my sample of random numbers seemed to depend on the sample size as:

$\sigma^2/\sqrt{N}$

Where $\sigma^2$ is the mean value of the measured variance of the samples and $N$ is the sample size used to calculate the measured variance. To measure the standard deviation on my value of the measured variance I simply produced of large number of samples and calculated the variance for each one and calculated the standard deviation of all of the values.

$\sigma^2/\sqrt{N}$ tended to slightly overestimate the measured value of the standard deviation of the variances however the values were always close.

I think I've found my answer but some theoretical justification would be nice.

 

[statdad]

First note that the sample standard deviation and variance are $$s$$ and $$s^2$$ (not sigma and sigma squared as written above)

First note that
$$E(s^2) = E\left(\frac 1 {n-1} \sum_{i=1}^n (x_i - \bar x)^2\right) = %<br /> \frac 1 {n-1} \sum_{i=1}^n E\left((x_i - \bar x)^2\right)$$

 

[Chain]

Okay so that would give $E(s^2)=\frac{n}{n-1}s^2$ and

$E(s^4)= (\frac{n}{n-1})^2\sum_{i=1}^{n} E((x_i-\bar{x})^4)$

Okay thank you for the responses :)