Understanding Sigma and Sample Sizes - (high Sigma in small samples)

[Moppy]

Please correct me where I am wrong but it seems to me that you could generate a very high value of sigma (e.g. 6 sigma accuracy) from a very small sample size. How then is sigma on its own reliable?

Let me see if I understand sigma.

To determine the standard deviation I first compute the mean average. For each data point, I take the difference from the mean, square it, determine the average of the squares, and then take the square root. Is this that one sigma?

If it is one sigma, with a mean of 20 and standard deviation of 2, values of 18-22 represent one sigma accuracy? Values of <8 or >32 represent 6 sigma accuracy?

What additional checks or sample size must accompany the sigma value to make it reliable?

 

[chiro]

Hey Moppy and welcome to the forums.

You're thinking is correct: when we talk about standard deviations we talk about usually with respect to the mean. Also I'm assuming that you are referring to population parameters and not a sample in this case (samples and estimators are a little different).

So in your example for a distribution with a mean of 20 and s.d. of 2 we have a six-sigma interval from the mean of [20-6*2,20+6*2] = [8,32].

Now again this is for the population with a known PDF, population mean, and population variance/standard deviation.

When we deal with a sample, we are typically dealing with a scenario where we are trying to estimate something: typically a population parameter or some particular function of parameters (like a difference in means for example).

These have estimators in which the variance of the estimator is not fixed, but depends on the size of the sample in which the variance shrinks as you get a higher sample size for the general estimator distribution.

With estimators and statistics in general, the whole point is to try and minimize the sigma under some known assumptions. This is what statisticians try and do in every situation including experimental design, survey design, and for the construction of estimators. Low variance of an estimator means a more reliable estimator for the given sample size under the assumptions that are used.

But be aware that the true population parameters don't change at all.

Finally, be aware that an estimator for a parameter usually relates to a non-bounded interval (for the mean it's the whole real line, for the variance its the whole positive real line) and because of this, there will always be a chance that even under a huge interval (like a six-sigma one each side of the estimators distribution mean) you will make a wrong assertion in a hypothesis test: the technical term is the Type I and Type II errors that occur and the way to analyze this is by the power of the test/distribution for estimator.

 

[Stephen Tashi]

Please correct me where I am wrong but it seems to me that you could generate a very high value of sigma (e.g. 6 sigma accuracy) from a very small sample size. How then is sigma on its own reliable?

You are not being precise, that is where you are wrong. "Standard deviation" has many different meanings. Among these are:

1. The standard deviation of a probability distribution (also also called the "population standard deviation"

2. The function that defines how to compute the standard deviation of a sample. This is called the "sample standard deviation". (There are at least two different definitions for which formula to use, depending on which books you consult.)

3. A specific value of the sample standard deviation function, as in the phrase "the sample standard deviation" is 26.52".

4. A function that defines a formula for estimating the standard deviation of the population using as its inputs the values in a sample. This is "an estimator for the standard deviation"

5. A specific value of an estimator for the standard deviation, as in the phrase "the standard deviation is 26.52" . (It's better to say "An estimate for the standard deviation is 26.52".)

You haven't said what you mean by 6 sigma "accuracy". Presumably you are alluding to the fact that the probability is very high that that one random sample from a normal distribution will be within plus or minus six standard deviations of the mean, where "standard devation" means the standard deviation of the distribution and "mean" means the mean of the distribution. I suspect that you think that a similar statement also applies to the "sample standard deviation" and "sample mean". It does not apply.

Let me see if I understand sigma.

To determine the standard deviation I first compute the mean average. For each data point, I take the difference from the mean, square it, determine the average of the squares, and then take the square root. Is this that one sigma?

That is one way to define the "sample standard deviation" and it is one estimator for the population standard deviation. This estimator, on the average, underestimates the population standard deviation.

If it is one sigma, with a mean of 20 and standard deviation of 2, values of 18-22 represent one sigma accuracy? Values of <8 or >32 represent 6 sigma accuracy?

Are you taling about the sample mean and sample standard deviation or the population mean and population standard deviation? And we again have the question of what you mean by "six sigma accuracy".

What additional checks or sample size must accompany the sigma value to make it reliable?

If you are talking about specific numerical values of the sample standard deviation and the sample mean, it is not reliable to think that the probability is nearly 1 that a random sample from the distribution has a probability of nearly 1 of being within 6 of those particular sample sigmas from that particlar sample mean. This kind of thinking can't be made reliable.

 

[haruspex]

Let me see if I understand sigma.

To determine the standard deviation I first compute the mean average. For each data point, I take the difference from the mean, square it, determine the average of the squares, and then take the square root. Is this that one sigma?

If it is one sigma, with a mean of 20 and standard deviation of 2, values of 18-22 represent one sigma accuracy? Values of <8 or >32 represent 6 sigma accuracy?

What additional checks or sample size must accompany the sigma value to make it reliable?

Not sure, but I think you might be confusing different things here.
In the first part, you mention computing sigma from datapoints. Are the datapoints your small sample, or are they the entire dataset?
Anyway, you somehow have an estimate of sigma for some random variable. Now you mention so many sigmas of "accuracy". Accuracy of what?
The term is used in the business world in reference to the accuracy of a process, and it means that the probability that the error exceeds the quoted limit is less than that of a random (normal?) variable being more than 4.5 sigmas off the mean, or about 0.0003%. (A 1.5 sigma margin is left on the basis that what starts out as a genuinely 6 sigma process can be expected to degrade somewhat.)
A six sigma process can very easily be achieved merely by specifying a broad enough tolerance. If I set out to get all my darts within 10 metres of the bull on the dartboard, six sigma should be a doddle.
In a scientific context, you might quote six sigma proof of having detected a particle. This means that the probability of some other event resulting in the observation is less than 2 in a billion (really 6 sigma this time). Note that a single event (sample) can achieve this level of unlikelihood.
If the above doesn't pinpoint any misunderstanding you had, please try to clarify with an example.

 

[Moppy]

In a scientific context, you might quote six sigma proof of having detected a particle. This means that the probability of some other event resulting in the observation is less than 2 in a billion (really 6 sigma this time). Note that a single event (sample) can achieve this level of unlikelihood.

That's exactly what I don't understand. How can a single event be a reliable indication?

 

[chiro]

That's exactly what I don't understand. How can a single event be a reliable indication?

A single event has no variance and thus no variation, so you can't really do anything useful with it especially if you need to calculate any measure of sample variance or related quantities.

 

[Moppy]

That is what I am getting at. If the data set is small, does that make the number of sigmas less significant?

If the data is (9, 9, 9, 9, 9, 9, 9, 9, 9, 12) then that 12 is several sigma (probably at least 3 by quick mental maths) but how sigificant is that, vs the same number of sigma from a larger data set? If there's a difference, why is only the sigma value claimed as being the basis for a discovery?

 

[Stephen Tashi]

That's exactly what I don't understand. How can a single event be a reliable indication?

What you don't understand is difference between a sample standard deviation (as a specific numerical value) and the population standard deviation.

Haruspex's statement about a single event is referring to situations where the parameters of a distribution are known (or assumed).

Your original post isn't clear, but it appears to be asking about making deductions based on the sample standard deviation of a specific set of data when the standard deviation of the distribution is not known.

You suspect that making a "six sigma accuracy" claim (whatever you mean by that) based on the standard deviation of a small sample is invalid. This is correct. You suspect that the reason that making such a claim is invalid is that the standard deviation of a small sample may "exagerate". The correct way to state this is that the sample standard deviation of a small sample may be larger than the population standard deviation. This is correct, but it is also true that the sample standard deviation may also be smaller than the population standard deviation. It is also true that the sample standard deviation computed from a large sample may be larger or smaller than the population standard deviation. The basic problem with making inferences based on the sample standard deviation is that the sample standard deviation and the population standard deviation are not the same thing.

People often choose to assume that the population standard deviation is equal to the sample standard deviation of some sample. However, there is (obviously) no mathematical proof that it is, so people who do this are making a subjective decision. What mathematics can show is that expected error in such an estimate using a large sample size is smaller than the expected error using a small sample size. This mathematics does not prove that a specific estimate from a large sample (such as 26.52) is a good estimate for the population standard deviation. (For a specific value like 26.52, you don't know whether you were "lucky or unlucky" in your sampling.)

 

[haruspex]

That's exactly what I don't understand. How can a single event be a reliable indication?

It might be clearer with genuine examples. Suppose your null hypothesis is that Mars is lifeless. Various types of evidence, trace gases etc., might be deemed unlikely on that hypothesis, but not so unlikely as to provide 6 sigma evidence for life. But find an organism you can actually see and that one event is as many sigmas as you want.
I'm not being facetious here. For a single event to have 6 sigma significance it would have to be something pretty dramatic.
(chiro, Stephen T, I think this is what Moppy's question is about, not issues of how the s.d. is calculated.)

 

[Moppy]

Haruspex, kind of a mix of both. I was thinking more about e.g. new particle detection. You have a background signal which represents everything you see. And you can explain all of the spikes on it, except for one. So is that spike a new particle, or is that spike just random fluctuation.

I'm reading up on statistics at the moment because I do not understand how to make the sigma calculations in this case. I suspect it's something to do with using a sample of the population to predict a curve, and then matching your actual curve to the predicted one. I am not there yet but still reading and doing the example problems.

I will post more questions as they arise. The subject is proving more complex than I expected.