How to filter erroneous reading from distribution of weights

[j777]

Hi,

I'm working with distributions of weights that are predominently "normal". The weights on the upper end of the distribution are in error and I'd like to find a method that I can use to automatically "chop" off this portion of the distribution. Based on my inexperienced inspection of these distributions it appears as though using the mean +/- the standard deviation as the range for "good" data and throwing everything else away yields a fairly accurate distribution but I'm not convinced that this is the correct/best way of filtering out the bad data.

I'm not a statistics expert so I'm hoping somebody who is can point me in the right direction.


Thanks

 

[HallsofIvy]

There is no "correct" way to do that without knowing what the true distribution is. And even with a specific normal distribution, it is always possible to have some "outliers". If you are looking for a way to throw away "erroneous" measurements, you will need some methods, outside of the data itself, to decide which measurements are "erroneous".

 

[statdad]

Throwing out values willy-nilly simply because they seem "too high" is an incredibly bad bit of work.
If you are concerned about the influences of values, either high or low, you should try a method that is more robust than the traditional mean+standard deviation (i.e., normal-distribution-based) theories.
You have a variety of choices: trimmed means, medians, Huber-estimators of location and scale, and so on. Noting that you are by admission "not a statistician", the simplest approach might be to start with a trimmed mean.
More information on your problem and what you are trying to do would be useful.

 

[NoMoreExams]

Plot the data, compute a few statistics, see what you get. What is your data reflective of? Are there any studies regarding this out there that suggest what kind of distribution/regression/etc. is appropriate?

 

[j777]

Thanks everybody for your input.

statdad -- One of the main calculations I'd like to do on the data uses the mean. Since the readings that are generally in error are on the upper end of the distributions I considered calculating the mean of the data within the interquartile range but I didn't find anything in my research that suggested that this is a good approach. Now that you've educated me on trimmed means I realize that this is in fact a common approach. As NoMoreExams suggests I'll do some computations using trimmed means and see what the results look like.

 

[statdad]

Good - the primary downfall of deleting outliers based on "experience" or "gut feeling" is that even in the best of situations your biases guide your decisions. As I said, use of a robust methodology, with measures of location and scale designed to work together, will serve you well.

 

[NoMoreExams]

Actually statdad suggested using trimmed means :-)

 

[j777]

Using a trimmed median as a measure of location works pretty well but SD as a measure of scale doesn't work so well because of the outliers present in the data. I'm trying to understand the calculations involved in using S_n (proposed by Rousseeuw and Croux) to estimate scale. Can anybody walk me through the calculation?

 

[j777]

Does anybody know anything about S_n or Q_n estimators?

 

[statdad]

The $$S_n$$ estimate I know of is

$$S_n = 1.1926\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j | \right)$$

If you don't have software to calculate this you should be able to do it rather easily in a spreadsheet:
Step 1: For each $$i$$ you calculate the median of $$|x_i - x_j |$$
Step 2: The estimate is the median of all the quantities you calculated in step 1 multiplied by $$1.1926$$

The multiplication at the end is done to make the current estimate consistent in the case of normally distributed data.
This estimate of scale does not require the underlying distribution for the data to be normal (or even symmetric).

On a side note, you might find the information at this link

http://www.technion.ac.il/docs/sas/qc/chap1/sect21.htm

helpful. (You might not too, but it can't hurt to check it.)

good luck - keep the questions coming if you have more.

 

[j777]

Thanks statdad. One question though: In step 1 am I right in saying that x_i is a datapoint and x_j is the previous data point and I must calculate the median of the absolute value of the difference between the two points?

 

[statdad]

No - the $$|x_i - x_j|$$ means that every difference is used. Since the absolute value is in there is some duplication in effort, and the cases where $$i = j$$ obviously cancel out, but unless your data set contains thousands of values the effort you'd expend in looking only at different x-values would far exceed the savings in calculation time.
What software are you using?

 

[statdad]

I think I mis-understood, or mis-answered, your question. Let me try again.

• Call your first data value $$x_1$$. Calculate every possible $$|x_1 - x_j |$$, then get the median of these values
• Take the second data value and calculate every possible $$|x_2 - x_j |$$, then get the median of these values
• Repeat the calculations shown above for every data value, taking the median of each set of differences. This gives you the $$\text{median}_{1 \le j \le n} |x_i - x_j|$$ terms
• Find the median of all the values found in the step immediately above this - this gives the $$\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j| \right)$$ term
• The final calculation is to multiply the result found above by $$1.1926$$ - this will give you $$S_n$$ for your data

Hope this (and all my responses) help.

 

[j777]

I'm writing the algorithm in C. In retrospect my explanation wasn't very clear. I'm trying to understand what step 1 involves as far is writing an algorithm to perform it.

 

[j777]

OK I got it now! That's a lot of computations.

 

[j777]

Thank you so much for the detailed explanation. Now I'm going to see how this measure of space performs compared to SD.

 

[statdad]

Can't help you with the C programming - it's been a loooong time since I did that.
As an aid in interpretation - the standard deviation, as well as the MAD you may have seen references to, both measure variability in terms of the distance data values are from a fixed reference ($$\overline X$$ for the standard deviation, the median for MAD), while $$S_n$$ measures variability in terms of (loosely) the distance between pairs of data values.

 

[statdad]

One more comment - sorry. The (free) statistics software R is very powerful, runs on Windows, Linux, Mac 0S X, and others, and has a module to compute the robust estimates we're discussing.