How to filter erroneous reading from distribution of weights

1. Aug 25, 2008

j777

Hi,

I'm working with distributions of weights that are predominently "normal". The weights on the upper end of the distribution are in error and I'd like to find a method that I can use to automatically "chop" off this portion of the distribution. Based on my inexperienced inspection of these distributions it appears as though using the mean +/- the standard deviation as the range for "good" data and throwing everything else away yields a fairly accurate distribution but I'm not convinced that this is the correct/best way of filtering out the bad data.

I'm not a statistics expert so I'm hoping somebody who is can point me in the right direction.

Thanks

2. Aug 25, 2008

HallsofIvy

There is no "correct" way to do that without knowing what the true distribution is. And even with a specific normal distribution, it is always possible to have some "outliers". If you are looking for a way to throw away "erroneous" measurements, you will need some methods, outside of the data itself, to decide which measurements are "erroneous".

3. Aug 25, 2008

Throwing out values willy-nilly simply because they seem "too high" is an incredibly bad bit of work.
If you are concerned about the influences of values, either high or low, you should try a method that is more robust than the traditional mean+standard deviation (i.e., normal-distribution-based) theories.
You have a variety of choices: trimmed means, medians, Huber-estimators of location and scale, and so on. Noting that you are by admission "not a statistician", the simplest approach might be to start with a trimmed mean.

4. Aug 25, 2008

NoMoreExams

Plot the data, compute a few statistics, see what you get. What is your data reflective of? Are there any studies regarding this out there that suggest what kind of distribution/regression/etc. is appropriate?

5. Aug 25, 2008

j777

statdad -- One of the main calculations I'd like to do on the data uses the mean. Since the readings that are generally in error are on the upper end of the distributions I considered calculating the mean of the data within the interquartile range but I didn't find anything in my research that suggested that this is a good approach. Now that you've educated me on trimmed means I realize that this is in fact a common approach. As NoMoreExams suggests I'll do some computations using trimmed means and see what the results look like.

6. Aug 25, 2008

Good - the primary downfall of deleting outliers based on "experience" or "gut feeling" is that even in the best of situations your biases guide your decisions. As I said, use of a robust methodology, with measures of location and scale designed to work together, will serve you well.

7. Aug 25, 2008

NoMoreExams

Actually statdad suggested using trimmed means :-)

8. Aug 27, 2008

j777

Using a trimmed median as a measure of location works pretty well but SD as a measure of scale doesn't work so well because of the outliers present in the data. I'm trying to understand the calculations involved in using Sn (proposed by Rousseeuw and Croux) to estimate scale. Can anybody walk me through the calculation?

9. Aug 28, 2008

j777

Does anybody know anything about Sn or Qn estimators?

10. Aug 28, 2008

The $$S_n$$ estimate I know of is

$$S_n = 1.1926\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j | \right)$$

If you don't have software to calculate this you should be able to do it rather easily in a spreadsheet:
Step 1: For each $$i$$ you calculate the median of $$|x_i - x_j |$$
Step 2: The estimate is the median of all the quantities you calculated in step 1 multiplied by $$1.1926$$

The multiplication at the end is done to make the current estimate consistent in the case of normally distributed data.
This estimate of scale does not require the underlying distribution for the data to be normal (or even symmetric).

On a side note, you might find the information at this link

http://www.technion.ac.il/docs/sas/qc/chap1/sect21.htm

helpful. (You might not too, but it can't hurt to check it.)

good luck - keep the questions coming if you have more.

Last edited by a moderator: Apr 23, 2017
11. Aug 28, 2008

j777

Thanks statdad. One question though: In step 1 am I right in saying that xi is a datapoint and xj is the previous data point and I must calculate the median of the absolute value of the difference between the two points?

12. Aug 28, 2008

No - the $$|x_i - x_j|$$ means that every difference is used. Since the absolute value is in there is some duplication in effort, and the cases where $$i = j$$ obviously cancel out, but unless your data set contains thousands of values the effort you'd expend in looking only at different x-values would far exceed the savings in calculation time.
What software are you using?

13. Aug 28, 2008

I think I mis-understood, or mis-answered, your question. Let me try again.

• Call your first data value $$x_1$$. Calculate every possible $$|x_1 - x_j |$$, then get the median of these values
• Take the second data value and calculate every possible $$|x_2 - x_j |$$, then get the median of these values
• Repeat the calculations shown above for every data value, taking the median of each set of differences. This gives you the $$\text{median}_{1 \le j \le n} |x_i - x_j|$$ terms
• Find the median of all the values found in the step immediately above this - this gives the $$\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j| \right)$$ term
• The final calculation is to multiply the result found above by $$1.1926$$ - this will give you $$S_n$$ for your data

Hope this (and all my responses) help.

14. Aug 28, 2008

j777

I'm writing the algorithm in C. In retrospect my explanation wasn't very clear. I'm trying to understand what step 1 involves as far is writing an algorithm to perform it.

15. Aug 28, 2008

j777

OK I got it now! That's a lot of computations.

16. Aug 28, 2008

j777

Thank you so much for the detailed explanation. Now I'm going to see how this measure of space performs compared to SD.

17. Aug 28, 2008

Can't help you with the C programming - it's been a loooong time since I did that.
As an aid in interpretation - the standard deviation, as well as the MAD you may have seen references to, both measure variability in terms of the distance data values are from a fixed reference ($$\overline X$$ for the standard deviation, the median for MAD), while $$S_n$$ measures variability in terms of (loosely) the distance between pairs of data values.

18. Aug 28, 2008

One more comment - sorry. The (free) statistics software R is very powerful, runs on Windows, Linux, Mac 0S X, and others, and has a module to compute the robust estimates we're discussing.