Register to reply

How to filter erroneous reading from distribution of weights

Share this thread:
j777
#1
Aug25-08, 10:25 AM
P: 148
Hi,

I'm working with distributions of weights that are predominently "normal". The weights on the upper end of the distribution are in error and I'd like to find a method that I can use to automatically "chop" off this portion of the distribution. Based on my inexperienced inspection of these distributions it appears as though using the mean +/- the standard deviation as the range for "good" data and throwing everything else away yields a fairly accurate distribution but I'm not convinced that this is the correct/best way of filtering out the bad data.

I'm not a statistics expert so I'm hoping somebody who is can point me in the right direction.


Thanks
Phys.Org News Partner Science news on Phys.org
'Office life' of bacteria may be their weak spot
Lunar explorers will walk at higher speeds than thought
Philips introduces BlueTouch, PulseRelief control for pain relief
HallsofIvy
#2
Aug25-08, 01:30 PM
Math
Emeritus
Sci Advisor
Thanks
PF Gold
P: 39,682
There is no "correct" way to do that without knowing what the true distribution is. And even with a specific normal distribution, it is always possible to have some "outliers". If you are looking for a way to throw away "erroneous" measurements, you will need some methods, outside of the data itself, to decide which measurements are "erroneous".
statdad
#3
Aug25-08, 02:57 PM
HW Helper
P: 1,378
Throwing out values willy-nilly simply because they seem "too high" is an incredibly bad bit of work.
If you are concerned about the influences of values, either high or low, you should try a method that is more robust than the traditional mean+standard deviation (i.e., normal-distribution-based) theories.
You have a variety of choices: trimmed means, medians, Huber-estimators of location and scale, and so on. Noting that you are by admission "not a statistician", the simplest approach might be to start with a trimmed mean.
More information on your problem and what you are trying to do would be useful.

NoMoreExams
#4
Aug25-08, 03:32 PM
P: 626
How to filter erroneous reading from distribution of weights

Plot the data, compute a few statistics, see what you get. What is your data reflective of? Are there any studies regarding this out there that suggest what kind of distribution/regression/etc. is appropriate?
j777
#5
Aug25-08, 04:06 PM
P: 148
Thanks everybody for your input.

statdad -- One of the main calculations I'd like to do on the data uses the mean. Since the readings that are generally in error are on the upper end of the distributions I considered calculating the mean of the data within the interquartile range but I didn't find anything in my research that suggested that this is a good approach. Now that you've educated me on trimmed means I realize that this is in fact a common approach. As NoMoreExams suggests I'll do some computations using trimmed means and see what the results look like.
statdad
#6
Aug25-08, 04:16 PM
HW Helper
P: 1,378
Good - the primary downfall of deleting outliers based on "experience" or "gut feeling" is that even in the best of situations your biases guide your decisions. As I said, use of a robust methodology, with measures of location and scale designed to work together, will serve you well.
NoMoreExams
#7
Aug25-08, 04:17 PM
P: 626
Actually statdad suggested using trimmed means :-)
j777
#8
Aug27-08, 04:21 PM
P: 148
Using a trimmed median as a measure of location works pretty well but SD as a measure of scale doesn't work so well because of the outliers present in the data. I'm trying to understand the calculations involved in using Sn (proposed by Rousseeuw and Croux) to estimate scale. Can anybody walk me through the calculation?
j777
#9
Aug28-08, 02:49 PM
P: 148
Does anybody know anything about Sn or Qn estimators?
statdad
#10
Aug28-08, 03:04 PM
HW Helper
P: 1,378
The [tex] S_n [/tex] estimate I know of is

[tex]
S_n = 1.1926\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j | \right)
[/tex]

If you don't have software to calculate this you should be able to do it rather easily in a spreadsheet:
Step 1: For each [tex] i [/tex] you calculate the median of [tex] |x_i - x_j | [/tex]
Step 2: The estimate is the median of all the quantities you calculated in step 1 multiplied by [tex] 1.1926 [/tex]

The multiplication at the end is done to make the current estimate consistent in the case of normally distributed data.
This estimate of scale does not require the underlying distribution for the data to be normal (or even symmetric).

On a side note, you might find the information at this link

http://www.technion.ac.il/docs/sas/qc/chap1/sect21.htm

helpful. (You might not too, but it can't hurt to check it.)

good luck - keep the questions coming if you have more.
j777
#11
Aug28-08, 03:30 PM
P: 148
Thanks statdad. One question though: In step 1 am I right in saying that xi is a datapoint and xj is the previous data point and I must calculate the median of the absolute value of the difference between the two points?
statdad
#12
Aug28-08, 03:45 PM
HW Helper
P: 1,378
No - the [tex] |x_i - x_j| [/tex] means that every difference is used. Since the absolute value is in there is some duplication in effort, and the cases where [tex] i = j [/tex] obviously cancel out, but unless your data set contains thousands of values the effort you'd expend in looking only at different x-values would far exceed the savings in calculation time.
What software are you using?
statdad
#13
Aug28-08, 03:53 PM
HW Helper
P: 1,378
I think I mis-understood, or mis-answered, your question. Let me try again.
  • Call your first data value [tex] x_1 [/tex]. Calculate every possible [tex] |x_1 - x_j | [/tex], then get the median of these values
  • Take the second data value and calculate every possible [tex] |x_2 - x_j | [/tex], then get the median of these values
  • Repeat the calculations shown above for every data value, taking the median of each set of differences. This gives you the [tex] \text{median}_{1 \le j \le n} |x_i - x_j| [/tex] terms
  • Find the median of all the values found in the step immediately above this - this gives the [tex] \text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} |x_i - x_j| \right) [/tex] term
  • The final calculation is to multiply the result found above by [tex] 1.1926 [/tex] - this will give you [tex] S_n [/tex] for your data

Hope this (and all my responses) help.
j777
#14
Aug28-08, 03:54 PM
P: 148
I'm writing the algorithm in C. In retrospect my explanation wasn't very clear. I'm trying to understand what step 1 involves as far is writing an algorithm to perform it.
j777
#15
Aug28-08, 03:56 PM
P: 148
OK I got it now! That's a lot of computations.
j777
#16
Aug28-08, 03:58 PM
P: 148
Thank you so much for the detailed explanation. Now I'm going to see how this measure of space performs compared to SD.
statdad
#17
Aug28-08, 04:08 PM
HW Helper
P: 1,378
Can't help you with the C programming - it's been a loooong time since I did that.
As an aid in interpretation - the standard deviation, as well as the MAD you may have seen references to, both measure variability in terms of the distance data values are from a fixed reference ([tex] \overline X [/tex] for the standard deviation, the median for MAD), while [tex] S_n [/tex] measures variability in terms of (loosely) the distance between pairs of data values.
statdad
#18
Aug28-08, 04:18 PM
HW Helper
P: 1,378
One more comment - sorry. The (free) statistics software R is very powerful, runs on Windows, Linux, Mac 0S X, and others, and has a module to compute the robust estimates we're discussing.


Register to reply

Related Discussions
Why is the bandwidth of the RF filter is wider compared to that of an IF Filter? General Engineering 4
NASA develops 'mind-reading' [nerve signal reading] system Computing & Technology 12
Where is erroneous? Precalculus Mathematics Homework 2
Series RLC filter (trap filter) Introductory Physics Homework 7
Erroneous statements? General Math 6