How to filter erroneous reading from distribution of weightsby j777 Tags: distribution, erroneous, filter, reading, weights 

#1
Aug2508, 10:25 AM

P: 148

Hi,
I'm working with distributions of weights that are predominently "normal". The weights on the upper end of the distribution are in error and I'd like to find a method that I can use to automatically "chop" off this portion of the distribution. Based on my inexperienced inspection of these distributions it appears as though using the mean +/ the standard deviation as the range for "good" data and throwing everything else away yields a fairly accurate distribution but I'm not convinced that this is the correct/best way of filtering out the bad data. I'm not a statistics expert so I'm hoping somebody who is can point me in the right direction. Thanks 



#2
Aug2508, 01:30 PM

Math
Emeritus
Sci Advisor
Thanks
PF Gold
P: 38,902

There is no "correct" way to do that without knowing what the true distribution is. And even with a specific normal distribution, it is always possible to have some "outliers". If you are looking for a way to throw away "erroneous" measurements, you will need some methods, outside of the data itself, to decide which measurements are "erroneous".




#3
Aug2508, 02:57 PM

HW Helper
P: 1,344

Throwing out values willynilly simply because they seem "too high" is an incredibly bad bit of work.
If you are concerned about the influences of values, either high or low, you should try a method that is more robust than the traditional mean+standard deviation (i.e., normaldistributionbased) theories. You have a variety of choices: trimmed means, medians, Huberestimators of location and scale, and so on. Noting that you are by admission "not a statistician", the simplest approach might be to start with a trimmed mean. More information on your problem and what you are trying to do would be useful. 



#4
Aug2508, 03:32 PM

P: 626

How to filter erroneous reading from distribution of weights
Plot the data, compute a few statistics, see what you get. What is your data reflective of? Are there any studies regarding this out there that suggest what kind of distribution/regression/etc. is appropriate?




#5
Aug2508, 04:06 PM

P: 148

Thanks everybody for your input.
statdad  One of the main calculations I'd like to do on the data uses the mean. Since the readings that are generally in error are on the upper end of the distributions I considered calculating the mean of the data within the interquartile range but I didn't find anything in my research that suggested that this is a good approach. Now that you've educated me on trimmed means I realize that this is in fact a common approach. As NoMoreExams suggests I'll do some computations using trimmed means and see what the results look like. 



#6
Aug2508, 04:16 PM

HW Helper
P: 1,344

Good  the primary downfall of deleting outliers based on "experience" or "gut feeling" is that even in the best of situations your biases guide your decisions. As I said, use of a robust methodology, with measures of location and scale designed to work together, will serve you well.




#7
Aug2508, 04:17 PM

P: 626

Actually statdad suggested using trimmed means :)




#8
Aug2708, 04:21 PM

P: 148

Using a trimmed median as a measure of location works pretty well but SD as a measure of scale doesn't work so well because of the outliers present in the data. I'm trying to understand the calculations involved in using S_{n} (proposed by Rousseeuw and Croux) to estimate scale. Can anybody walk me through the calculation?




#9
Aug2808, 02:49 PM

P: 148

Does anybody know anything about S_{n} or Q_{n} estimators?




#10
Aug2808, 03:04 PM

HW Helper
P: 1,344

The [tex] S_n [/tex] estimate I know of is
[tex] S_n = 1.1926\text{median}_{1 \le i \le n} \left( \text{median}_{1 \le j \le n} x_i  x_j  \right) [/tex] If you don't have software to calculate this you should be able to do it rather easily in a spreadsheet: Step 1: For each [tex] i [/tex] you calculate the median of [tex] x_i  x_j  [/tex] Step 2: The estimate is the median of all the quantities you calculated in step 1 multiplied by [tex] 1.1926 [/tex] The multiplication at the end is done to make the current estimate consistent in the case of normally distributed data. This estimate of scale does not require the underlying distribution for the data to be normal (or even symmetric). On a side note, you might find the information at this link http://www.technion.ac.il/docs/sas/qc/chap1/sect21.htm helpful. (You might not too, but it can't hurt to check it.) good luck  keep the questions coming if you have more. 



#11
Aug2808, 03:30 PM

P: 148

Thanks statdad. One question though: In step 1 am I right in saying that x_{i} is a datapoint and x_{j} is the previous data point and I must calculate the median of the absolute value of the difference between the two points?




#12
Aug2808, 03:45 PM

HW Helper
P: 1,344

No  the [tex] x_i  x_j [/tex] means that every difference is used. Since the absolute value is in there is some duplication in effort, and the cases where [tex] i = j [/tex] obviously cancel out, but unless your data set contains thousands of values the effort you'd expend in looking only at different xvalues would far exceed the savings in calculation time.
What software are you using? 



#13
Aug2808, 03:53 PM

HW Helper
P: 1,344

I think I misunderstood, or misanswered, your question. Let me try again.
Hope this (and all my responses) help. 



#14
Aug2808, 03:54 PM

P: 148

I'm writing the algorithm in C. In retrospect my explanation wasn't very clear. I'm trying to understand what step 1 involves as far is writing an algorithm to perform it.




#15
Aug2808, 03:56 PM

P: 148

OK I got it now! That's a lot of computations.




#16
Aug2808, 03:58 PM

P: 148

Thank you so much for the detailed explanation. Now I'm going to see how this measure of space performs compared to SD.




#17
Aug2808, 04:08 PM

HW Helper
P: 1,344

Can't help you with the C programming  it's been a loooong time since I did that.
As an aid in interpretation  the standard deviation, as well as the MAD you may have seen references to, both measure variability in terms of the distance data values are from a fixed reference ([tex] \overline X [/tex] for the standard deviation, the median for MAD), while [tex] S_n [/tex] measures variability in terms of (loosely) the distance between pairs of data values. 



#18
Aug2808, 04:18 PM

HW Helper
P: 1,344

One more comment  sorry. The (free) statistics software R is very powerful, runs on Windows, Linux, Mac 0S X, and others, and has a module to compute the robust estimates we're discussing.



Register to reply 
Related Discussions  
Why is the bandwidth of the RF filter is wider compared to that of an IF Filter?  General Engineering  4  
NASA develops 'mindreading' [nerve signal reading] system  Computing & Technology  12  
where is erroneous???  Precalculus Mathematics Homework  2  
Series RLC filter (trap filter)  Introductory Physics Homework  7  
Erroneous statements?  General Math  6 