Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Outlier detection

  1. Apr 12, 2013 #1
    Hi all, I have data similar to the following

    plot.png

    where the x axis is time and the y axis is magnetic field. At around t = 20 (and t = -80) there is a systematic error (probably due to some other current switching on and then switching off) which I want to get rid of in my data.

    Can anyone recommend a good algorithm to detect when this happens in my time series and exclude it from my data set?

    I plotted the moving average too which seems to indicate it is not as simple as simply searching for large deviations from the mean.
     
  2. jcsd
  3. Apr 15, 2013 #2

    Stephen Tashi

    User Avatar
    Science Advisor

    Moving averages can be taken over windows of various sizes and the windows can include both the past and future. You could try various windows.

    Your goal isn't precisely defined yet. It could be either one of the following:

    1) I want an algorithm to detect the regions of the curve affected by switching currents. Suggest an algorithm. I'll try it and decide myself if it works. There doesn't have to be any statistical justification for it. This is not for a published paper or anything that needs academic scrutiny.

    2) I want an algorithm that can stand academic scrutiny and not attract criticism if I write up what I'm doing as a report.
     
  4. Apr 16, 2013 #3
    Option 1 would be fine for me
     
  5. Apr 16, 2013 #4

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    Judging from the curve, you have very large differences between adjacent bins at the edges of those outliers. If you just plot ##|n_i-n_{i-1}|##, they should give two nice peaks. Use the moving average of a few bins instead of the original values if the dataset is too noisy.
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook