# Outlier detection

1. Apr 12, 2013

### vibe3

Hi all, I have data similar to the following

where the x axis is time and the y axis is magnetic field. At around t = 20 (and t = -80) there is a systematic error (probably due to some other current switching on and then switching off) which I want to get rid of in my data.

Can anyone recommend a good algorithm to detect when this happens in my time series and exclude it from my data set?

I plotted the moving average too which seems to indicate it is not as simple as simply searching for large deviations from the mean.

2. Apr 15, 2013

### Stephen Tashi

Moving averages can be taken over windows of various sizes and the windows can include both the past and future. You could try various windows.

Your goal isn't precisely defined yet. It could be either one of the following:

1) I want an algorithm to detect the regions of the curve affected by switching currents. Suggest an algorithm. I'll try it and decide myself if it works. There doesn't have to be any statistical justification for it. This is not for a published paper or anything that needs academic scrutiny.

2) I want an algorithm that can stand academic scrutiny and not attract criticism if I write up what I'm doing as a report.

3. Apr 16, 2013

### vibe3

Option 1 would be fine for me

4. Apr 16, 2013

### Staff: Mentor

Judging from the curve, you have very large differences between adjacent bins at the edges of those outliers. If you just plot $|n_i-n_{i-1}|$, they should give two nice peaks. Use the moving average of a few bins instead of the original values if the dataset is too noisy.

Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook