Outlier Detection - Algorithm to Exclude Systematic Error from Data Set

[vibe3]

Hi all, I have data similar to the following

where the x-axis is time and the y-axis is magnetic field. At around t = 20 (and t = -80) there is a systematic error (probably due to some other current switching on and then switching off) which I want to get rid of in my data.

Can anyone recommend a good algorithm to detect when this happens in my time series and exclude it from my data set?

I plotted the moving average too which seems to indicate it is not as simple as simply searching for large deviations from the mean.

 

[Stephen Tashi]

I plotted the moving average.

Moving averages can be taken over windows of various sizes and the windows can include both the past and future. You could try various windows.

Your goal isn't precisely defined yet. It could be either one of the following:

1) I want an algorithm to detect the regions of the curve affected by switching currents. Suggest an algorithm. I'll try it and decide myself if it works. There doesn't have to be any statistical justification for it. This is not for a published paper or anything that needs academic scrutiny.

2) I want an algorithm that can stand academic scrutiny and not attract criticism if I write up what I'm doing as a report.

 

[vibe3]

Option 1 would be fine for me

 

[mfb]

Judging from the curve, you have very large differences between adjacent bins at the edges of those outliers. If you just plot $|n_i-n_{i-1}|$, they should give two nice peaks. Use the moving average of a few bins instead of the original values if the dataset is too noisy.