Filtering Data Signals to Reduce Noise: Solutions?

[DanatAMFL]

Hello all.

I have a data signal that I'm looking to reduce the noise in. I'm recording voltage at 12000hz and am getting high frequency noise induced by nearby power supplies in the lines that I am using. The noise amplitude is approximately .2-.3 Volts and the signal I am trying to get out has an amplitude of approximately .8-.9 volts. The baseline hovers around 4V (so when a sample is detected at the sensor, it jumps up to ~5v, then back down to 4V when the sample leaves the sensor, samples only last at the sensor for a fraction of a second). Does anyone know a filtering technique that can smooth the noise in order for my signal squaring program to square the sample data and neglect the noise? The squaring program works off a slope criterion; when it sees a steep slope for x number of points in a row, it raises the squared signal, and does the same thing for lowering it. I have tried using a median filter, however the noise amplitude seems to be too large for it to work effectively. If I increase the N value (number of points to sample) in the median filter too much, it greatly affects my measurement. I need the processed signal to climb as soon as the raw signal begins to climb for a legitimate sample. Ultimately I'm looking to integrate the signal in order to find a ratio between the total amount of time that the sensor is detecting a sample to the total time of measurement.

Thanks for any help you guys can suggest!

 

[yungman]

I would work on the analog front end to reduce the noise first. You need to provide the schematic, a few picture on the front end, the cable that connect from the detector to the amplifier. Power supply noise is not the hardest to get rid of, just provide more info.

 

[carlgrace]

I agree with yungman, kill that noise at the source!

Make sure your power supply is properly bypassed... you shouldn't be getting HF from a power supply.

 

[jim hardy]

I'm ... getting high frequency noise induced by nearby power supplies in the lines that I am using.

Switching power supplies are notorious for clobbering low level measurements by "noise".
Back in my day we wouldn't put a switcher in same chassis with sensitive analog stuff.

First thing i'd try is see whether you can replace the offending power supplies with "Linear" type.

I'm recording voltage at 12000hz

Is that the frequency of interest, or your sample rate?

Also back in my day we sometimes used "Rockland" filters to clean up interference
https://www.amazon.com/dp/B0064SGFCY/?tag=pfamazon01-20
they were just an excellent adjustable filter with sharp cutoff.

Shielding - the basic premise is to build a Faraday cage around the signal source, masuring system and the cables in between.

i'm an old hardware guy. There's plenty of DSP filtering techniques too, and I'm not qualified in that field.

 

[DanatAMFL]

The power supplies that I was referring to are very large high power (on the order of a MegaWatt) power supplies that power the facility I am testing in. They can't be avoided. The lines from the sensor to the data acquisition system are relatively long and run past them. The facility is full of other powered equipment and instrumentation, so the noise I'm afraid may be unavoidable. I may look into sheilded wires, however they are expensive and I would need about 16 - 17 of them (I'm actually aquiring multiple signals from multiple sensors). I'd like to find a technique for evaluating the data before I spend the money on trying to physically reduce the noise. I can't show actual data from the facility, but I made a small plot to show you what I'm trying to accomplish.

 

[DanatAMFL]

12000 hz is my sample rate.

 

[AlephZero]

You could try putting the D/A converter as close as possible to the sensors. Digital data transmission over a "long" distance should be more immune to noise than analog.

If that isn't an optipon, RF shielded cable for signals of a few kHz shouldn't be expensive - try an audio microphone cable before you spend money on something exotic.

 

[yungman]

Fixing the noise is not necessary expensive. What you are dealing with is very common and there are a different kind of techniques to cut the noise down. You don't need to change power supply. I deal with micro volt level signal and I specifically use switching power supply because it's much smaller. I have no issue with noise when I do it right. Why don't you show the schematics, take a few pictures before you decide that digital filtering is the way to go.

I think you are making things difficult by making the wrong assumption. With noise 1/3 the amplitude of the signal, good luck in filtering it out reliably with only digital.

ON TOP, if your have a bad front end design, noise level and characteristics change with slight movement. Even you manage to fix it digitally, tomorrow, the problem might appear again. You have to fix it at the source. You move the wires, noise change and you're screwed again. Fix the problem at the source.

 

[jim hardy]

That's the trouble with rate detection(differentiation) it's a great noise detector.

By "squared" do you mean:
that you have arithmetically squared its value, as in RMS conversion,
or
that you have simply triggered a bistable to make a pulse with a square shape?

Looking at that graph you made
it looks to me like the question is not
"do i have slope> x"
but
"has my slope greater than x persisted for >y microseconds"

old jim

 

[Mike_In_Plano]

If the only option is to accept the corrupted signal, you'll always have some measure of difficulty when processing.

For short spikes, I find the following algorithm often helps:
1. Accumulate N samples, x[n], where N is an odd value (i.e. N=15)
2. Sort the values in either ascending or descending order
x[n] -> y[n], where y[n] < y[n+1] for n over 1 to N-1
3. Average the middle values of y[n] and that is your result.
(i.e. for N=15, result = (y[6]+y[7]+y[8]+y[9]+y[10]) / 5)

This works by clipping the highest and lowest values (from noise peaks) off.

If your signal is a tach pulse, and your not needing a rapid response an fft or phase locked loop can pull the signal free from the noise.

 

[NascentOxygen]

The lines from the sensor to the data acquisition system are relatively long and run past them. The facility is full of other powered equipment and instrumentation, so the noise I'm afraid may be unavoidable. I may look into sheilded wires, however they are expensive and I would need about 16 - 17 of them (I'm actually aquiring multiple signals from multiple sensors).

Perhaps you could run one additional line, but which carries no data. It would provide you with the noise signal, and you could subtract that from the signal on each data line. Should be easy enough to test to see whether it will prove successful.

 

[Enthalpy]

Perhaps you could run one additional line, but which carries no data. It would provide you with the noise signal, and you could subtract that from the signal on each data line. Should be easy enough to test to see whether it will prove successful.

Called a differential line. Transmission does not need a symmetrical signal, though it's often done so.

Cables exist with a differential pair plus shielding around both. Useful if strong currents flow in the shield.

Anyway, 0.2V stray if too much. Either this cabling is bad and should be improved, or the power supply is wrongly designed, but this is harder to improve.