How can I effectively suppress Poisson noise in my data?

[arcTomato]

TL;DR Summary

Fourier transform

Hi all

I would like to know how to suppress the spectrum of Poisson noise.
At first, I tried "binning". I made the data of Poisson noise which sums up 4Hz sin wave.(the data number$N=10000$,and the data time is 1s) and I average out the data every 100bins. After this, I derived the power spectrum of the data like this.
Second image is the power spectrum without binning.

As you can see, the noise power spectrum isn't suppressed.
All signal (Poisson + sin) are suppressed.

Is this means binning has no effect on Poisson noise??
I would like to know why this happens and how to suppress the poisson noise.
If you can help me, Please.(and sorry for my bad English)

Thank you

 

[Tom.G]

Not my field, but Google seemed to help! Take a look at section 2.9 on pg 7 of the file:
https://arxiv.org/ftp/arxiv/papers/1505/1505.03489.pdf

above found with:
https://www.google.com/search?&q=poisson+noise+vs+gaussian+noise

Cheers,
Tom

 

[arcTomato]

Thank you @Tom.G !
I appreciate for your helping.

I have found this sentence of this paper(p13 on this paper):
https://dare.uva.nl/search?identifier=41d71571-7c62-4ef4-871d-87cfbfec499e

I'm not sure (cause my English is bud)but, does "This noisy character of the power spectrum can not be improved by taking a coarser time step" says about "binning"?

 

[Tom.G]

Not sure.
I think it says using fewer frequency bins in the Fourier transform does not improve the results. That it is better to use a greater number of bins on small sections of the signal and then processing (averaging?) the bins from different sections.

Time to get help from people with more knowledge of signal processing, ping @berkeman, can you help or suggest someone that can?

Cheers,
Tom

 

[arcTomato]

Thank you for helping @Tom.G !
I'm grad for you're so kind

 

[berkeman]

Time to get help from people with more knowledge of signal processing, ping @berkeman, can you help or suggest someone that can?

This is different from the signal processing that I typically work with. I think @Dr. Courtney may be able to help, though.

 

[arcTomato]

Hi @berkeman and Thank you.
I will wait for @Dr. Courtney !

 

[Dr. Courtney]

Averaging the data every hundred bins is like applying a kind of low pass filter. You are impacting the higher frequencies in the FT a lot more than the lower frequencies. It is also notable that different kinds of averaging techniques are possible. Playing with a moving average (each data point becomes the average of its N nearest neighbors) can provide some insight. Then trying moving weighted averages, where points further away are weighted less) can provide additional insight. But all these approaches are more like low pass filters.

Suppressing 4 Hz noise in your original signal would require a high pass filter or something that functioned like one. Off the top of my head, I can't recall simple averaging techniques that work like high pass filters. I'd recommend the extra work of a real high pass digital filter.

 

[arcTomato]

Thank you @Dr. Courtney !
So you mean that binning (=lowpass filter) cannot suppress the poisson noise, right?
But can we smooth the noise by "rebinning"(=binning after Forier transform) and dividing the data??

I think the sentence of the image I uploaded says like that.

 

[Dr. Courtney]

Thank you @Dr. Courtney !
So you mean that binning (=lowpass filter) cannot suppress the poisson noise, right?
But can we smooth the noise by "rebinning"(=binning after Forier transform) and dividing the data??

I think the sentence of the image I uploaded says like that.

That doesn't seem quite right. One doesn't really "bin" the data after the FT. One would apply some sort of cut off function (0 at low frequencies, 1 at high frequencies, monotonic transition in between) to the FT, and then one would apply an inverse Fourier transform to recover time domain data again with the low frequencies filtered out.

 

[Baluncore]

I would like to know why this happens and how to suppress the poisson noise.

You suppress the noise by lifting the wanted signal out of the noise.

A running average, (binning), is a Low-Pass Filter, LPF.
A LPF allows low-frequency noise to contaminate the detected signal.

You need a Band-Pass Filter, BPF, to reduce low-frequency and high-frequency noise.
The BPF required is constructed from the spectrum of the wanted signal.

 

[Baluncore]

Data = Poisson noise + 4 Hz signal.
You generate 10,000 data points over 1 second.

FFT will give the spectrum from 0 to 5 kHz, with resolution of 1 Hz.

But signal is at 4 Hz which is much too close to the DC = zero frequency noise component.
Change the 4 Hz generated signal component to a 400 Hz sinewave.
That will move the signal away from the sidelobes of the DC noise spike.

Increase the signal amplitude until you see the signal above the noise floor.
You may then also see unwanted aliases, and harmonics of the signal, generated by the FFT.
You can then test your detection filter that will improve signal to noise ratio.

 

[arcTomato]

Thank you @Dr. Courtney and @Baluncore !
Ok, I can understand about the filter.

I made the programs about the DFT, so I tried many things.
so I realized when I increase the number of data N=1000, I can recognize the difference between sin and noise.(because Parseval's theorem, I think)

I know this is not noise suppression(just increase power). but I can recognize so ,,,
Is this not practical?

 

[arcTomato]

Ok @Baluncore
I tried N=10000, sampling time $δt=0.01s$, so means all data time $T=100s$ version.
This is the spectrum.(I have changed the x-axis wave number to frequency)

In addition, I tried "rebinning". I average the data every 100 frequency bins.

It becomes like this.
The spectrum certainly is smoothed as the paper said, but I don't know this is meaning

 

[Baluncore]

You change too many parameters.
4 Hz is a very low carrier frequency, or data rate.

What is the origin of your real signal?
How will it change or how is it modulated?
Are you looking for a regular pattern in x-rays from pulsars?

You must specify the centre frequency and modulation bandwidth of the signal you will detect.
Then we can fix sensible data acquisition parameters that will work later with real data.

 

[arcTomato]

Thank you @Baluncore !

No, I just made a sine wave signal.So this is a general case.
And I think I have changed only the data number$N=$100→10000.($T=δt*N$)

Please let me organize my question.
As you can see the image, I think I can recognize the difference between sin and noise. Because I change the data number $N$, I think.
And When I rebinning the data $N=10000$ , the noise spectrum is smoothed.
But Does it have meaning?? I already recognized the difference.
And I hope my English isn't so bad

 

[Baluncore]

Your English is good.

But Does it have meaning??

Maybe, but probably not.
Your perfect sinewave signal has an integer frequency and is very big, so it is easy to see in the spectrum. I have not seen the time domain data, or generator code.

If your raw data is the envelope of random noise pulses, and that is buried in random noise, then elimination of noise may not be helpful.

Cyclic variation in a noise envelope will probably not be a sinewave. For example, pulsar data has a very regular cyclic envelope, with a distinct pulse shape, maybe with a double pulse. You must first generate a power envelope. Then you must detect regular variation of the envelope. Information will be carried by the fundamental and by phase of harmonics due to envelope pulse shape.

You must specify the real signal before you can detect it. You will not know if the noise reduction is useful until you process real data. You have focused on the noise, but it is the spectral character of the signal that is critically important.

Please define the signal that must be detected.
1. What is the origin of your real signal?
2. How will it change or how is it modulated?
3. Are you also looking for a regular pattern in x-rays from pulsars?

 

[arcTomato]

@Baluncore Thank you !
These are the answers.
1. I don't have the real signal.(I would like to research in a half year)
So, I just use the simple sine wave.

2. I just sums up Poisson noise and sine wave.

3. No, at least now.

 

[Baluncore]

Without a real signal there are so many possibilities that it is not sensible to guess what will be needed. The FT now gives you the amplitude and phase, or power of the known ideal 4 Hz sinewave.

 

[arcTomato]

Ok, I got it.
Thank you for teaching and helping me @Baluncore ,@Dr. Courtney , @berkeman @Tom.G . I will try hard.
If i post another question, I hope you help me next time too