Suppress of the Poisson noise

  • #1
79
24

Summary:

Fourier transform

Main Question or Discussion Point

Hi all

I would like to know how to suppress the spectrum of Poisson noise.
At first, I tried "binning". I made the data of Poisson noise which sums up 4Hz sin wave.(the data number##N=10000##,and the data time is 1s) and I average out the data every 100bins. After this, I derived the power spectrum of the data like this.
Second image is the power spectrum without binning.

1575597378890.png
1575597630165.png



As you can see, the noise power spectrum isn't suppressed.
All signal (Poisson + sin) are suppressed.

Is this means binning has no effect on Poisson noise??
I would like to know why this happens and how to suppress the poisson noise.
If you can help me, Please.(and sorry for my bad English)

Thank you
 

Attachments

Answers and Replies

  • #3
79
24
Thank you @Tom.G !
I appreciate for your helping.

I have found this sentence of this paper(p13 on this paper):
https://dare.uva.nl/search?identifier=41d71571-7c62-4ef4-871d-87cfbfec499e

スクリーンショット 2019-12-06 14.22.08.png

I'm not sure (cause my English is bud)but, does "This noisy character of the power spectrum can not be improved by taking a coarser time step" says about "binning"????
 
  • #4
Tom.G
Science Advisor
2,949
1,698
Not sure.
I think it says using fewer frequency bins in the Fourier transform does not improve the results. That it is better to use a greater number of bins on small sections of the signal and then processing (averaging?) the bins from different sections.

Time to get help from people with more knowledge of signal processing, ping @berkeman, can you help or suggest someone that can?

Cheers,
Tom
 
  • #5
79
24
Thank you for helping @Tom.G !
I'm grad for you're so kind :smile:
 
  • #6
berkeman
Mentor
56,502
6,402
Time to get help from people with more knowledge of signal processing, ping @berkeman, can you help or suggest someone that can?
This is different from the signal processing that I typically work with. I think @Dr. Courtney may be able to help, though. :smile:
 
  • #7
79
24
  • #8
Dr. Courtney
Education Advisor
Insights Author
Gold Member
3,041
2,003
Averaging the data every hundred bins is like applying a kind of low pass filter. You are impacting the higher frequencies in the FT a lot more than the lower frequencies. It is also notable that different kinds of averaging techniques are possible. Playing with a moving average (each data point becomes the average of its N nearest neighbors) can provide some insight. Then trying moving weighted averages, where points further away are weighted less) can provide additional insight. But all these approaches are more like low pass filters.

Suppressing 4 Hz noise in your original signal would require a high pass filter or something that functioned like one. Off the top of my head, I can't recall simple averaging techniques that work like high pass filters. I'd recommend the extra work of a real high pass digital filter.
 
  • #9
79
24
Thank you @Dr. Courtney !
So you mean that binning (=lowpass filter) cannot suppress the poisson noise, right?
But can we smooth the noise by "rebinning"(=binning after Forier transform) and dividing the data??

I think the sentence of the image I uploaded says like that.
 
  • #10
Dr. Courtney
Education Advisor
Insights Author
Gold Member
3,041
2,003
Thank you @Dr. Courtney !
So you mean that binning (=lowpass filter) cannot suppress the poisson noise, right?
But can we smooth the noise by "rebinning"(=binning after Forier transform) and dividing the data??

I think the sentence of the image I uploaded says like that.
That doesn't seem quite right. One doesn't really "bin" the data after the FT. One would apply some sort of cut off function (0 at low frequencies, 1 at high frequencies, monotonic transition in between) to the FT, and then one would apply an inverse Fourier transform to recover time domain data again with the low frequencies filtered out.
 
  • #11
Baluncore
Science Advisor
2019 Award
7,242
2,341
I would like to know why this happens and how to suppress the poisson noise.
You suppress the noise by lifting the wanted signal out of the noise.

A running average, (binning), is a Low-Pass Filter, LPF.
A LPF allows low-frequency noise to contaminate the detected signal.

You need a Band-Pass Filter, BPF, to reduce low-frequency and high-frequency noise.
The BPF required is constructed from the spectrum of the wanted signal.
 
  • #12
Baluncore
Science Advisor
2019 Award
7,242
2,341
Data = Poisson noise + 4 Hz signal.
You generate 10,000 data points over 1 second.

FFT will give the spectrum from 0 to 5 kHz, with resolution of 1 Hz.

But signal is at 4 Hz which is much too close to the DC = zero frequency noise component.
Change the 4 Hz generated signal component to a 400 Hz sinewave.
That will move the signal away from the sidelobes of the DC noise spike.

Increase the signal amplitude until you see the signal above the noise floor.
You may then also see unwanted aliases, and harmonics of the signal, generated by the FFT.
You can then test your detection filter that will improve signal to noise ratio.
 
  • #13
79
24
Thank you @Dr. Courtney and @Baluncore !
Ok, I can understand about the filter.

I made the programs about the DFT, so I tried many things.
so I realized when I increase the number of data N=1000, I can recognize the difference between sin and noise.(because Parseval's theorem, I think)
1575685903828.png

I know this is not noise suppression(just increase power). but I can recognize so ,,,
Is this not practical?
 
  • #14
79
24
Ok @Baluncore
I tried N=10000, sampling time ##δt=0.01s##, so means all data time ##T=100s## version.
This is the spectrum.(I have changed the x axis wave number to frequency)
1575686706411.png


In addition, I tried "rebinning". I average the data every 100 frequency bins.
1575687037513.png

It becomes like this.
The spectrum certainly is smoothed as the paper said, but I don't know this is meaning 😅
 
  • #15
Baluncore
Science Advisor
2019 Award
7,242
2,341
You change too many parameters.
4 Hz is a very low carrier frequency, or data rate.

What is the origin of your real signal?
How will it change or how is it modulated?
Are you looking for a regular pattern in x-rays from pulsars?

You must specify the centre frequency and modulation bandwidth of the signal you will detect.
Then we can fix sensible data acquisition parameters that will work later with real data.
 
  • #16
79
24
Thank you @Baluncore !

No, I just made a sine wave signal.So this is a general case.
And I think I have changed only the data number##N=##100→10000.(##T=δt*N##)

Please let me organize my question.
As you can see the image, I think I can recognize the difference between sin and noise. Because I change the data number ##N##, I think.
And When I rebinning the data ##N=10000## , the noise spectrum is smoothed.
But Does it have meaning?? I already recognized the difference.



And I hope my English isn't so bad 😭
 
  • #17
Baluncore
Science Advisor
2019 Award
7,242
2,341
Your English is good.

But Does it have meaning??
Maybe, but probably not.
Your perfect sinewave signal has an integer frequency and is very big, so it is easy to see in the spectrum. I have not seen the time domain data, or generator code.

If your raw data is the envelope of random noise pulses, and that is buried in random noise, then elimination of noise may not be helpful.

Cyclic variation in a noise envelope will probably not be a sinewave. For example, pulsar data has a very regular cyclic envelope, with a distinct pulse shape, maybe with a double pulse. You must first generate a power envelope. Then you must detect regular variation of the envelope. Information will be carried by the fundamental and by phase of harmonics due to envelope pulse shape.

You must specify the real signal before you can detect it. You will not know if the noise reduction is useful until you process real data. You have focused on the noise, but it is the spectral character of the signal that is critically important.

Please define the signal that must be detected.
1. What is the origin of your real signal?
2. How will it change or how is it modulated?
3. Are you also looking for a regular pattern in x-rays from pulsars?
 
  • #18
79
24
@Baluncore Thank you !
These are the answers.
1. I don't have the real signal.(I would like to research in a half year)
So, I just use the simple sine wave.

2. I just sums up Poisson noise and sine wave.

3. No, at least now.
 
  • #19
Baluncore
Science Advisor
2019 Award
7,242
2,341
Without a real signal there are so many possibilities that it is not sensible to guess what will be needed. The FT now gives you the amplitude and phase, or power of the known ideal 4 Hz sinewave.
 

Related Threads for: Suppress of the Poisson noise

  • Last Post
Replies
7
Views
236
  • Last Post
Replies
6
Views
3K
  • Last Post
Replies
7
Views
990
Replies
6
Views
1K
  • Last Post
Replies
19
Views
6K
Replies
3
Views
3K
Replies
42
Views
3K
  • Last Post
Replies
5
Views
2K
Top