Fourier smoothing and Savitzky-Golay filtering

  • I
  • Thread starter roam
  • Start date
  • #1
1,266
11

Main Question or Discussion Point

I am trying to recover a laser beam profile using numerical differentiation of the data obtained from a "knife-edge scan". I am trying to select between two different methods to smooth out the numerical noise.

Here is my raw data and the derivative:

4PcV49l.png


Here, I arbitrarily chose the 13 points, 4th order Savitzky-Golay method.

And here is the frequency spectrum of the derivative:

sfrJ03T.png


I have read that high frequencies caused purely by noise can be identified and removed in Fourier space, while the noise contamination of the low order components cannot be eliminated. So, in my spectrum how do we tell where the noise band starts?

Due to instrumental limitations, I am restricted to 2 decimal places when making measurements. If we had data with more significant figures, would that make the noise band more discernable?

In general, how does the Fourier smoothing compare to the Savitzky-Golay method?
 

Attachments

Answers and Replies

  • #2
Svein
Science Advisor
Insights Author
2,025
649
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.
 
  • #3
olivermsun
Science Advisor
1,244
118
In general, how does the Fourier smoothing compare to the Savitzky-Golay method?
There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!
 
  • #4
1,266
11
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.
All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.

The laser beam I am working with is similar to the Fresnel pattern you would get from a rectangular source field. But it has aberrations due to imperfections in manufacturing and the optics that are involved. So we don't know the underlying function describing the beam.

When I look at the burn patterns of the laser, its general features agree with the results of the numerical differentiation. It's those smaller wiggles that I think are caused by noise. Some papers use a simple moving average on the derivative, which I think hides useful information by flatting out the curve. One paper uses Savitzky-Golay. I thought about using Fourier filtering, but how would you decide what coefficients to remove?

There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!
Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
 
  • #5
olivermsun
Science Advisor
1,244
118
Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).
 
  • #6
1,266
11
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).
But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
 
  • #7
olivermsun
Science Advisor
1,244
118
But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
The frequency response of the S-G filter shows you exactly what effect it has on the spectrum. You can compare that to any Fourier filter you like.
 
  • #8
Svein
Science Advisor
Insights Author
2,025
649
All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.
If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}} [/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}} [/itex]. The former does a better job numerically.
 
  • #9
1,266
11
Hi Svein,

If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}} [/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}} [/itex]. The former does a better job numerically.
Yes, I have already used the central difference approximation.

Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
 
  • #10
Svein
Science Advisor
Insights Author
2,025
649
Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.
 
  • #11
1,266
11
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.
Interesting. I tried that, this is what I got:

iibDUJM.png


Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
 

Attachments

  • #12
Svein
Science Advisor
Insights Author
2,025
649
Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
5. harmonic - I looked at your Fourier spectrum and saw that the amplitude for the 5. harmonic and above were very small compared to the first four. It might have been easier to guess if your Fourier amplitudes had been plotted logarithmically (in dB).

Oversmoothing - I would have tried 5., 7. etc. until the noise started being irritating. There is no hard and fast rule.

BTW: What did the smoothed beam profile tell you?
 

Related Threads for: Fourier smoothing and Savitzky-Golay filtering

  • Last Post
Replies
6
Views
2K
Replies
4
Views
12K
  • Last Post
Replies
7
Views
3K
Replies
1
Views
1K
Replies
15
Views
4K
  • Last Post
Replies
4
Views
3K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
3
Views
5K
Top