Fourier smoothing and Savitzky-Golay filtering

  • Level: Undergrad 
  • Thread starter roam
  • Start date
  • Tags Tags
    Fourier
11 replies · 5K views · last reply
roam
Messages
1,265
Reaction score
12
I am trying to recover a laser beam profile using numerical differentiation of the data obtained from a "knife-edge scan". I am trying to select between two different methods to smooth out the numerical noise.

Here is my raw data and the derivative:

Line plot of normalized power versus displacement in cm: a raw data curve rising smoothly to 1, a noisy differentiated signal, and a smoothed signal tracking it, all peaking near 1 near 0.22 cm.


Here, I arbitrarily chose the 13 points, 4th order Savitzky-Golay method.

And here is the frequency spectrum of the derivative:

Stem plot of a discrete signal over indices -200 to about 190, with a single spike of value 1 at index 0 and small noisy values near zero elsewhere, with a dip to about -0.3 at indices near 0.


I have read that high frequencies caused purely by noise can be identified and removed in Fourier space, while the noise contamination of the low order components cannot be eliminated. So, in my spectrum how do we tell where the noise band starts?

Due to instrumental limitations, I am restricted to 2 decimal places when making measurements. If we had data with more significant figures, would that make the noise band more discernable?

In general, how does the Fourier smoothing compare to the Savitzky-Golay method?
 
Attachments
  • Line plot of normalized power versus displacement in cm: a raw data curve rising smoothly to 1, a noisy differentiated signal, and a smoothed signal tracking it, all peaking near 1 near 0.22 cm.
    4PcV49l.png
    29 KB · Views: 2,117
  • Stem plot of a discrete signal over indices -200 to about 190, with a single spike of value 1 at index 0 and small noisy values near zero elsewhere, with a dip to about -0.3 at indices near 0.
    sfrJ03T.png
    6.5 KB · Views: 1,263
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.
 
roam said:
In general, how does the Fourier smoothing compare to the Savitzky-Golay method?

There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!
 
Svein said:
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.

All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.

The laser beam I am working with is similar to the Fresnel pattern you would get from a rectangular source field. But it has aberrations due to imperfections in manufacturing and the optics that are involved. So we don't know the underlying function describing the beam.

When I look at the burn patterns of the laser, its general features agree with the results of the numerical differentiation. It's those smaller wiggles that I think are caused by noise. Some papers use a simple moving average on the derivative, which I think hides useful information by flatting out the curve. One paper uses Savitzky-Golay. I thought about using Fourier filtering, but how would you decide what coefficients to remove?

olivermsun said:
There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!

Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
 
roam said:
Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).
 
olivermsun said:
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).

But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
 
roam said:
But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
The frequency response of the S-G filter shows you exactly what effect it has on the spectrum. You can compare that to any Fourier filter you like.
 
roam said:
All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.
If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}}[/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}}[/itex]. The former does a better job numerically.
 
Hi Svein,

Svein said:
If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}}[/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}}[/itex]. The former does a better job numerically.

Yes, I have already used the central difference approximation.

Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
 
roam said:
Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.
 
  • Like
Likes   Reactions: roam
Svein said:
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.

Interesting. I tried that, this is what I got:

Two MATLAB-style plots side by side: left, Fourier spectrum with original and new spectrum vs frequency from -200 to 200; right, beam profile with smoothed signal and original differentiated signal vs x from 0 to 0.38.


Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
 
Attachments
  • Two MATLAB-style plots side by side: left, Fourier spectrum with original and new spectrum vs frequency from -200 to 200; right, beam profile with smoothed signal and original differentiated signal vs x from 0 to 0.38.
    iibDUJM.png
    18.3 KB · Views: 995
roam said:
Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
5. harmonic - I looked at your Fourier spectrum and saw that the amplitude for the 5. harmonic and above were very small compared to the first four. It might have been easier to guess if your Fourier amplitudes had been plotted logarithmically (in dB).

Oversmoothing - I would have tried 5., 7. etc. until the noise started being irritating. There is no hard and fast rule.

BTW: What did the smoothed beam profile tell you?
 
  • Like
Likes   Reactions: roam