Fourier smoothing and Savitzky-Golay filtering

  • I
  • Thread starter roam
  • Start date
  • Tags
    Fourier
In summary, the conversation discusses the process of recovering a laser beam profile using numerical differentiation and selecting between two methods to smooth out numerical noise. The methods discussed include a linear fit, Savitzky-Golay filtering, and Fourier filtering. The conversation also mentions that numerical differentiation is equivalent to a "high-pass" filter and suggests using curve fitting and symbolic differentiation for more accurate results. The paper "What Is a Savitzky-Golay Filter" by R.W. Schafer is referenced and its comparison of Fourier and S-G filtering is mentioned. The conversation also discusses the possibility of distinguishing noise from the actual signal by looking at the Fourier spectrum and mentions the idea of oversmoothing.
  • #1
roam
1,271
12
I am trying to recover a laser beam profile using numerical differentiation of the data obtained from a "knife-edge scan". I am trying to select between two different methods to smooth out the numerical noise.

Here is my raw data and the derivative:

4PcV49l.png


Here, I arbitrarily chose the 13 points, 4th order Savitzky-Golay method.

And here is the frequency spectrum of the derivative:

sfrJ03T.png


I have read that high frequencies caused purely by noise can be identified and removed in Fourier space, while the noise contamination of the low order components cannot be eliminated. So, in my spectrum how do we tell where the noise band starts?

Due to instrumental limitations, I am restricted to 2 decimal places when making measurements. If we had data with more significant figures, would that make the noise band more discernable?

In general, how does the Fourier smoothing compare to the Savitzky-Golay method?
 

Attachments

  • 4PcV49l.png
    4PcV49l.png
    29 KB · Views: 1,884
  • sfrJ03T.png
    sfrJ03T.png
    6.5 KB · Views: 1,085
Mathematics news on Phys.org
  • #2
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.
 
  • #3
roam said:
In general, how does the Fourier smoothing compare to the Savitzky-Golay method?

There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!
 
  • #4
Svein said:
Differentiation of a set of measurements is equivalent to a "high-pass" filter, you remove the basic information and are left with the noise. If you really want the derivative of a measurement data set, I suggest you do a curve fitting of some sort and and do a symbolic differentiation of the fitted curve.

All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.

The laser beam I am working with is similar to the Fresnel pattern you would get from a rectangular source field. But it has aberrations due to imperfections in manufacturing and the optics that are involved. So we don't know the underlying function describing the beam.

When I look at the burn patterns of the laser, its general features agree with the results of the numerical differentiation. It's those smaller wiggles that I think are caused by noise. Some papers use a simple moving average on the derivative, which I think hides useful information by flatting out the curve. One paper uses Savitzky-Golay. I thought about using Fourier filtering, but how would you decide what coefficients to remove?

olivermsun said:
There’s a nice “Lecture Notes” article by R.W. Schafer, “What Is a Savitzky-Golay Filter,” in IEEE Signal Processing Magazine, July 2011, that discusses this very topic!

Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
 
  • #5
roam said:
Could you please explain? I had a cursory look at the paper and it doesn't really compare Fourier filtering with S-G filtering.
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).
 
  • #6
olivermsun said:
The paper discusses the frequency (Fourier) domain interpretation of the S-G filter (see Fig. 5).

But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
 
  • #7
roam said:
But they are comparing to Parks-McClellan algorithm, not the method that I am asking about (which is simply to exclude some of the coefficients from the Fourier spectrum).
The frequency response of the S-G filter shows you exactly what effect it has on the spectrum. You can compare that to any Fourier filter you like.
 
  • #8
roam said:
All academic papers that I have seen use numerical differentiation. I tried what you suggested in another thread dealing with a similar problem. It doesn't work, because a linear fit shows a very good r2 value — yet you can't obtain the laser beam profile by differentiating it. The derivative of a quadratic and a cubic also show no resemblance to the actual laser beam.
If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}} [/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}} [/itex]. The former does a better job numerically.
 
  • #9
Hi Svein,

Svein said:
If you really have to do a numerical differentiation, try using [itex]y_{n}'=\frac{y_{n+1}-y_{n-1}}{x_{n+1}-x_{n-1}} [/itex] instead of [itex]y_{n}'=\frac{y_{n+1}-y_{n}}{x_{n+1}-x_{n}} [/itex]. The former does a better job numerically.

Yes, I have already used the central difference approximation.

Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
 
  • #10
roam said:
Do you think it is possible to distinguish noise from the actual deterministic part of the signal by looking at the Fourier spectrum? :confused: The spectrum of the derivative (calculated using CDA) is shown in my first post...
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.
 
  • Like
Likes roam
  • #11
Svein said:
I have done this and it works - sort of. Try throwing away everything above the 5. harmonic and then transfer back.

Interesting. I tried that, this is what I got:

iibDUJM.png


Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
 

Attachments

  • iibDUJM.png
    iibDUJM.png
    18.3 KB · Views: 822
  • #12
roam said:
Is there a reason you chose the 5th harmonic? Is there any way to tell if we are oversmoothing?
5. harmonic - I looked at your Fourier spectrum and saw that the amplitude for the 5. harmonic and above were very small compared to the first four. It might have been easier to guess if your Fourier amplitudes had been plotted logarithmically (in dB).

Oversmoothing - I would have tried 5., 7. etc. until the noise started being irritating. There is no hard and fast rule.

BTW: What did the smoothed beam profile tell you?
 
  • Like
Likes roam

1. What is Fourier smoothing and Savitzky-Golay filtering?

Fourier smoothing and Savitzky-Golay filtering are two commonly used signal processing techniques used to remove noise and enhance signal features in a dataset. They can be used in a variety of fields such as image processing, chemistry, and physics.

2. How does Fourier smoothing work?

Fourier smoothing works by taking a dataset and decomposing it into a series of sine and cosine waves using a technique called Fourier transform. The high-frequency components, which represent noise in the dataset, are then removed, resulting in a smoother signal.

3. What is the difference between Fourier smoothing and Savitzky-Golay filtering?

The main difference between Fourier smoothing and Savitzky-Golay filtering is the way they handle the noise in the dataset. While Fourier smoothing removes high-frequency components, Savitzky-Golay filtering uses a sliding window to fit a polynomial curve to the data, which effectively removes noise while preserving the overall shape of the signal.

4. When should I use Fourier smoothing versus Savitzky-Golay filtering?

The choice between Fourier smoothing and Savitzky-Golay filtering depends on the type of noise present in the dataset and the desired outcome. Fourier smoothing is more effective for removing high-frequency noise, while Savitzky-Golay filtering is better at smoothing out low-frequency noise. It is recommended to try both techniques and compare the results to determine which one is more suitable for a given dataset.

5. Are there any limitations to Fourier smoothing and Savitzky-Golay filtering?

While Fourier smoothing and Savitzky-Golay filtering are powerful tools for noise removal and signal enhancement, they do have some limitations. They may not work well on datasets with a highly irregular or non-periodic signal. Additionally, improper parameter selection can lead to oversmoothing or distortion of the signal, so it is important to carefully choose the parameters for each specific dataset.

Similar threads

  • General Math
Replies
28
Views
4K
Replies
2
Views
4K
  • Programming and Computer Science
Replies
2
Views
2K
  • Other Physics Topics
Replies
1
Views
1K
  • Electrical Engineering
Replies
11
Views
2K
Replies
7
Views
3K
  • Electrical Engineering
Replies
7
Views
4K
  • STEM Academic Advising
Replies
13
Views
2K
Replies
2
Views
2K
  • Programming and Computer Science
Replies
11
Views
6K
Back
Top