# I Smoothing Numerical Differentiation Noise

Tags:
1. Jun 13, 2018

### roam

I am using the "knife-edge" technique to find the intensity profile of a rectangular laser beam. The data that is obtained using this method is power, the integral of intensity. Therefore, to get the intensity profile we must differentiate the data.

So, as expected, my data looks like a ramp (integral of a rectangular function). But when I performed the numerical differentiation on the data the result was too noisy:

This doesn't really resemble the actual beam that we have. This is more like what is expected from a rectangular/tophat beam:

So, what kind of smoothing algorithm can I use on the differentiated data? How do we decide what smoothing would be the most appropriate and accurate in this situation?

Any help is greatly appreciated.

P. S. I could try to obtain more data points, but I am not sure if that would help. I've used this technique before on Gaussian beams (this time, the raw data was an error function erf) – I had far fewer data points, yet I didn't get this much noise. Why?

2. Jun 13, 2018

### BvU

Your derivative appears to be very discretized: it can assume five different values. The second picture shows a granularity of more than a hundred steps. In the third picture there are only seven different values for the derivaative. Not much either.

Can you increase the intensity, improve the scale on the sensor, or something like that ?

3. Jun 13, 2018

### Staff: Mentor

A smaller number of data points means a larger difference between them - noise plays a smaller role. That is something you can do with your data, e.g. combine three adjacent bins to a single bin. Alternatively use one of the many smoothing methods. Just taking a weighted average of bins around your bin should give a nice approximation already.

4. Jun 13, 2018

### Staff: Mentor

What you do is fit the original data with something much lower order than the number of data points. Looking at your data, I would recommend a cubic spline with say 5-10 nodes. You then differentiate the low order function analytically.

5. Jun 13, 2018

### roam

So, if I understand correctly, we replace each element with the average of the three adjacent elements. i.e.,

$$\left(y\left(1\right)+y\left(2\right)+y\left(3\right)\right)/3\to y\left(1\right)$$

So we will have ~3x fewer data points to plot. Is that correct?

What if we use Matlab's smooth(.) function with a span 3 moving average filter? This would also be a 3-point smoothing algorithm except we have the same number of data points as we started. Is this method more or less accurate than combining the points?

Here is what I got (smoothed = magenta, original derivative = brown):

I need to reconstruct the actual beam profile as accurately as possible.

I can try finding a better power meter. Maybe a digital would be more helpful (this is from an analog output and it is hard to read off minor changes in power).

If I was to use some kind of adjacent average smoothing, would it be helpful to collect more data points?

6. Jun 13, 2018

### Staff: Mentor

More readings for the same curve will probably help.
A slightly more aggressive smoothing should help as well.

7. Jun 14, 2018

### roam

By more aggressive do you mean taking a larger span? I mean, instead of averaging 3 points, you could average a larger number of points.

How would you decide what is a good place to stop smoothing? I've noted that beyond a certain point (in my case, $\text{span}=33$), you will not see any additional changes to the plot.

8. Jun 14, 2018

### Staff: Mentor

For example, yes. You can also average over points with different weights for them (larger weights for points nearby).

What is best depends on your application.

9. Jun 14, 2018

### Svein

I would start with a linear regression on the data, then the derivative is just the slope of the line. Check out the r2 value of the regression. Then (looking at the curve) do a third degree regression. Now the derivative is analytically calculable. Again, check the r2 value of this regression. If it is better (higher), stay with the third degree, otherwise use the linear fit.

10. Jun 14, 2018

### FactChecker

As @BvU pointed out, your data is very crude. I think that you need to address that before post-processing. Smoothing may just be "putting lipstick on a pig." It may mask the information that you are looking for.

11. Jun 16, 2018

### roam

So, the idea is to use symbolic differentiation to avoid the noise problem in the numerical computation?

As you suggested, I did fit the data using polynomials of different degree, and here are the first few R2 values:

$$\begin{array}{c|ccccc} \hline \text{degree} & 1 & 2 & 3 & 4 & 5\\ \hline R^{2} & 0.9885 & 0.9941 & 0.9945 & 0.9945 & 0.9969 \\\hline \end{array}$$

I went up to degree 9 and R2 kept increasing. But that might be because we are just modeling the noise in the data at that point.

For the quadratic for instance, you have an equation of the form $ax^2 + bx + c$ which has the analytic derivative $2ax+b$. But when I plot it, I get this which doesn't look anything like the beam profile:

What is wrong here?

That is true. But in what way would you say my data is crude?

My data looks like ramp, and this is what I would expect if the beam has a nearly rectangular intensity profile (my signal is its spatial integration). I am not sure in what way I could improve the data except collecting more points...

12. Jun 16, 2018

### BvU

Basically nothing. You fit a parabola, you get a parabola. What you want to fit is ideally an almost square block with a lot of detail (*). Try to see what kind of integrated function that would yield.

(*) Detail that is not present in your data: steep edges, with possibly some small deviations on the flat part.
In short: you need higher resolution in both directions: less coarsely rounded off data points an a lot more of them

13. Jun 16, 2018

### FactChecker

Sorry, I didn't realize that the derivative was not the raw data. Your calculated derivative numbers have very little resolution -- only 5 descrete values. That is very crude data to work with. In general, taking a derivative, whether symbolically or calculating, will introduce significant noise. Trying to smooth the noise out later is just undoing the derivative, perhaps in a bad way.

14. Jun 16, 2018

### Svein

The linear regression shows a very high value for r2, so use that for a basic approximation.

Now I (being curious) would subtract the linear regression values from your data and do an FFT on the differences. The lower frequencies of the transformed data might tell you something (try throwing away everything but the three lowest frequencies and transform that back).

15. Jun 17, 2018

### roam

Hi Svein,

So what is the idea behind subtracting the regression from the data? And are we discarding the high frequencies as being noise?

I did what you suggested. In the DFT, I only kept the 3 lowest terms next to the DC term. Here are the results:

How can we use this information to reconstruct the beam profile?

16. Jun 17, 2018

### Svein

Well, you know much more about the experiment than I do. What I was looking for in the DFT was some regularity in the deviations from the straight line, but there does not seem to be any. My conclusion would be that the linear regression is a very good fit to your experimental data (an r2 of 0.9885 - there are sciences where an r2 of 0.1 is considered exceptionally good) and the deviations are due to noise/measurement accuracy.

17. Jun 17, 2018

### FactChecker

I agree. I think that the deltas for the derivatives being such a small set of fixed values shows that the measurement accuracy is a limiting factor and will not allow better results.

18. Jun 18, 2018

### roam

During my measurements, the increase in data points appeared to occur at fixed increments (that's how I recorded the data). That was the best I could do with my measuring instrument – it wasn't possible to record the values precisely (i.e. using longer decimal format). So, is this what causes the discreteness of the derivative values?

Any explanation would be appreciated.

What would the regularities tell us though? As shown in the second figure in my first post, the fluctuations in a tophat beam aren't regular usually...

I guess the only option would be to obtain more precise measurements (more decimal places). I don't see a benefit to using a linear regression in this problem because the analytic derivative would just be a straight line that looks nothing like a beam profile. The analytic differentiation seems to be less useful here than its numerical counterpart.

19. Jun 18, 2018

### FactChecker

It certainly appears that way.
To keep things simple, suppose one is measuring values between -1.5 and +1.5 but the recorded values are always rounded to the nearest integer -1, 0, +1.
There are only the following 9 cases of (rounded value of $y_{i+1}$, rounded value of $y_i$): $$(-1,+1), (-1,0), (-1,-1), (0,+1), (0,0), (0,-1), (+1,+1), (+1,0), (+1,-1).$$ They give the 5 cases for $y_{i+1} - y_i$: -2, -1, 0, +1, +2.
It appears that these 5 cases correspond to the 5 values that you are getting (except that your Y values trend upward so the deltas are always nonnegative). So it looks like the recorded values are always rounded to values that do not allow very much resolution for the derivative. Any detailed conclusions you reach by analysing the derivative may be saying more about your rounding process than about the derivative itself.

Last edited: Jun 18, 2018
20. Jun 18, 2018

### Svein

Trying to calculate derivatives on measured data is the equivalent of a high-pass filter: Only the noise gets through.

If you insist on trying to calculate derivatives directly from the data, I'll give you a tip: Instead of doing $f_{n}'=\frac{f_{n+1}-f_{n}}{x_{n+1}-x_{n}}$ (which calculates the secant, not the tangent), try using $f_{n}'=\frac{f_{n+1}-f_{n-1}}{x_{n+1}-x_{n-1}}$ which gives a much better approximation to the tangent.