Smoothing Numerical Differentiation Noise

  • #26
FactChecker
Science Advisor
Gold Member
5,991
2,306
I think it is better to obtain more precise data first. I am using high energy lasers so using current modulation I should be able to lower the power and use a very sensitive meter.
Good. I agree that getting some data with greater resolution is essential. If you try to use higher order models on data where the largest part is due to round-off, you will get a mess. That being said, the suggestion of using data i-1 and i+1 to estimate i would double the true change of the function value and make the round-off half as significant. (It's also a better way to estimate the slope at point i.) Likewise, using i-2 and i+2 would improve things by a larger factor. But if you carry that to the extreme, you will miss local changes in the function.
 
  • Like
Likes roam
  • #27
832
30
@Telemachus Sure thing. Since this thread is under a mathematics forum, I will message you the details of the physics of the setup.
Hi again. I already gave a read to your pm. I think the physics is actually important, because it could tell you what function you should be expecting, and what type of function to use for your data. For example, if the data were instead of this intensity profile, just some records for position and time in a free fall at the earth surface, you would know that it should be a parabola from the physics, and interpolate a parabola, no matter how coarse your data is.

I think that knowing more about the physics in this situation is actually something very important to do the data analysis. Do you have any clue of what type of function you should be expecting?
 
  • Like
Likes roam
  • #28
Mark Harder
Gold Member
241
59
Hi Svein and Mark Harder, thank you so much for the thorough explanations.

Yes, indeed I had used the forward difference approximation as the simplest approximation for the 1st derivative. In fact, I had generated vectors of differences using diff(.) in Matlab, so that diff(f)./diff(x) would be equivalent to FDA. I will instead try the central difference approximation and see how it goes.

Also: if I obtain high precision data and want to use symbolic computation, would differentiating a quadratic polynomial work? It clearly didn't work using the current data (my post #11)...

@Mark Harder I could send you my data if it helps, but as others suggested I think it is better to obtain more precise data first. I am using high energy lasers so using current modulation I should be able to lower the power and use a very sensitive meter.

@Telemachus Sure thing. Since this thread is under a mathematics forum, I will message you the details of the physics of the setup.
Yes, I agree. Having more data and/or better quality data is the best solution if you can do it. You can also increase the also increase the quality of the data by increasing the sampling time for each measurement. The signal/noise ration increases as the square root of the counts (or continuous time). So you'd need to collect four times as long at each point to double the S/N ration. After a while, the time needed becomes impractical; but if you can, spending more time collecting at each point will 'smooth' the data without any risking the distortion of your original data or introducing biases into the experiment.
 
  • Like
Likes roam and FactChecker
  • #29
Mark Harder
Gold Member
241
59
Hi Svein and Mark Harder, thank you so much for the thorough explanations.

Yes, indeed I had used the forward difference approximation as the simplest approximation for the 1st derivative. In fact, I had generated vectors of differences using diff(.) in Matlab, so that diff(f)./diff(x) would be equivalent to FDA. I will instead try the central difference approximation and see how it goes.

Also: if I obtain high precision data and want to use symbolic computation, would differentiating a quadratic polynomial work? It clearly didn't work using the current data (my post #11)...

@Mark Harder I could send you my data if it helps, but as others suggested I think it is better to obtain more precise data first. I am using high energy lasers so using current modulation I should be able to lower the power and use a very sensitive meter.

@Telemachus Sure thing. Since this thread is under a mathematics forum, I will message you the details of the physics of the setup.
Roam, I forgot to mention that I was assuming in my analysis that the x-intervals in your measurements were all equal. If the intervals between points vary across the data, then the problem is more like an interpolation.
 
  • Like
Likes roam

Related Threads on Smoothing Numerical Differentiation Noise

Replies
6
Views
3K
  • Last Post
Replies
4
Views
3K
  • Last Post
Replies
1
Views
2K
  • Last Post
Replies
1
Views
1K
Replies
4
Views
13K
  • Last Post
Replies
2
Views
1K
  • Last Post
Replies
5
Views
674
  • Last Post
Replies
7
Views
3K
  • Last Post
Replies
1
Views
2K
Top