1. Limited time only! Sign up for a free 30min personal tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Numerical differentiation of a dataset

  1. May 23, 2012 #1
    I have a dataset in two columns X and Y, sorted in ascending values of X.

    I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless.

    How do people do this?
  2. jcsd
  3. May 23, 2012 #2
    Have you tried smoothing out your data first? There are an incredible number of different ways to do so, you may want to try a quick literature search.
  4. May 23, 2012 #3
    You could try passing some sort of "best fit" function through the data and then simply differentiating that function.
  5. May 24, 2012 #4
    The dataset already seemed quite smooth upon an observation.
  6. May 25, 2012 #5
    Can you post it for us in some way? I think kj's "best fit" option would work if you can fit it reasonably well.
  7. May 25, 2012 #6

    Stephen Tashi

    User Avatar
    Science Advisor

    If you are willing to make the judgment that a rapdily varying derivative is a senseless result then you should be able to cite some theoretical model that explains why it shouldn't be. This would include a model for any noise. The problem is then how to incorporate this model into your calculations.

    If you think there is no noise in the data, then you could use the multi-point methods for estimating numerical derivatives. (For some reason, the Wikipeida only hints at such methods in the article on numerical differentiatiion and links to its Finite Difference Coefficient Article: http://en.wikipedia.org/wiki/Finite_difference_coefficients for more information. An interesting series of lectures covering numerical methods useful in physics is on the Perimeter Scholars website. I don't recall which of these lectures explains the multi-point method. http://www.perimeterscholars.org/274.html [Broken] The coding is done in FORTRAN.)
    Last edited by a moderator: May 6, 2017
  8. May 25, 2012 #7

    D H

    User Avatar
    Staff Emeritus
    Science Advisor

    That's a typical problem with numerical differentiation. There is no magic bullet even for numerical quadrature / numerical integration, and numerical quadrature is easy compared to numerical differentiation.

    Are those X values uniformly spaced, such as measurements taken once per hour over several days? If so, there are a number of techniques available that are far better (less noisy) than a simple forward or backward difference. Either a finite or infinite impulse response filter can be of aid. Another approach is to use wavelets.

    Fewer techniques are available for nonuniformly sampled data. FIR and IIR filtering techniques pretty much assumes uniformly sampled data. Some, but not all, wavelet transforms assume uniformly sampled data.

    Yet another approach is, as has been previously suggested, to fit the data to some model and analytically differentiate the resultant model.
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook