Numerical differentiation of a dataset

[exmachina]

I have a dataset in two columns X and Y, sorted in ascending values of X.

I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless.

How do people do this?

 

[theorem4.5.9]

Have you tried smoothing out your data first? There are an incredible number of different ways to do so, you may want to try a quick literature search.

 

[kjohnson]

You could try passing some sort of "best fit" function through the data and then simply differentiating that function.

 

[exmachina]

The dataset already seemed quite smooth upon an observation.

 

[Cadaei]

Can you post it for us in some way? I think kj's "best fit" option would work if you can fit it reasonably well.

 

[Stephen Tashi]

the numerical derivative looks utterly senseless.

How do people do this?

If you are willing to make the judgment that a rapdily varying derivative is a senseless result then you should be able to cite some theoretical model that explains why it shouldn't be. This would include a model for any noise. The problem is then how to incorporate this model into your calculations.

If you think there is no noise in the data, then you could use the multi-point methods for estimating numerical derivatives. (For some reason, the Wikipeida only hints at such methods in the article on numerical differentiatiion and links to its Finite Difference Coefficient Article: http://en.wikipedia.org/wiki/Finite_difference_coefficients for more information. An interesting series of lectures covering numerical methods useful in physics is on the Perimeter Scholars website. I don't recall which of these lectures explains the multi-point method. http://www.perimeterscholars.org/274.html The coding is done in FORTRAN.)

 

[D H]

I have a dataset in two columns X and Y, sorted in ascending values of X.

I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless.

How do people do this?

That's a typical problem with numerical differentiation. There is no magic bullet even for numerical quadrature / numerical integration, and numerical quadrature is easy compared to numerical differentiation.

Are those X values uniformly spaced, such as measurements taken once per hour over several days? If so, there are a number of techniques available that are far better (less noisy) than a simple forward or backward difference. Either a finite or infinite impulse response filter can be of aid. Another approach is to use wavelets.

Fewer techniques are available for nonuniformly sampled data. FIR and IIR filtering techniques pretty much assumes uniformly sampled data. Some, but not all, wavelet transforms assume uniformly sampled data.

Yet another approach is, as has been previously suggested, to fit the data to some model and analytically differentiate the resultant model.