Register to reply

Numerical differentiation of a dataset

by exmachina
Tags: dataset, differentiation, numerical
Share this thread:
exmachina
#1
May23-12, 10:13 PM
P: 44
I have a dataset in two columns X and Y, sorted in ascending values of X.

I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless.

How do people do this?
Phys.Org News Partner Mathematics news on Phys.org
Professor quantifies how 'one thing leads to another'
Team announces construction of a formal computer-verified proof of the Kepler conjecture
Iranian is first woman to win 'Nobel Prize of maths' (Update)
theorem4.5.9
#2
May23-12, 10:49 PM
P: 101
Have you tried smoothing out your data first? There are an incredible number of different ways to do so, you may want to try a quick literature search.
kjohnson
#3
May23-12, 11:12 PM
kjohnson's Avatar
P: 158
You could try passing some sort of "best fit" function through the data and then simply differentiating that function.

exmachina
#4
May24-12, 03:02 AM
P: 44
Numerical differentiation of a dataset

The dataset already seemed quite smooth upon an observation.
Cadaei
#5
May25-12, 01:20 AM
P: 12
Can you post it for us in some way? I think kj's "best fit" option would work if you can fit it reasonably well.
Stephen Tashi
#6
May25-12, 09:38 AM
Sci Advisor
P: 3,285
Quote Quote by exmachina View Post
the numerical derivative looks utterly senseless.

How do people do this?
If you are willing to make the judgment that a rapdily varying derivative is a senseless result then you should be able to cite some theoretical model that explains why it shouldn't be. This would include a model for any noise. The problem is then how to incorporate this model into your calculations.

If you think there is no noise in the data, then you could use the multi-point methods for estimating numerical derivatives. (For some reason, the Wikipeida only hints at such methods in the article on numerical differentiatiion and links to its Finite Difference Coefficient Article: http://en.wikipedia.org/wiki/Finite_...e_coefficients for more information. An interesting series of lectures covering numerical methods useful in physics is on the Perimeter Scholars website. I don't recall which of these lectures explains the multi-point method. http://www.perimeterscholars.org/274.html The coding is done in FORTRAN.)
D H
#7
May25-12, 10:33 AM
Mentor
P: 15,153
Quote Quote by exmachina View Post
I have a dataset in two columns X and Y, sorted in ascending values of X.

I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless.

How do people do this?
That's a typical problem with numerical differentiation. There is no magic bullet even for numerical quadrature / numerical integration, and numerical quadrature is easy compared to numerical differentiation.

Are those X values uniformly spaced, such as measurements taken once per hour over several days? If so, there are a number of techniques available that are far better (less noisy) than a simple forward or backward difference. Either a finite or infinite impulse response filter can be of aid. Another approach is to use wavelets.

Fewer techniques are available for nonuniformly sampled data. FIR and IIR filtering techniques pretty much assumes uniformly sampled data. Some, but not all, wavelet transforms assume uniformly sampled data.

Yet another approach is, as has been previously suggested, to fit the data to some model and analytically differentiate the resultant model.


Register to reply

Related Discussions
How to Normalize a simulated dataset to fit the actual dataset? Linear & Abstract Algebra 1
Implicit numerical differentiation Calculus & Beyond Homework 1
Numerical Integration of a dataset (what is the best method?) General Engineering 2
Numerical Differentiation: Difference approximation on numerical data Calculus & Beyond Homework 2