
#1
May2312, 10:13 PM

P: 44

I have a dataset in two columns X and Y, sorted in ascending values of X.
I'm trying to find its numerical derivative, however, the "noise" (it's very hard to see any noise in the dataset itself when plotted), but the noise gets massively amplified to the point where the numerical derivative looks utterly senseless. How do people do this? 



#2
May2312, 10:49 PM

P: 101

Have you tried smoothing out your data first? There are an incredible number of different ways to do so, you may want to try a quick literature search.




#3
May2312, 11:12 PM

P: 158

You could try passing some sort of "best fit" function through the data and then simply differentiating that function.




#4
May2412, 03:02 AM

P: 44

Numerical differentiation of a dataset
The dataset already seemed quite smooth upon an observation.




#5
May2512, 01:20 AM

P: 12

Can you post it for us in some way? I think kj's "best fit" option would work if you can fit it reasonably well.




#6
May2512, 09:38 AM

Sci Advisor
P: 3,177

If you think there is no noise in the data, then you could use the multipoint methods for estimating numerical derivatives. (For some reason, the Wikipeida only hints at such methods in the article on numerical differentiatiion and links to its Finite Difference Coefficient Article: http://en.wikipedia.org/wiki/Finite_...e_coefficients for more information. An interesting series of lectures covering numerical methods useful in physics is on the Perimeter Scholars website. I don't recall which of these lectures explains the multipoint method. http://www.perimeterscholars.org/274.html The coding is done in FORTRAN.) 



#7
May2512, 10:33 AM

Mentor
P: 14,483

Are those X values uniformly spaced, such as measurements taken once per hour over several days? If so, there are a number of techniques available that are far better (less noisy) than a simple forward or backward difference. Either a finite or infinite impulse response filter can be of aid. Another approach is to use wavelets. Fewer techniques are available for nonuniformly sampled data. FIR and IIR filtering techniques pretty much assumes uniformly sampled data. Some, but not all, wavelet transforms assume uniformly sampled data. Yet another approach is, as has been previously suggested, to fit the data to some model and analytically differentiate the resultant model. 


Register to reply 
Related Discussions  
How to Normalize a simulated dataset to fit the actual dataset?  Linear & Abstract Algebra  1  
Implicit numerical differentiation  Calculus & Beyond Homework  1  
Numerical Integration of a dataset (what is the best method?)  General Engineering  2  
Numerical Differentiation: Difference approximation on numerical data  Calculus & Beyond Homework  2  
Numerical Differentiation and Integration  Physics Learning Materials  0 