Smoothing algorithm for times series and slope measurement

[nickmath]

Hi,

I'm doing an experiment that gathers one data point approximately each second. I have plotted all the datapoints on a graph. The graph has many tops and bottoms. Now due to the fact that there are so many data points and that I can visualle tell that there are several overal trends: one period my data is going up, the next period it is level again for a period of time and the next period it can be down again. And in each of these periods the data can go up, down or level again, although the overal direction is up, down or level. I have searched for different smoothing methods that can follow all these small movements, but at the same time (using a second smoothing algorithm/parameter) can follow the overal movement. I was thinking about using splines or lowess. Are splines or lowess able to do this?

Second problem I'm having is, after having applied a smoothing algorithm, I need to be able to measure the slope of the smoothing line. I need to know what the slope/curvature of that smoothing line is at any giving moment in time. All the pointers you can give me are really welcome. Thanks.

Kind regards,

Nick

 

[xnull]

Foremost: I am an undergraduate CS student and therefore my advise should be taken with a skeptical ear.

I think it depends on what you want to use the smoothed values for.
If you want to do some sort of predictions or make a general model of the data I think what you want to do is model the data with Minimum Description Length using the order of a polynomial and a polynomial that fits the data. Doing a spline will probably overfit your data if this is what you want to smooth for.

If you want to make a nicer visualization of the data, spline might do the trick for you. I'm not really sure though - I'm hesitant to agree that it would do a very good job.

 

[mXSCNT]

Are you sure that all of these little ups and downs are meaningful data? I guess LOESS would be as good as any method for connecting the dots. If you are interested in long-term trends, you could just fit another LOESS model, with a larger smoothing parameter.