First Post: How to Smooth End point of Finite Data Series time series

[Cardinal Gramm]

I wish it wasn't out of desperation that I'm making this first post!

I have a neural network that is making predictions, the next 5 time points per training.

Back testing consists of appending these 5 point sets together to produce a data set that spans time over a much longer period.

The problem is that the results are pretty good except that there is too much "noise" present.

If the data set was periodic, I'd use DFT (discrete Fourier transform), toss the higher coefficients (now have LP filter) and move on.

However, its not periodic.

My thinking is I have 2 options:

1. DFT tricks
2. Other component methods: PCA, ICA, ?

I really don't want to get into polynomials or beam-fit splines. I'd rather stick with actual components of the data set.

Any suggestions would be greatly appreciated. So far I have burned over a man-week trying to make something work.

Thanks in advance,
Tom

 

[milesyoung]

I have very limited experience with neural networks so I might not be qualified to answer this, but could you clarify some things for me though:

Are you using a window of historic data to train the network, then using this model to predict ahead of the window, then passing this time series through a discrete-time low-pass filter?

If so, what's wrong with the output you get from the low-pass filter and what would you consider "too much noise"?

 

[Cardinal Gramm]

Thanks for the reply!

Yes, it s a window that moves along and makes the Next Prediction.

As far as the smoothing problem, just think of any non-periodic signal with noise - they all have this problem.

First, I have no access to advanced suites of math functions, I have to write my own - so I have no digital LP filter.

However, I'm guessing this filter would have lag like all filters do. Unfortunately, lag kills the prediction since the last thing you want to do is lag your prediction.

I don't know how digital filters are different from analog, but I'm not constrained to have to do this in real time. Real time filtering pretty much has to lag.

If my signal was periodic, I'd just do the Discrete Fourier Transform and I'd be done with it.
____________________________

I was just thinking that when you do a modal analysis on a vibrating cantilever, you get the frequencies and the mode shapes. You can construct any harmonic motion by summing the appropriate mode shapes AND you can exactly demonstrate the tip displacement.

If you look at the mode shapes, they are anything but periodic. Yet, they comprise an orthogonal set and can be summed.

Need something like that I'm hoping.

Thanks for bringing up digital filters - need to know more about them.

Tom