Must On-Line Smooth Forecasted Point relative to previous points.

[Cardinal Gramm]

This is a prediction that is made every day.

If I do a back test assemble a curve composed of each days' prediction, I get fair results.

However, if I smooth this backtest curve, I get fantastic results.

So what I need to do is take today's prediction and the prediction time history leading up to today, and smooth today's prediction.

The problem is that this is EndPoint Smoothing which is special case which most smoothing algorithms do not address.

What would be perfect, would be to identify the harmonics of the finite, discrete time, aperiodic curve and simply let them establish the end point. DFT's end point constraint is unacceptable. DFT with a mirrored input... maybe...

Splines are just visual aids which do not really take into account signal content.

Butterworth is NG since need end points and padding is not representative of the signal content.

Kalman theoretically may work except that there is nothing really known about the system - it would be like using a Kalman to remove hiss from an audio signal (music, etc.) on-line without ANY delay.

I was thinking that something like Principal Component Analysis might somehow be used on a single input - of course it is intended to reduce the number of inputs(!)

My Inner Mad Scientist considered this Fourier mal-use:

1. Take the 1st Fourier harmonic.
2. Establish its phase by choosing the maximum correlation achieved by surveying lag.
3. Compile a "response" curve for this maximal correlation for each of the Fourier terms.
4. Pick a set of dominant harmonics below some smoothing frequency cutoff.
5. Sum the harmonics with weights equal to the corrrelations.
6. Confirm a nice fit over the curve interior.
7. Use the end point value as the smoothed value.

I don't have the math at hand to demonstrate that my set of harmonics with "adjusted" phase would be orthogonal (can be linearly super positioned) or that the actual correlation coef would be the correct weighting factor - it may need processed into its square or normalized by standard deviation or some other "metric."
________________________

Oh, regarding Fourier Transforms, Discrete Time and Continuous, while aperiodic, they always use a signal that starts and stops at zero in examples. Same for Convolution. So I'm not sure what can be done with this.

It seems like the examples are finite length signals as opposed to continuous (infinite) signals for which you select some arbitrary interval to perform the FT.
________________________

I am not an electrical engineer but I suspect what I am trying to do may be extremely advanced because the only thing that I have encountered that seems like it would work is the Kalman and you better remember all your Controls Engineering to set that one up if you are lucky enough to have a model. The concept that a Kalman uses "all available information" is quite impressive and seriously overwhealming.

Thanks in advance,
Tom

 

[Stephen Tashi]

I suggest you make another attempt to state the question.

However, if I smooth this backtest curve, I get fantastic results.

What is the "backtest curve" and what "results" are you talking about?

So what I need to do is take today's prediction and the prediction time history leading up to today, and smooth today's prediction.

You say you have smoothed one type of curve (the "backtest curve") and yet you ask about how to smooth a different type of curve and dismiss various techniques for smoothing it. It is unclear why whatever technique you are using on the "backtest curve" isn't applicable to the second type of curve.