Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Must On-Line Smooth Forecasted Point relative to previous points.

  1. Apr 8, 2013 #1
    This is a prediction that is made every day.

    If I do a back test assemble a curve composed of each days' prediction, I get fair results.

    However, if I smooth this backtest curve, I get fantastic results.

    So what I need to do is take today's prediction and the prediction time history leading up to today, and smooth today's prediction.

    The problem is that this is EndPoint Smoothing which is special case which most smoothing algorithms do not address.

    What would be perfect, would be to identify the harmonics of the finite, discrete time, aperiodic curve and simply let them establish the end point. DFT's end point constraint is unacceptable. DFT with a mirrored input... maybe....

    Splines are just visual aids which do not really take into account signal content.

    Butterworth is NG since need end points and padding is not representative of the signal content.

    Kalman theoretically may work except that there is nothing really known about the system - it would be like using a Kalman to remove hiss from an audio signal (music, etc.) on-line without ANY delay.

    I was thinking that something like Principal Component Analysis might somehow be used on a single input - of course it is intended to reduce the number of inputs(!)

    My Inner Mad Scientist considered this Fourier mal-use:
    1. Take the 1st Fourier harmonic.
    2. Establish its phase by choosing the maximum correlation achieved by surveying lag.
    3. Compile a "response" curve for this maximal correlation for each of the Fourier terms.
    4. Pick a set of dominant harmonics below some smoothing frequency cutoff.
    5. Sum the harmonics with weights equal to the corrrelations.
    6. Confirm a nice fit over the curve interior.
    7. Use the end point value as the smoothed value.

    I don't have the math at hand to demonstrate that my set of harmonics with "adjusted" phase would be orthogonal (can be linearly super positioned) or that the actual correlation coef would be the correct weighting factor - it may need processed into its square or normalized by standard deviation or some other "metric."

    Oh, regarding Fourier Transforms, Discrete Time and Continuous, while aperiodic, they always use a signal that starts and stops at zero in examples. Same for Convolution. So I'm not sure what can be done with this.

    It seems like the examples are finite length signals as opposed to continuous (infinite) signals for which you select some arbitrary interval to perform the FT.

    I am not an electrical engineer but I suspect what I am trying to do may be extremely advanced because the only thing that I have encountered that seems like it would work is the Kalman and you better remember all your Controls Engineering to set that one up if you are lucky enough to have a model. The concept that a Kalman uses "all available information" is quite impressive and seriously overwhealming.

    Thanks in advance,
  2. jcsd
Share this great discussion with others via Reddit, Google+, Twitter, or Facebook

Can you offer guidance or do you also need help?
Draft saved Draft deleted