Kalman Filtering with missing observations

[Todor]

Hi All,

Apologies if this is not the right place for this question.

I have been digging quite a bit around Kalman Filtering recently and I understand how the base case works. However, I am trying to solve quite a complicated problem using the Kalman filter, but am not sure whether the Kalman Filter can be applied in this context.

I am doing work on UK interest rates and for the data inputs for the analysis I am using Government Spot Rates (government bonds) = C (below), SWAP rates = A (below) and a War Bond Perpetuity = B (below). Now, I have the history of Government bonds (for example) from 01/01/1980 until 31/12/2017. I have the history of the perpetuity from 01/01/1932 until 31/12/2017. I have the history of SWAPs from 01/01/2002 until 31/12/2017. What I am trying to do is to use a Kalman Filter to backfill the SWAP history, so say from 01/01/2002 back to 01/01/1980 using inputs and relationships from the available data that I have described above.

The problem can be described in more simple terms imagining the following:

We have three cars – A, B and C. A is driving on a road in the middle, B is driving on a road above A and C is driving on a road below A (as graph below):

B--------------------------------------------------------------------

A----------------------------------------------------X

C--------------------------------------------------------------------

A simple model to calculate the position of A would be to take the difference between B and C and the difference between A and C so that A = (B-C) – (A-C)

We also know that there is correlation between the movements of the cars (i.e. B is correlated to A, A is correlated to C and C is correlated to B).

I am going to use the observed values I have as my measurement inputs (so A, B and C) which feed into the Kalman Filter algorithm to get my estimated state based on my measurements (so by saying measurements I mean the observed data I have) and my simple equation above (theoretical state) – also introducing some noise around these.

The question I have is what happens when I run out of observations for A and I also loose the relationship between A and C (indicated by the X in the above chart) as the Kalman Filter relies on receiving observations. I have thought of setting the measurement of A to 0 and giving it infinitely big variance, so it gets ignored. So the big questions is - is there a way to use the Kalman Filter to “predicted” where my A will go, given I have lost measurement updates for A at point X (if I was using a GPS for the measurement the GPS has stopped working) but I still know where my B and C are, and I also know the distance between B and C but I have lost the distance between A and C (so my model starts relying on less information)? I was thinking something along the lines of Markov Chain Monte Carlo applying the known correlation structure from before but not sure how this can feed into a Kalman Filter process.

Any thoughts will be much appreciated.

Thanks in advance,

Todor

 

[marcusl]

Caveat, I'm not an expert in Kalman filtering so I hope my comments are helpful and not misleading...
Kalman filters assume some model for the "plant" or process being simulated, so that state variables can be chosen intelligently. A classic case is tracking the position of an airplane using ground-based radar observations. Variables are position (x, y, z), velocity (vx, vy, vz) and, sometimes, acceleration (ax, ay, az). Training these parameters using past observations permits prediction of future positions. What are the mechanisms underpinning bond prices, and how are they correlated? Your success at setting up a KF for this problem will depend keenly on the validity of your model.

As for missed observations, the variance of the prediction grows as predictions are made increasingly far from observations. At some point the predictions may become worthless. In radar tracking, tracks are carried along for some time in the absence of new data, but are eventually "killed off."

 

[scottdave]

Caveat, I'm not an expert in Kalman filtering so I hope my comments are helpful and not misleading...
Kalman filters assume some model for the "plant" or process being simulated, so that state variables can be chosen intelligently. A classic case is tracking the position of an airplane using ground-based radar observations... Training these parameters using past observations permits prediction of future positions.

As for missed observations, the variance of the prediction grows as predictions are made increasingly far from observations. At some point the predictions may become worthless. In radar tracking, tracks are carried along for some time in the absence of new data, but are eventually "killed off."

Your description is how I remembered it. The RADAR analogy is a good one for people to picture.

 

[Todor]

Hi,

Thank you both for the responses. Indeed this is the way I understand the Kalman Filter to work.

The question effectively boils down to:

If you run out of observations (measurements) you would set your Kalman Gain to 0 and rely solely on your theoretical model that you are using. So assuming your car has crashed and is no longer there (hence you can't get observations) the theoretical model would say, OK, if you car was still there, this is the potential path it could have gone on to. Do you think that this is a correct interpretation for a situation where your observations completely disappear after a certain point?

Thanks in advance.

Todor

 

[Stephen Tashi]

I notice there are papers written about doing Kalman filtering where data is being lost by transmission over unreliable networks, but I don't find any free copies online.

Kalman filtering is usually done with the idea of controlling a process. How is the idea of control involved in your scenario?

 

[tnich]

I notice there are papers written about doing Kalman filtering where data is being lost by transmission over unreliable networks, but I don't find any free copies online.

Kalman filtering is usually done with the idea of controlling a process. How is the idea of control involved in your scenario?

While it is true that the Kalman filter was originally developed as a method of controlling a process, it needs to predict the process state to control it. As @marcusl pointed out, it is often used in radar tracking applications for predicting positions of aircraft.

Caveat, I'm not an expert in Kalman filtering so I hope my comments are helpful and not misleading...
Kalman filters assume some model for the "plant" or process being simulated, so that state variables can be chosen intelligently. A classic case is tracking the position of an airplane using ground-based radar observations. Variables are position (x, y, z), velocity (vx, vy, vz) and, sometimes, acceleration (ax, ay, az). Training these parameters using past observations permits prediction of future positions. What are the mechanisms underpinning bond prices, and how are they correlated? Your success at setting up a KF for this problem will depend keenly on the validity of your model.

I agree with everything marcusl said here. Let me add a little bit to it. To build a Kalman filter, you need three things:
1) a process model that describes how the state of process $X(t)$ evolves over time. In your case the state vector $X(t)$ would need to include the bond rates, A, B and C, and whatever rates of change or other variables you need to predict the estimated state $\tilde X(t+\Delta t)$ given $\tilde X(t)$.
2) an estimate of the process noise - a function that estimates the growth of the error in the estimated state as a function of time
3) a function $H(X(t))$ that yields a predicted value of the measured values (A, B and C) at time t given the state at time t.

If you can write out these three functions and linearize them, then you can apply an extended Kalman filter to update the state vector and covariance and predict the state covariance into the future.

The measurements do not need to be at regular time intervals, and incomplete measurements can be used to update the filter. So if you only have measurements measurements of B and C, you can still use them to update the filter and make predictions as long as you have a process model that predicts with reasonable accuracy the value of A at time $t+\Delta t$ given $X(t)$. Without that it's garbage in, garbage out.

 

[tnich]

While it is true that the Kalman filter was originally developed as a method of controlling a process, it needs to predict the process state to control it. As @marcusl pointed out, it is often used in radar tracking applications for predicting positions of aircraft.I agree with everything marcusl said here. Let me add a little bit to it. To build a Kalman filter, you need three things:
1) a process model that describes how the state of process $X(t)$ evolves over time. In your case the state vector $X(t)$ would need to include the bond rates, A, B and C, and whatever rates of change or other variables you need to predict the estimated state $\tilde X(t+\Delta t)$ given $\tilde X(t)$.
2) an estimate of the process noise - a function that estimates the growth of the error in the estimated state as a function of time
3) a function $H(X(t))$ that yields a predicted value of the measured values (A, B and C) at time t given the state at time t.

If you can write out these three functions and linearize them, then you can apply an extended Kalman filter to update the state vector and covariance and predict the state covariance into the future.

The measurements do not need to be at regular time intervals, and incomplete measurements can be used to update the filter. So if you only have measurements measurements of B and C, you can still use them to update the filter and make predictions as long as you have a process model that predicts with reasonable accuracy the value of A at time $t+\Delta t$ given $X(t)$. Without that it's garbage in, garbage out.

It seems to me that since you are not trying to predict into the future, but just estimate past values of A, a sequential filter is not necessary. You might get better results with a batch MLE.