Register to reply 
Conditional & uncoditional MSE (in MMSE estimation) 
Share this thread: 
#1
Apr1610, 10:04 PM

P: 16

Hi,
1 Please explain conditional & unconditional mean square error, and their difference. 2 Which one is the solution for minimum MSE estimation? (that is conditional expectation: [tex] E \left[ XY \right] [/tex]. I meant which one is minimized by selecting the conditional expectation.) 3 What is the relation between these two and covariance matrix in Kalman Filter? IMO, the trace of Kalman's covariance (error covariance matrix) is one of these MSEs, but I don't know which one. 4 Is there any other interpretation of Kalman's covariance matrix than the one I mentioned above? (of course there is. I meant I don't know any other and please help me :)) Thanks a lot. 


#2
Apr1910, 09:57 PM

P: 2,504

You can minimize the MSE by minimizing the trace of the posterior error estimate covariance matrix. The trace is minimized when the matrix derivative is zero 


#3
Apr2010, 04:37 AM

P: 16

Thanks for your reply. Actually I've read it. My question is about MMSE estimation in general (and Kalman filter, only as one of its implementations for some particular case). Let me explain more. As I've asked in (1) and (2), I'm not sure what conditional/unconditional MSE exactly are (and which one is minimized by MMSE estimator), but I think they are something like: [tex] E \left[ \left( x  \hat{x} \right) \left( x  \hat{x} \right)^{T}  Z \right] [/tex] and [tex] E \left[ \left( x  \hat{x} \right) \left( x  \hat{x} \right)^{T} \right] [/tex] (where [tex] Z [/tex] is the observation (or sequence of observations as in Kalman) and [tex] \hat{x}=E \left[ x  Z \right] [/tex]). Again, if we look at Kalman as an implementation of MMSE estimator, in some references the conditional MSE is expanded to reach Kalman's covariances, and in some others, the unconditional MSE is used to do so. (BTW, I won't be surprised if someone show that they're equal for Gaussian/linear case, and both references are right). Thanks a lot. 


#4
Apr2010, 08:15 AM

P: 2,504

Conditional & uncoditional MSE (in MMSE estimation)
http://cnx.org/content/m11267/latest/ I take it that P(Z) is your unconditional probability density and p(Zx) is your likelihood function. Then taking the joint density p(x)p(Zx) you can use Bayes Theorem for the posterior density which is the conditional p(xZ)=p(Zx)p(x)/p(Z). I'm not sure why you think the unconditional and conditional probability densities would be equal unless, of course, the prior density and the posterior density were equal. It appears that the MMSE estimate applies to the posterior density p(xZ). EDIT: The link is a bit slow, but works as of my testing at the edit time. 


#5
Apr2010, 10:15 AM

P: 16

The posterior [tex] p \left( xZ \right) [/tex], has a mean and a (co)variance. Its mean is the MMSE estimator, [tex] E \left[ xZ \right] [/tex], and its variance (or the trace of its covariance matrix, if it's a random vector) is the minimum mean squared error. Am I right? So the trace of conditional (co)variance ((co)variance of conditional pdf), that is the trace of [tex] E \left[ \left( x  E \left[ xZ \right] \right) \left( x  E \left[ xZ \right] \right)^{T}  Z \right] [/tex] is the minimum MSE (and [tex] E \left[ \left(xE \left[ xZ \right] \right)^2  Z \right] [/tex] for the case of scalar RV). Is it correct? And then what is the trace of [tex] E \left[ \left( x  E \left[ xZ \right] \right) \left( x  E \left[ xZ \right] \right)^{T}\right] [/tex] ? (or [tex] E \left[ \left(xE \left[ xZ \right] \right)^2 \right] [/tex] for the case of scaler RV). Part Two: As I know MMSE estimation is about finding [tex] h \left( . \right) [/tex] that minimizes the [tex] E \left[ \left( x  h \left( Z \right) \right)^2 \right] [/tex] (MSE). And the answer is [tex] h \left( Z \right) = E \left[ x  Z \right] [/tex]. So the MMSE is [tex] E \left[ \left(xE \left[ xZ \right] \right)^2 \right] [/tex]. Can you see the problem? And a new one :D Maybe it's the answer. Orthogonality principle implies [tex] E \left[ \left( x  E \left[ xZ \right] \right)Z \right] = 0 [/tex], which implies [tex] E \left[ \left( x  E \left[ xZ \right] \right) Z \right] = E \left[ \left( x  E \left[ xZ \right] \right) \right] [/tex]. Does it also imply: [tex] E \left[ \left( x  E \left[ xZ \right] \right) ^2  Z \right] = E \left[ \left( x  E \left[ xZ \right] \right)^2 \right] [/tex]? Is it correct? Thanks. 


#6
Apr2010, 11:27 AM

P: 2,504




#7
Apr2010, 12:19 PM

P: 16

[tex] R_{XX}R_{XZ}R_{ZZ}^{1}R_{ZX} [/tex] which its trace is the *minimum* MSE. I believe the minimization took place when you selected [tex] E \left[ xZ \right] [/tex] as your estimator. About double conditioning, that's the part I do not fully understand either. But you can find it in many references. For example: "Estimation with Applications to Tracking and Navigation" by BarShalom. http://books.google.com/books?id=xz9...page&q&f=false see the bottom of page 204 for example. (There are plenty of these in this book (and also others), I just found one that is included in Google's preview.) Thanks again. Any other ideas? 


#8
Apr2010, 01:35 PM

P: 2,504

Not really. I was thinking of the discussion re the Kalman filter where the trace is minimized using the Kalman gain [tex]K_{k}[/tex] and setting:
[tex]\frac{\partial tr(P_{kk})}{\partial K_{k}}= 0 [/tex] 


#9
Apr2010, 02:05 PM

P: 16

What I understand is that Kalman and MMSE are related (in fact, I think Kalman is the MMSE estimator for the case of Gaussian variables (or Linear MMSE estimator without the assumption of Gaussian variables), for associated linear state (process) and observation equations (models)). Did you see the book? 


#10
Apr2010, 02:05 PM

P: 16

What I understand is that Kalman and MMSE are related (in fact, I think Kalman is the MMSE estimator for the case of Gaussian variables (or Linear MMSE estimator without the assumption of Gaussian variables), for associated linear state (process) and observation equations (models)). Did you see the book? 


#11
Apr2010, 02:43 PM

P: 2,504

If you go back to the wiki article and go down to "Kalman gain derivation" you'll see the equation I wrote. This is how the author suggests minimizing the trace of [tex]P_{kk}[/tex] (posterior estimate covariance matrix). http://en.wikipedia.org/wiki/Kalman_filter And yes, the Kalman Filter is a MMSE estimator. 


#12
Apr2010, 02:58 PM

P: 16

So you're confused about conditional/unconditional MSE too (just like me), right? :D



#13
Apr2010, 03:13 PM

P: 2,504

EDIT:If your reading this in your mail, go to the forum. The post has been edited. The calculation in the Wiki link is specific to the Kalman Filter. 


#14
Apr2010, 03:52 PM

P: 16

In my notation, [tex] X [/tex] is the RV which we're trying to estimate, so the prior (unconditional pdf, which in case of the Kalman filter, is our estimate at the previous step) is [tex] p(x) [/tex].
Actually nothing is wrong with it (using Bayes in order to reach to the posterior). I believe I explained my confusions clear, especially in post #5. What do you think about my statement at the end of that post? Is it true? Thanks a lot. BTW, anyone else has any ideas about our discussion? 


#15
Apr2010, 05:26 PM

P: 2,504

Also, I don't see any problem to getting the MSE from any sample vector. It's the MMSE that can be a challenge. 


Register to reply 
Related Discussions  
Estimation problem. help!  Introductory Physics Homework  1  
Point Estimation  Set Theory, Logic, Probability, Statistics  5  
Error estimation  Advanced Physics Homework  1  
Uncertainty without a given estimation  Introductory Physics Homework  1 