How to prove Bayes' rule for probability measures?

[winterfors]

Consider a probability space (\Theta, \Sigma_\Theta, P_\Theta), where P_\Theta is a probability measure on the sigma-algebra \Sigma_\Theta.

Each element x \in \Theta maps onto another probability measure P_{\Omega | x}, on a sigma-algebra \Sigma_\Omega on another space \Omega.

In this situation, one should (as far as I can see) be able to write write a measure-theoretic generalization of Bayes' rule

{P_{\Theta |y}}(A) = \int\limits_{x \in A} {\frac{{d{P_{\Omega |x}}}}{{d{P_\Omega }}}(y)d{P_\Theta }}

for any A \subseteq \Theta, given an observation y \in \Omega where

{P_\Omega } = \int\limits_{x \in \Theta } {{P_{\Omega |x}}d{P_\Theta }}

and {d{P_{\Omega |x}}/d{P_\Omega }} is the Radon–Nikodym derivative of P_{\Omega |x} with respect to P_{\Omega}.


The problem is that I cannot see how to prove it (I;m sure the proof is fairly simple). Anyone wants to help?

 

[SW VandeCarr]

This proof may or may not be what you are looking for. It's based on a filtration: F_1, F_2, ...,F_{N-1}, F_N as an increasing sequence of sigma algebras.

http://01law.wordpress.com/2011/04/09/bayes-rule-and-forward-measure/

 

[winterfors]

This proof may or may not be what you are looking for. It's based on a filtration: F_1, F_2, ...,F_{N-1}, F_N as an increasing sequence of sigma algebras.

http://01law.wordpress.com/2011/04/09/bayes-rule-and-forward-measure/

As far as I can see, it's not quite what I'm looking for. What I'm trying to do above is to reformulate Bayes' rule for probability densities, usually expressed

p(x|y) = \frac{{p(y|x)}}{{p(y)}}p(x)

which follows trivially from the definition of a joint probability density p(x,y) = p(y|x) p(x). But for probability measures, it gets slightly more tricky...

 

[SW VandeCarr]

As far as I can see, it's not quite what I'm looking for. What I'm trying to do above is to reformulate Bayes' rule for probability densities, usually expressed

p(x|y) = \frac{{p(y|x)}}{{p(y)}}p(x)

which follows trivially from the definition of a joint probability density p(x,y) = p(y|x) p(x). But for probability measures, it gets slightly more tricky...

The rule for probability densities follows from Bayes' Rule and the Law of Total Probabilities:

f_X(x|Y=y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{\int_{-\infty}^{\infty} f_Y(y|X=\xi )\,f_X(\xi )\,d\xi }\!.

Do you want to reformulate this?

EDIT: Any reformulation will need to take into account that the application of Bayes' Rule must be expressed as a posterior probability which is defined in terms of a probability space and the Law of Total Probabilities.

 

[vladb]

Each element x \in \Theta maps onto another probability measure P_{\Omega | x}, on a sigma-algebra \Sigma_\Omega on another space \Omega.

What is this mapping? How exactly is P_{\Omega | x} defined?

 

[winterfors]

What is this mapping? How exactly is P_{\Omega | x} defined?

I only meant that for every x \in \Theta there is one probability measure P_{\Omega | x} on \Sigma_\Omega over the space \Omega. The probaility measures P_{\Omega | x} are thus consitional on x.

This allows us to define a joint probability measure on the (Cartesian) product space (\Theta \times \Omega ,{\Sigma _\Theta } \times {\Sigma _\Omega })
{P_{\Theta \times \Omega }}(C) \equiv \int\limits_{x\in A} {{P_{\Omega |x}}({B_x})d{P_\Theta }} for any C \in {\Sigma _\Theta } \times {\Sigma _\Omega }, where A \in {\Sigma _\Theta } and {B_x} \in {\Sigma _\Omega } are defined as A = \{ x:(x,y) \in C\} and {B_x} = \{ y:(x,y) \in C\}.If one could prove that {P_{\Omega |x}} = \frac{{d{P_{\Theta \times \Omega }}}}{{d{P_\Theta }}}(x) where d{P_{\Theta \times \Omega }}/d{P_\Theta } is the Radon–Nikodym derivative of {P_{\Theta \times \Omega }} with respect to {P_\Theta }, then the theorem above (that I want to prove), i.e.
{P_{\Theta |y}}(A) = \int\limits_{x\in A} {\frac{{d{P_{\Omega |x}}}}{{d{P_\Omega }}}(y)d{P_\Theta }} would follow by symmetry, but I'm not sure of how to do that...

 

[vladb]

This allows us to define a joint probability measure on the (Cartesian) product space (\Theta \times \Omega ,{\Sigma _\Theta } \times {\Sigma _\Omega })
{P_{\Theta \times \Omega }}(C) \equiv \int\limits_{x\in A} {{P_{\Omega |x}}({B_x})d{P_\Theta }}

Perhaps I've misunderstood you, but... f would be a Radon-Nikodym derivative if
<br /> P_{\Theta\times\Omega}(A) = \int_A f \, dP_\Theta<br /> for all A \in \Sigma_\Theta\times\Sigma_\Omega (product sigma-field). However this doesn't make any sense, because you have different measurable spaces on the left and right hand sides. Actually, the right hand side does not mean anything.

 

[winterfors]

Perhaps I've misunderstood you, but... f would be a Radon-Nikodym derivative if
<br /> P_{\Theta\times\Omega}(A) = \int_A f \, dP_\Theta<br /> for all A \in \Sigma_\Theta\times\Sigma_\Omega (product sigma-field). However this doesn't make any sense, because you have different measurable spaces on the left and right hand sides. Actually, the right hand side does not mean anything.

In the equation you refer to, the integration is not over a subset A \in \Sigma_\Theta\times\Sigma_\Omega but over a subset of \Theta.

 

[vladb]

In the equation you refer to, the integration is not over a subset A \in \Sigma_\Theta\times\Sigma_\Omega but over a subset of \Theta.

I was just quoting your definition of P_{\Theta\times\Omega}. Later in you question you refer to \frac{dP_{\Theta\times\Omega}}{dP_\Omega} as a Radon-Nikodym derivative, but this, as I tried to point out in my previous reply, doesn't make sense.

 

[winterfors]

I was just quoting your definition of P_{\Theta\times\Omega}. Later in you question you refer to \frac{dP_{\Theta\times\Omega}}{dP_\Omega} as a Radon-Nikodym derivative, but this, as I tried to point out in my previous reply, doesn't make sense.

Yeah, I meant that {dP_{\Theta\times\Omega}}/{dP_\Omega} would not be a function \Theta\rightarrow[0,\infty), but a function \Theta\rightarrow\Gamma, where \Gamma is a set of probaility measures on \Sigma_\Omega. It might be a bit of a strech to call it a Radon-Nikodym derivative...

 

[winterfors]

The rule for probability densities follows from Bayes' Rule and the Law of Total Probabilities:

f_X(x|Y=y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{\int_{-\infty}^{\infty} f_Y(y|X=\xi )\,f_X(\xi )\,d\xi }\!.

Do you want to reformulate this?

EDIT: Any reformulation will need to take into account that the application of Bayes' Rule must be expressed as a posterior probability which is defined in terms of a probability space and the Law of Total Probabilities.

Yes, reformulating this using probaility measures instead of probability densities would allow me to prove what I want.

 

[vladb]

Hmm.. you denote by P_{\Omega | x} just some arbitarily chosen probability measure on (\Omega, \Sigma_\Omega), i.e. you have a family of measures parametrized by x \in \Theta. Then you define

a joint probability measure on the (Cartesian) product space (\Theta \times \Omega ,{\Sigma _\Theta } \times {\Sigma _\Omega })
{P_{\Theta \times \Omega }}(C) \equiv \int\limits_{x\in A} {{P_{\Omega |x}}({B_x})d{P_\Theta }}

To answer the question, whether
{P_{\Omega |x}} = \frac{{d{P_{\Theta \times \Omega }}}}{{d{P_\Theta }}}(x), where the RHS is not a Radon-Nikodym derivative, you need to define the RHS, i.e. it cannot be just some function \Theta \to \Gamma. Also you can't define it using the same equation you used for definition of P_{\Theta\times\Omega}. A theorem would show that two separately defined things are equal, but I see only one defined object.

I sort of understand what you are trying to do, but not quite. Can you give a simple example of calculations with concrete values/sets of what the theorem would look like? E.g. in discrete case?