Help with this problem of stationary distributions

[jakub jemez]

I need help with this

Consider an irreducible Markov chain with $\left|S\right|<\infty $ and transition function $p$.

Suppose that $p\left(x,x\right)=0,\ x\in S$ and that the chain has a stationary distribution $\pi .$

Let $p_x,x\ \in S,$ such that $0<\ p_x<1$ and $Q\left(x,y\right),\ x\in S,\ y\ \in S\ $where
$Q\left(x,x\right)=1-p_x$
$Q\left(x,y\right)=p_xp\left(x,y\right),\ if\ y\neq x.$
Proof that

1)The $Q$ function is a transition function and the chain becomes irreducible again with the $Q$ function.

2)The chain has a stationary distribution $\widehat{\pi }$ ($about\ Q$) given by
$\widehat{\pi }\left(x\right)=\frac{p^{-1}_x}{{\mathrm{\Sigma }}_{y\in S}p^{-1}_y\pi (y)\pi (x)},\ x\in S.$

What interpretation does the transition function $Q$ ?

Do you have any suggestions for part one?

Now, for part 2, it occurs to me that it could be as follows;
From what we already have by hypothesis, that is, that ${\mathrm{\Sigma }}_{x\in S}\pi \left(x\right)=1y$

$\sum_{x\in S}{\pi \left(x\right)}p\left(x,y\right)=\pi \left(y\right)\_\_\_\_\_\_\_\_\_(1)$

then, we look carefully (after doing several operations) that (1) (and for de definition of $Q$) it follows that

$\sum_{x\in S}{\frac{\pi \left(x\right)}{p_x}}Q\left(x,y\right)=\frac{\pi \left(y\right)}{p_y}\_\_\_\_\_\_\_\_\_(2)$And then we can see thanks to (2) that $\widehat{\pi }\left(x\right)=\frac{\widehat{\pi }\left(x\right)}{p_x}\ $it's stationary, but it's still missing that ${\mathrm{\Sigma }}_x\widehat{\pi }\left(x\right)=1$

So, I need help to complete $\widehat{\pi }\left(x\right)=\frac{\widehat{\pi }\left(x\right)}{p_x}$ in such a way that it is satisfied ${\mathrm{\Sigma }}_x\widehat{\pi }\left(x\right)=1$

 

[StoneTemplePython]

Welcome to PF. Try encapsulating your LaTeX in double hashtags, instead of single dollar signs -- that should fix rendering issues.

There are a lot of different approaches here. I am partial to the approach in MIT's 6.262, available on OCW. In particular, this course suggests using matrices for finite state markov chains and renewal theory for countably infinite chains.

Given that you have $\left|S\right|<\infty$, I read this as a finite state markov chain, and hence $Q$ (and $p$) should be interpreted as a matrix.
- - - -
Under course notes, they have a draft of the author's text on this chapter:

https://ocw.mit.edu/courses/electri...ring-2011/course-notes/MIT6_262S11_chap03.pdf

I'd suggest looking through that.
- - - -

To prove that $Q$ acts as a transition matrix, you need to verify that each component is real non-negative, and each row (or column, depending on convention) sums to one. You may want to look at the $kth$ row of $Q$ and show what it looks like in general.

I would also strongly suggest working out a full small example. I.e. suppose that there are 5 states. What does $p$ look like? And what does $Q$ look like? PF guidelines generally require you to show your work, and working through a full example would go a long way here.

I found the notation a bit hard to follow, but it looks like Q has the effect of allocating probability along the diagonal (i.e. self loops), which must get rid of periodicity in a graph (why?). If the graph is still connected after $Q$, then you must get an irreducible graph after large enough iterations (why?). Put differently, the above two considerations would ensure a single recurrent class with no periodicity in the graph.