# Markov Chain Expected Value

1. Sep 10, 2015

### PhyzX

1. The problem statement, all variables and given/known data
Markov process has probabilities $p_{j,j+1} = 1-p_{j,0} = (\frac{j+1}{j+2})^k$ for j=0,1,2,...
If $T_j = min[n>1 : X_n=j]$
What is $E[T(j)|X_0=j]$ for j=0,1,2,...?

2. Relevant equations

3. The attempt at a solution
I figured that $T(j)|X_0=j$ is j+1 but don't know how to work out the expectation of this

Last edited: Sep 10, 2015
2. Sep 10, 2015

### Ray Vickson

Your statement makes no sense: $j+1$ is an integer number, while $T(j)|X_0 = j$ is a random variable. What do you really mean, here?

Also: get rid of that pesky "?", because the way you have written it, you are asking us whether or not you know how to do something.

3. Sep 10, 2015

### PhyzX

I think I don't quite understand the question. My interpretation of $T(j)|X_0 = j$ is that it is equal to the minimum number of steps needed to return to state $j$. Looking at the transition probabilities I figured that the chain should immediately return to state 0 and then climb back up to state $j$ hence taking $j+1$ steps. My thinking might be entirely wrong though.

4. Sep 10, 2015

### Ray Vickson

It is entirely wrong.

If $X_0 = j$, where can the chain go next? It can go to state 0 or state j+1. If it had gone to 0 it can flip numerous times back and forth between state 0 and states 1,2, ..., j-1, then go to j in the next step after that. If it had gone to state j+1, it can next go to j+2 or to 0; once at 0 the behavior just described takes over. If at j+2 it next goes either to j+3 or down to 0, etc. etc.

Getting the exact probability distribution of the first return-time to j is not easy, but there are (in your textbook, or notes, or on-line) some equations which, when solved, spit out the expected return time as part of their solution. I have not looked at the details in this case, so I don't know how easy it would be to solve the equations.

However, there is a more serious issue: can we be sure that the chain does, indeed, return to state j with probability 1? If so, can we be sure the expected return time is finite? The point is that $p_{j,j+1} \to 1$ as $j \to \infty$, so the farther out we are, the less chance there is of going back. There are test criteria available that allow you to deal with some such cases, but how easy they are to apply in this case is something I have not investigated.

5. Sep 10, 2015

### PhyzX

Thank you, I understand what the question is asking now. Would you happen to know what these equations might be called so that I can search them up.

6. Sep 10, 2015

### Ray Vickson

Are you telling me that your course notes and/or textbook does not have such material? I am surprised, since it is so very fundamental to the subject. However, you can Google "expected first-passage time" or "expected first-passage time + markov chain".