Markov Chain Steady State (?were am i going wrong?)

[Tamis]

Homework Statement

Subpart of the question requires me to find the steady state of the transition matrix:
$P=\begin{bmatrix} 0.1 & 0.7 & 0.2 \\ 0.1 & 0.8 & 0.1\\ 0.3 & 0.1 & 0.6 \end{bmatrix}$

Homework Equations

We thus need to find vector $\boldsymbol{v}$ in the equation $P\boldsymbol{v}=\boldsymbol{v}$ under the constraint $sum(\boldsymbol{v})=1$.

The Attempt at a Solution

Basically a system of linear equations with the added constraint:
$\begin{matrix} -0.9x_1 &+& 0.7x_2 &+& 0.2x_3 & = 0 \\ 0.1x_1 &+& -0.2x_2 &+& 0.1x_3 & = 0 \\ 0.3x_1 &+& 0.1x_2 &+& -0.4x_3 & = 0 \\ x_1 &+& x_2 &+& x_3 & = 1 \end{matrix}$

I put this in matrix form:
$\begin{bmatrix} -0.9 & 0.7 & 0.2 & | 0\\ 0.1 & -0.2 & 0.1 & | 0\\ 0.3 & 0.1 & -0.4 & | 0\\ 1 & 1 & 1 & | 1 \end{bmatrix}$

and solve:
$\begin{bmatrix} 1 & 0 & 0 & | \frac{1}{3}\\ 0 & 1 & 0 & | \frac{1}{3}\\ 0 & 0 & 0 & | 0\\ 0 & 0 & 1 & | \frac{1}{3} \end{bmatrix}$

However this is wrong as the answer states:
$\boldsymbol{v}=\begin{bmatrix} 0.1+0.2\frac{3}{16} \\ 0.9-1.2\frac{3}{16} \\ \frac{3}{16} \\ \end{bmatrix}$


Can anyone tell me were i am going wrong? Not entirely sure how to find the steady state of a markov chain. The only examples i can find are examples of steady states in 2x2 transition matrices.

 

[Ray Vickson]

Homework Statement

Subpart of the question requires me to find the steady state of the transition matrix:
$P=\begin{bmatrix} 0.1 & 0.7 & 0.2 \\ 0.1 & 0.8 & 0.1\\ 0.3 & 0.1 & 0.6 \end{bmatrix}$

Homework Equations

We thus need to find vector $\boldsymbol{v}$ in the equation $P\boldsymbol{v}=\boldsymbol{v}$ under the constraint $sum(\boldsymbol{v})=1$.

The Attempt at a Solution

Basically a system of linear equations with the added constraint:
$\begin{matrix} -0.9x_1 &+& 0.7x_2 &+& 0.2x_3 & = 0 \\ 0.1x_1 &+& -0.2x_2 &+& 0.1x_3 & = 0 \\ 0.3x_1 &+& 0.1x_2 &+& -0.4x_3 & = 0 \\ x_1 &+& x_2 &+& x_3 & = 1 \end{matrix}$

I put this in matrix form:
$\begin{bmatrix} -0.9 & 0.7 & 0.2 & | 0\\ 0.1 & -0.2 & 0.1 & | 0\\ 0.3 & 0.1 & -0.4 & | 0\\ 1 & 1 & 1 & | 1 \end{bmatrix}$

and solve:
$\begin{bmatrix} 1 & 0 & 0 & | \frac{1}{3}\\ 0 & 1 & 0 & | \frac{1}{3}\\ 0 & 0 & 0 & | 0\\ 0 & 0 & 1 & | \frac{1}{3} \end{bmatrix}$

However this is wrong as the answer states:
$\boldsymbol{v}=\begin{bmatrix} 0.1+0.2\frac{3}{16} \\ 0.9-1.2\frac{3}{16} \\ \frac{3}{16} \\ \end{bmatrix}$Can anyone tell me were i am going wrong? Not entirely sure how to find the steady state of a markov chain. The only examples i can find are examples of steady states in 2x2 transition matrices.

You need to solve πP = π for a row vector π, not Pv = v for a column vector v. Also, you should leave out one of the three equations in π = πP and replace it by sum(π) = 1.

Note that *some* sources use transition matrices where the columns sum to 1 instead of the rows; that is, their transition matrices are transposes of yours. In those cases it is, indeed, true that you would need to solve Tv = v, where T is the transition matrix (with column-sums = 1). Every textbook or research monograph I have ever seen about probability and its applications, or about Operations Research, etc., uses the convention that rows sum to 1. For some mysterious reason, many web pages use the other convention, and that can be dangerous if you do not check before using their results. Of course, there are hundreds of books I have not seen, and some of them may use the column-sum = 1 convention; I think it is more common in Asia.

 

[Tamis]

Thnx for the reply! I wasn't aware of this row vector convention, seems rather strange to me. Now i just transpose the transition matrix and use the 'normal method'.

 

[Ray Vickson]

Thnx for the reply! I wasn't aware of this row vector convention, seems rather strange to me. Now i just transpose the transition matrix and use the 'normal method'.

What I wrote is not a 'vector convention'. The steady-state distribution obeys some equations, and the issue is: how do you remember those equations? The easiest way is to recognize that they *can be written as* π = πP for a row vector π; but that is not the original source of the equations, it is just a memory aid.

The real issue is to always check with a reference as to whether their 'transition' matrices have row-sums or column-sums equal to 1.