1. Not finding help here? Sign up for a free 30min tutor trial with Chegg Tutors
    Dismiss Notice
Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

Markov chain: finding a general solution

  1. Oct 2, 2015 #1
    1. The problem statement
    Given a stochastic matrix P with states [itex]s_1...s_5[/itex]:

    [itex]
    P =
    \begin{pmatrix}
    1 & p_2 & 0 & 0 & 0\\
    0 & 0 & p_3 & 0 & 0\\
    0 & q_2 & 0 & p_4 & 0\\
    0 & 0 & q_3 & 0 & 0 \\
    0 & 0 & 0 & q_4 & 1
    \end{pmatrix}
    [/itex]

    and the matrix A (which is obviously related to P, but I can't see how... ):

    [itex]
    A =
    \begin{pmatrix}
    1 & -q_2 & 0 \\
    -p_3 & 1 & -q_3 \\
    0 & -p_4 & 1
    \end{pmatrix}
    [/itex]

    The question is how the vector [itex]y = (x_2,x_3, x_4)[/itex] is a solution to the system [itex]Ay =b [/itex] for a certain b that I am supposed to find.

    2. Relevant equations
    The relevant equations are:
    [itex]
    x_j^K = 1
    [/itex]
    for all closed states

    [itex]
    x_j^K = \sum_{i=1}^n p_{ij }x_i^K
    [/itex]
    for all non-closed states


    3. The attempt at a solution
    I started by expanding Ay:

    [itex]
    Ay=
    \left(
    \begin{matrix}
    1 & -q_2 & 0 \\
    -p_3 & 1 & -q_3 \\
    0 & -p_4 & 1
    \end{matrix}
    \right)
    \left(
    \begin{matrix}
    x_2\\
    x_3\\
    x_4
    \end{matrix}
    \right)
    =
    \left(
    \begin{matrix}
    1x_2 + -q_2x_3 \\
    -p_3x_2 + x_3 -q_3x_4 \\
    -p_4x_3 + x_4
    \end{matrix}
    \right)
    = b
    [/itex]

    But that seems to get me nowhere. The question hints to using the formulas listed above, but I cant see how I can use them to find b.

    I appreciate all help.
     
  2. jcsd
  3. Oct 2, 2015 #2

    Ray Vickson

    User Avatar
    Science Advisor
    Homework Helper

    Your question uses the exact opposite convention for ##P## used in all the books on my shelves and all the papers I have ever read on the subject: usually, in the English-speaking world, stochastic matrices have rows summing to 1, not the columns. I would love to replace your ##P## by its transpose, but that would create too much confusion; so I will write ##P(i \to j)## for the one-step transition probability from state i to state j. (In your convention, ##p_{ij} = P(j \to i)##, but in my convention---by far the majority--- ##p_{ij} = P(i \to j)##.) The clumsier notation avoids confusion.

    Anyway, if ##I## is the ##5 \times 5## identity matrix, your matrix ##A## is the submatrix of ##I - P## consisting of rows 2--4 and columns 2--4, so it has to do with the equations (a): ##x_i = \sum_{j \neq 1,5} P(i \to j) x_j + b_i, \; i = 2,3,4##, or maybe the equations (b): ##X_j = \sum_{i \neq 1,4} X_i P( i \to j) + R_j, \; j = 2,3,4##.

    First, note that states 1 and 5 are absorbing, and states {2,3,4} form an intercommunicating but transient class. That is, starting from state 2, 3 or 4, the process will eventually (i.e., with probability 1) be absorbed into state 1 or state 4. One way equations like (a) arise is in computing the absorption probability for state 1, starting from states 2, 3 or 4. If we let ##f_i = ## probability of absorption in state 1, starting from state ##i##, then these can be found from the standard equations
    [tex] f_i = P(i \to 1) + \sum_{j =2}^4 P(i \to j) f_j, \; i = 2,3,4 [/tex] If you re-write these as ##f_i - \sum P(i \to j) f_j = \text{something}## you essentially involve the matrix ##A## you were given.

    Other than that, I have no idea what the questioner wants.
     
  4. Oct 2, 2015 #3
    Thank you, this gave me something to work on.
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted



Similar Discussions: Markov chain: finding a general solution
  1. Markov chains (Replies: 0)

  2. Markov Chain (Replies: 1)

Loading...