How can I compute Markov transition probabilities from given data?

In summary: The results I got were:[1] 49903[2] 22974[3] 0.4608319I'm not sure if this is the correct approach, but it seems to make sense to me.
  • #1
user366312
Gold Member
89
3
TL;DR Summary
I know how to mathematically calculate the probability of various Markov Properties.
But, how can I calculate Markov probability from data?
Suppose, I have a Markov Chain as follows:

##S=\{1, 2\}##

##
\alpha = \begin{bmatrix}
0.5&0.5\end{bmatrix}
##

##
P =
\begin{bmatrix}
0.5&0.5\\
0&1
\end{bmatrix}
##

And, the following data regarding 5 steps of Markov Chain taken 12 times:
Markov Chain:
     Steps    1    2    3    4    5
     --------------------------------
       1      2    2    1    1    1
       2      1    1    1    1    1
       3      1    1    1    1    1
       4      1    1    1    1    1
       5      2    2    2    1    1
       6      2    2    2    1    1
       7      1    1    1    1    1
       8      1    1    1    1    1
       9      1    1    1    1    1
      10      2    2    2    2    2
      11      1    1    1    1    1
      12      1    1    1    1    1

How can I calculate,say, ##P(X_1 = 1|X_0 = 1)## and ##P(X_5 = 2|X_2 = 1)## from this table?

Kindly, explain your answer.
 
Physics news on Phys.org
  • #2
In the data, count the number of transitions from one state directly to another (or to itself). That should give you probabilities for a transition matrix. From that, the rest follows. The data you show doesn't seem to match the probability matrix, P. Maybe there is not enough data, but the difference is large.

I don't understand why you are asking about ##P(X_1=1|X_0=1)##. Are you sure that you really understand the transition matrix, P?
 
  • #3
FactChecker said:
In the data, count the number of transitions from one state directly to another (or to itself). That should give you probabilities for a transition matrix. From that, the rest follows. The data you show doesn't seem to match the probability matrix, P. Maybe there is not enough data, but the difference is large.

I used the following code:

R code:
alpha <- c(1, 1) / 2
mat <- matrix(c(1 / 2, 0, 1 / 2, 1), byrow=TRUE, nrow = 2, ncol = 2) # Different than yours

chainSim <- function(alpha, mat, n)
{
    out <- numeric(n) # create a numeric vector of size-n
    out[1] <- sample(1:2, 1, prob = alpha) # create 1 random number
    for(i in 2:n)
    {
        #set.seed(1)
        out[i] <- sample(1:2, 1, prob = mat[out[i - 1], ])
    }
    out
}

set.seed(1)
# Doing once
#sss <- chainSim(alpha, mat, 1 + 5)
#sss
# [1] 2 2 2 2 2 2

# Doing 100 times
sim <- replicate(chainSim(alpha, mat, 1 + 5), n = 12)
t(sim)
# rowMeans(sim - 1)

# mean(sim[2, sim[1, ] == 1] == 1)
# [1] 0.4583333
FactChecker said:
I don't understand why you are asking about ##P(X_1=1|X_0=1)##.

Coz, that is what I was told in my class to find from the simulation.

FactChecker said:
Are you sure that you really understand the transition matrix, P?
I think so. Coz, I passed the MidTerm test.
 
  • #4
I don't see anything wrong with the code, but your results have many more transitions from 2 to 2 than I get from 2 to one (9 versus 3). I thought those two transitions should have equal probability. I have tried a couple of seeds and no seed at all and seem to keep getting more transitions 2 -> 2 than 2->1. I don't understand that. I will have to defer to anyone who understands this better.

CORRECTION: From the data, I thought that 1 was supposed to be the absorbing state. That is wrong. From the definition of P, 2 is the absorbing state and 1 transitions to 1 and 2 with equal probabilities.
 
Last edited:
  • #5
I made a couple of 1000 simulation runs (using different seeds) and keep getting about twice as many 2->2 transitions than 2->1 transitions. So there is definitely something that I don't understand.

CORRECTION: From the data, I thought that 1 was supposed to be the absorbing state. That is wrong. From the definition of P, 2 is the absorbing state and 1 transitions to 1 and 2 with equal probabilities.
 
Last edited:
  • #6
I think that your code defining mat is wrong: mat <- matrix(c(1 / 2, 0, 1 / 2, 1), byrow=TRUE, nrow = 2, ncol = 2)
That gives probabilities [0.5, 1.0] in chainSim where it should be using [0.5, 0.5].
I changed it to mat <- matrix(c(1 / 2, 1/2, 0, 1), byrow=TRUE, nrow = 2, ncol = 2) and got transition results that make sense. 2 is an absorbing state and the transitions 1->1 and 1->2 are equally probable. Looking at your code, that seems to be what you intended.
 
  • #7
FactChecker said:
..., that seems to be what you intended.

Okay. Thanks.

Could you now shed some light on finding ##P(X_1=1|X_0=1)##?

The exact question was like the following:

Estimate P(X1 = 1|X0 = 1) from simulation. Compare your result with the exact probability you calculated.
 
  • #8
user366312 said:
Okay. Thanks.

Could you now shed some light on finding ##P(X_1=1|X_0=1)##?

The exact question was like the following:
To do that, I recommend greatly increasing the number of runs. Then count how many times the initial state is 1, #initialState1. In those cases, also count how many runs have the second state of 1, #secondState1. the estimated probability would be #secondState1/#initialState1

I set the R code to do 100000 runs and used this code at the end to do the counting:

R code:
numInitial=which(sim[1,]==1)
N1= length(numInitial)
print(N1)
numSecond = which(sim[2,numInitial]==1)
N2 = length(numSecond)
print(N2)
prob = N2/N1
print(prob)
 

1. How do I define a Markov chain?

A Markov chain is a mathematical model that describes a sequence of events where the probability of transitioning from one state to another depends only on the current state and not on the previous states.

2. What data do I need to compute Markov transition probabilities?

You will need a dataset that includes the states or categories of interest and the corresponding transition probabilities between those states.

3. How do I calculate the transition probabilities?

The transition probabilities can be calculated by dividing the number of transitions from one state to another by the total number of transitions from that state.

4. Can I use any type of data to compute Markov transition probabilities?

Markov transition probabilities can be computed from various types of data, such as time series data, survey data, or any other type of data that includes categorical variables and their transitions.

5. What are some common applications of Markov transition probabilities?

Markov transition probabilities are commonly used in fields such as finance, economics, biology, and engineering to model and predict the behavior of complex systems. They can also be used in natural language processing, speech recognition, and other areas of artificial intelligence.

Similar threads

  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
2K
Replies
0
Views
263
  • Set Theory, Logic, Probability, Statistics
2
Replies
41
Views
4K
  • Set Theory, Logic, Probability, Statistics
Replies
17
Views
803
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
923
  • Set Theory, Logic, Probability, Statistics
Replies
4
Views
2K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
6
Views
1K
  • Set Theory, Logic, Probability, Statistics
Replies
1
Views
992
  • Set Theory, Logic, Probability, Statistics
Replies
2
Views
1K
Back
Top