# Markov Process Limit: Calculating $u_k$ as a Function of $a,b$

• MHB
• mathmari
In summary, if the transition matrix $P$ is row stochastic, then the equation $u_k=x_k+y_k$ is a Markov process. The values of $a$ and $b$ that make the equation a Markov process are when $P$ is a non-negative matrix and the sum of the columns is 1. To calculate $u_k$ as a function of $a,b$, which conditions do $a,b$ have to satisfy so that $u_k$ approximates a finite limit as $k\rightarrow \infty$ and which is that limit?
mathmari
Gold Member
MHB
Hey!

We consider the equation \begin{equation*}u_{k+1}=\begin{pmatrix}a & b \\ 1-a & 1-b\end{pmatrix}u_k \ \text{ with } \ u_0=\begin{pmatrix}1 \\1 \end{pmatrix}\end{equation*}

For which values of $a$ and $b$ is the above equation a Markov process?
Calculate $u_k$ as a function of $a,b$.
Which conditions do $a,b$ have to satisfy so that $u_k$ approximates a finite limit as $k\rightarrow \infty$ and which is that limit? I have done the following:

So that the equation is a Markov process the element of the matrix must be non-negativ and the sum of the colums must be $1$.
So, we get $0\leq a,b\leq 1$.

\begin{align*}&u_0=\begin{pmatrix}1 \\ 1\end{pmatrix} \\ &u_1=\begin{pmatrix}a+b \\ 2-(a+b)\end{pmatrix} \\ &u_2=\begin{pmatrix}a^2+2b-b^2 \\ 2-(a^2+2b-b^2)\end{pmatrix}\\ &u_3=\begin{pmatrix}a^3+2ab-b^2a+2b-a^2b-2b^2+b^3 \\ 2-(a^3+2ab-b^2a+2b-a^2b-2b^2+b^3)\end{pmatrix} \\ & \ldots\end{align*}

Let $u_k=(x_k,y_k)^T$ then it holds that \begin{align*}&x_k+y_k=2 \\ & x_{k+1}=(a−b)x_k+2b \\ & x0=y0=1\end{align*}

We have that \begin{align*} x_{k+1}&=(a−b)x_k+2b \\ & =(a-b)^2x_{k-1}+2b(1-b)+2b \\ & = (a-b)^3x_{k-2}+2b(a-b)^2+2b(a-b)+2b \\ & =\ldots \\ & = (a-b)^{k+1}x_0+2b\left (\sum_{i=0}^k(a-b)^i\right )\end{align*}

Therefore, we get \begin{equation*}u_k=\begin{pmatrix}x_k \\ y_k\end{pmatrix}=\begin{pmatrix}(a-b)^{k+1}x_0+2b\left (\sum_{i=0}^k(a-b)^i\right )\\ 2-(a-b)^{k+1}x_0+2b\left (\sum_{i=0}^k(a-b)^i\right )\end{pmatrix} \end{equation*}

So that $u_k$ approximates a finite limit as $k\rightarrow \infty$ it must be $0<a-b<1$.

Is everything correct? (Wondering)

mathmari said:
Hey!

We consider the equation \begin{equation*}u_{k+1}=\begin{pmatrix}a & b \\ 1-a & 1-b\end{pmatrix}u_k \ \text{ with } \ u_0=\begin{pmatrix}1 \\1 \end{pmatrix}\end{equation*}

For which values of $a$ and $b$ is the above equation a Markov process?
Calculate $u_k$ as a function of $a,b$.
Which conditions do $a,b$ have to satisfy so that $u_k$ approximates a finite limit as $k\rightarrow \infty$ and which is that limit?

$P := \begin{pmatrix}a & b \\ 1-a & 1-b\end{pmatrix}$

it's customary in probability theory to make $P$ row stochastics (rows sum to one, nonnegative entries) not column stochastic, but in principle there's no issue with being column stochastic. You've stated the conditions for being column stochastic just fine. Note with $\mathbf 1$ the ones vector (which $\mathbf u_0$ is for some reason), being column stochastic means

$\mathbf 1^T P = \mathbf 1^T$
or
$P^T \mathbf 1 = \mathbf 1$

so the ones vector is a (left) eigenvector with eigenvalue of 1.

We'll call the eigenvalue $\lambda_1 = 1$, which is of course an eigenvalue of $P$ and $P^T$
mathmari said:
... Therefore, we get \begin{equation*}u_k=\begin{pmatrix}x_k \\ y_k\end{pmatrix}=\begin{pmatrix}(a-b)^{k+1}x_0+2b\left (\sum_{i=0}^k(a-b)^i\right )\\ 2-(a-b)^{k+1}x_0+2b\left (\sum_{i=0}^k(a-b)^i\right )\end{pmatrix} \end{equation*}

So that $u_k$ approximates a finite limit as $k\rightarrow \infty$ it must be $0<a-b<1$.

Is everything correct? (Wondering)

I appreciate all the work, but the conclusion doesn't follow. I see no problem with e.g. $b= 0.7$ and $a = 0.2$, you'll still get a valid limit.

A much simpler approach is to make use of trace, so

$\text{tr} \big(P\big) = a + 1-b = \lambda_1 + \lambda_2 = 1 +\lambda_2$

so
$a -b = \lambda_2$

if $\big \vert \lambda_2 \big \vert \lt 1$, then you must get a finite limit (why?). You should be able to find the single (right) eigenvector in the nullspace of $\big(P-I\big)$ and from here figure out the limiting value of $\mathbf u_k$.

If $\lambda_2 = -1$ you have periodic behavior (period of 2) and no limit can exist; in this case the matrix is a pairwise swap permutation matrix. The special case of $\lambda_2 = 1$ remains to be considered and you can check that this implies $P$ is the identity matrix, which is no problem for limits...

## 1. What is a Markov process?

A Markov process is a stochastic model used to describe the evolution of a system over time. It is a type of random process where the future state of the system depends only on its current state, and not on any previous states.

## 2. What is the Markov process limit?

The Markov process limit is the long-term behavior of a Markov process, where the system reaches a steady state and the probabilities of being in each state remain constant over time.

## 3. How is u_k calculated as a function of a and b?

The value of u_k is calculated using the formula u_k = a + b * u_(k-1), where a and b are constants and u_(k-1) is the previous value of u_k.

## 4. What is the significance of u_k in a Markov process?

u_k represents the probability of being in a certain state after k steps in a Markov process. It is used to analyze the long-term behavior of the system and make predictions about its future states.

## 5. Can the values of a and b change in a Markov process?

Yes, the values of a and b can change in a Markov process depending on the specific system being modeled. These values are determined by the initial conditions and the transition probabilities between states.

Replies
4
Views
2K
Replies
6
Views
912
Replies
10
Views
1K
Replies
34
Views
2K
Replies
5
Views
1K
Replies
2
Views
1K
Replies
14
Views
2K
Replies
14
Views
2K
Replies
8
Views
1K
Replies
8
Views
3K