What is Markov process: Definition and 17 Discussions

A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. A countably infinite sequence, in which the chain moves state at discrete time steps, gives a discrete-time Markov chain (DTMC). A continuous-time process is called a continuous-time Markov chain (CTMC). It is named after the Russian mathematician Andrey Markov.
Markov chains have many applications as statistical models of real-world processes, such as studying cruise control systems in motor vehicles, queues or lines of customers arriving at an airport, currency exchange rates and animal population dynamics.Markov processes are the basis for general stochastic simulation methods known as Markov chain Monte Carlo, which are used for simulating sampling from complex probability distributions, and have found application in Bayesian statistics, thermodynamics, statistical mechanics, physics, chemistry, economics, finance, signal processing, information theory and speech processing.The adjectives Markovian and Markov are used to describe something that is related to a Markov process.

View More On Wikipedia.org
Recent contents Top users
  1. chwala

    Solve the given problem involving conditional probability

    Phew! took time to figure this out...i guess there may be a way to use combinations or markov process i do not know... anyway, it was pretty straightforward, we have the ##P_r(w) = \dfrac{n-3}{n}## from box ##X## and this will result in ##P_r(w) = \dfrac{4}{n+1}## in box ##Y##. Together i...
  2. K

    I Determine the limit in a Markov process over time

    I have already in a previous task shown that A is not irreducible and not regular, which I think is correct. I don't know if I can use that fact here in some way. I guess one way of solving this problem could be to find all eigenvalues, eigenvectors and diagonalize but that is a lot of work and...
  3. M

    MHB Markov Process Limit: Calculating $u_k$ as a Function of $a,b$

    Hey! :o We consider the equation \begin{equation*}u_{k+1}=\begin{pmatrix}a & b \\ 1-a & 1-b\end{pmatrix}u_k \ \text{ with } \ u_0=\begin{pmatrix}1 \\1 \end{pmatrix}\end{equation*} For which values of $a$ and $b$ is the above equation a Markov process? Calculate $u_k$ as a function of $a,b$...
  4. Aslet

    I Probability of a Stochastic Markov process

    Hi everyone! I'm approaching the physics of stochastic processes. In particular I am studying from "Handbook of stochastic processes - Gardiner". This book defines a stationary process like: $$ p(x_1, t_1; x_2, t_2; ...; x_n, t_n) = p(x_1, t_1 + \epsilon; x_2, t_2 + \epsilon; ...; x_n, t_n +...
  5. redtree

    A Deriving Probability Amplitude from Markov Density Function

    1. Given a Markov state density function: ## P((\textbf{r}_{n}| \textbf{r}_{n-1})) ## ##P## describes the probability of transitioning from a state at ## \textbf{r}_{n-1}## to a state at ##\textbf{r}_{n} ##. If ## \textbf{r}_{n-1} = \textbf{r}_{n}##, then ##P## describes the probability of...
  6. J

    Limit of a continuous time Markov chain

    Homework Statement Calculate the limit $$lim_{s,t→∞} R_X(s, s+t) = lim_{s,t→∞}E(X(s)X(s+t))$$ for a continuous time Markov chain $$(X(t) ; t ≥ 0)$$ with state space S and generator G given by $$S = (0, 1)$$ $$ G= \begin{pmatrix} -\alpha & \alpha \\ \beta & -\beta\...
  7. J

    Birth and death process -- Total time spent in state i

    Homework Statement Let X(t) be a birth-death process with parameters $$\lambda_n = \lambda > 0 , \mu_n = \mu > 0,$$ where $$\lambda > \mu , X(0) = 0$$ Show that the total time T_i spent in state i is $$exp(\lambda−\mu)-distributed$$ 3. Solution I have a hard time understanding this...
  8. R

    Noise modeling with Markov modeling

    Hi I'm using accelerometer & horizontal gyroscope in order to replace GPS. Now, I'want to model the noise with first order markov process, to use it in kalman filter. I recorded measurement on all axes and computed auto-correlation. This picture represents auto-correlation on one of axes...
  9. M

    Random process derived from Markov process

    I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks. Let r(t) be a finite-state Markov jump process described by \begin{alignat*}{1} \lim_{dt\rightarrow 0}\frac{Pr\{r(t+dt)=j/r(t)=i\}}{dt} & =q_{ij} \end{alignat*} when i \ne...
  10. W

    Is Markov process a Brownian process?

    Hi all, I know that Brownian process can be shown as Markov process but is the converse possible? I mean can we show that a markov process is a brownian process? Thanks in advance.
  11. S

    Gibbs sampler as a Markov process

    I'm trying to learn more about Markov chains and came across the Gibbs sampler x_1{t+1} ~ p(x_1|x_2 = x_2{t},...x_n{t}) x_2{t+1} ~ p(x_2|x_1 = x_1{t+1},x_3 = x_3{t},...,x_n{t}) . . . x_i{t+1} ~ p(x_i|x_1 = x_1{t+1},...,x_(i-1) = x_(i-1){t+1},x_(i+1) = x_(i+1){t},...,x_n{t}) Supposedly...
  12. P

    MHB Calculating Probabilities of Exchange Rate Fluctuations with Markov Processes

    It is widely believed that the daily change in currency exchange rates is a random variable with mean 0 and variance vThat is, if Yn represents the exchange rate on the nth day, Yn = Yn−1 + Xn, n = 1, 2, . . . where X1,X2, . . . are independent and identicallydistributed normal random variables...
  13. H

    Prove Markov Process Using Induction

    Hi, I proved the following statement by induction. Does anyone see any oversights or glaring errors? It is for a class where the homework is assigned but not collected, and I just want to know if I did it right. ThanksQUESTION: Consider the stochastic process \{X_t,\,t=0,1,2,\ldots\} described...
  14. D

    Need Help calculating Entropy & probabilities of a simple Markov Process

    Homework Statement I am in a hurry with the following problem: We have a source that produces binary symbols, 0 and 1. 0 follows a 0 at probability 7/8 1 follows a 1 at probability 1/2 A) Calculate probability of the symbols 0 and 1 to appear. B) Calculate entropy of source. The...
  15. G

    Sojourn time in subset of states for Markov process?

    I have a graph where from each node the state can change randomly from one node to some of the other nodes. My task is to estimate how long the state will stay within a subset of all these nodes. Is there a way to characterize the network with some parameters to find the answer (maybe for a...
  16. S

    Proving that an ODE is a markov process

    I am trying to solve a problem from Van kampens book, page 73. I am trying to self learn stochastic processes for research sake ! A ODE is given: dx/dt = f(x). write the solution with initial values x0 and t0 in the form x = phi(x0, t - t0). Show that x obeys the defintion of the markov...
  17. G

    Proof that a stochastic process isn't a Markov Process

    I've been trying to solve this problem for a week now, but haven't been able to. Basically I need to prove that a certain process satisfies Chapman-Kolmogorov equations, yet it isn't a Markov Process (it doesn't satisfy the Markovian Property). I attached the problem as a .doc below...
Back
Top