Let $(\Omega,\mathcal F,\mathbb P)$ be a [probability space](/page/Probability%20Space), let $S$ be a [countable set](/page/Countable%20Set) equipped with the discrete $\sigma$-algebra $\mathcal P(S)$, and let $X_n:(\Omega,\mathcal F)\to(S,\mathcal P(S))$ be random variables for $n\ge 0$. Suppose $(X_n)_{n\ge 0}$ is a time-homogeneous [Markov chain](/page/Markov%20Chain) with transition matrix $P=(p_{ij})_{i,j\in S}$, meaning that $p_{ij}\ge 0$, $\sum_{j\in S}p_{ij}=1$ for every $i\in S$, and for every $m\ge 0$ and every $i_0,\ldots,i_m,j\in S$ with $\mathbb P(X_0=i_0,\ldots,X_m=i_m)>0$,\n\begin{align*}\n\mathbb P(X_{m+1}=j\mid X_0=i_0,\ldots,X_m=i_m)=p_{i_mj}.\n\end{align*}\nLet $\lambda=(\lambda_i)_{i\in S}$ be the initial distribution, so that $\lambda_i=\mathbb P(X_0=i)$ for every $i\in S$. Then, for every $n\ge 0$ and every $i_0,\ldots,i_n\in S$,\n\begin{align*}\n\mathbb P(X_0=i_0, X_1=i_1,\ldots,X_n=i_n)=\lambda_{i_0}\prod_{r=0}^{n-1}p_{i_ri_{r+1}},\n\end{align*}\nwhere the product is interpreted as the empty product $1$ when $n=0$.