[proofplan]
We prove the four properties of the return-time measure $\rho_i = \mathbb{E}_k[W_i]$ in sequence. Part (1) follows directly from the definition of the excursion. Part (2) sums the visit indicators to recover the return time, then interchanges sum and expectation via monotone convergence. Part (3) conditions on the state at time $m - 1$ and applies the Markov property, then re-indexes to identify the inner sum with $\rho_i$, treating the cases $i = k$ and $i \neqeq k$ separately. Part (4) uses the invariance $\rho = \rho P^n$ together with irreducibility to sandwich $\rho_i$ between two positive finite bounds.
[/proofplan]
[step:Verify $\rho_k = 1$: the base state is visited exactly once per excursion]
The excursion from $k$ runs from time $1$ to time $T_k = \min\{n \geq 1 : X_n = k\}$. The state $k$ is visited during the interval $\{1, \ldots, T_k\}$ exactly at time $T_k$ (since $T_k$ is the first time after $0$ that the chain returns to $k$, so $X_m \neqeq k$ for $1 \leq m \leq T_k - 1$ and $X_{T_k} = k$). Therefore
\begin{align*}
W_k = \sum_{m=1}^{T_k} \mathbb{1}_{X_m = k} = 1 \quad \text{$\mathbb{P}_k$-almost surely.}
\end{align*}
Taking expectations: $\rho_k = \mathbb{E}_k[W_k] = 1$.
[/step]
[step:Compute $\sum_{i \in S} \rho_i = \mu_k$ by summing visit indicators]
At each time $m \in \{1, \ldots, T_k\}$, the chain occupies exactly one state, so $\sum_{i \in S} \mathbb{1}_{X_m = i} = 1$. Summing over time:
\begin{align*}
\sum_{i \in S} W_i = \sum_{i \in S} \sum_{m=1}^{T_k} \mathbb{1}_{X_m = i} = \sum_{m=1}^{T_k} \sum_{i \in S} \mathbb{1}_{X_m = i} = \sum_{m=1}^{T_k} 1 = T_k.
\end{align*}
The interchange of the two sums is justified because all terms $\mathbb{1}_{X_m = i}$ are non-negative. Taking expectations and applying the [monotone convergence theorem](/theorems/???) to interchange $\mathbb{E}_k$ with $\sum_{i \in S}$ (valid since $W_i \geq 0$):
\begin{align*}
\sum_{i \in S} \rho_i = \sum_{i \in S} \mathbb{E}_k[W_i] = \mathbb{E}_k\!\left[\sum_{i \in S} W_i\right] = \mathbb{E}_k[T_k] = \mu_k.
\end{align*}
[guided]
The idea is simple: at each time step, the chain is at exactly one state, so the total number of visits to all states equals the total number of steps. The only subtlety is justifying the interchange of expectation and summation over the (potentially countably infinite) state space $S$.
At each time $m \in \{1, \ldots, T_k\}$, the indicators $\{\mathbb{1}_{X_m = i} : i \in S\}$ form a partition of unity: $\sum_{i \in S} \mathbb{1}_{X_m = i} = 1$. Summing over time and states:
\begin{align*}
\sum_{i \in S} W_i = \sum_{i \in S} \sum_{m=1}^{T_k} \mathbb{1}_{X_m = i} = \sum_{m=1}^{T_k} 1 = T_k.
\end{align*}
Now take expectations. Since $W_i \geq 0$ for all $i$, the sequence of partial sums $S_N = \sum_{i \in F_N} W_i$ (for any exhaustion $F_1 \subset F_2 \subset \cdots$ with $\bigcup F_N = S$) is increasing and converges to $T_k$. By the monotone convergence theorem:
\begin{align*}
\sum_{i \in S} \rho_i = \sum_{i \in S} \mathbb{E}_k[W_i] = \mathbb{E}_k\!\left[\sum_{i \in S} W_i\right] = \mathbb{E}_k[T_k] = \mu_k.
\end{align*}
[/guided]
[/step]
[step:Prove the invariance relation $\rho = \rho P$ by conditioning on the previous state]
We must show $\rho_j = \sum_{i \in S} p_{i,j} \, \rho_i$ for every $j \in S$. Expanding the definition:
\begin{align*}
\rho_j = \mathbb{E}_k[W_j] = \sum_{m=1}^{\infty} \mathbb{P}_k(X_m = j,\, T_k \geq m),
\end{align*}
where the upper limit is extended to $\infty$ since $\mathbb{P}_k(X_m = j, T_k \geq m) = 0$ for $m > T_k$.
Condition on the state $X_{m-1}$ at the previous time step. The event $\{T_k \geq m\}$ means $X_1, \ldots, X_{m-1}$ are all different from $k$, which is determined by the trajectory up to time $m-1$. By the Markov property, $\mathbb{P}_k(X_m = j \mid X_{m-1} = i, T_k \geq m) = p_{i,j}$, since $\{T_k \geq m\}$ depends only on $(X_1, \ldots, X_{m-1})$. Therefore:
\begin{align*}
\rho_j &= \sum_{m=1}^{\infty} \sum_{i \in S} \mathbb{P}_k(X_m = j \mid X_{m-1} = i) \, \mathbb{P}_k(X_{m-1} = i,\, T_k \geq m) \\
&= \sum_{i \in S} p_{i,j} \sum_{m=1}^{\infty} \mathbb{P}_k(X_{m-1} = i,\, T_k \geq m).
\end{align*}
Re-index by $r = m - 1$:
\begin{align*}
\rho_j = \sum_{i \in S} p_{i,j} \sum_{r=0}^{\infty} \mathbb{P}_k(X_r = i,\, T_k \geq r + 1).
\end{align*}
[claim:The inner sum equals $\rho_i$]
For every $i \in S$,
\begin{align*}
\sum_{r=0}^{\infty} \mathbb{P}_k(X_r = i,\, T_k \geq r + 1) = \rho_i.
\end{align*}
[/claim]
[proof]
We handle two cases.
**Case $i \neqeq k$.** For $i \neqeq k$ and any $r \geq 1$, the event $\{X_r = i\}$ (with $i \neqeq k$) implies $X_r \neqeq k$, so $T_k \neqeq r$. Therefore $\{T_k \geq r\} = \{T_k \geq r+1\} \cup \{T_k = r\}$ simplifies to $\{T_k \geq r\} = \{T_k \geq r+1\}$ on the event $\{X_r = i\}$. Hence
\begin{align*}
\mathbb{P}_k(X_r = i, T_k \geq r+1) = \mathbb{P}_k(X_r = i, T_k \geq r) \quad \text{for } r \geq 1.
\end{align*}
For $r = 0$: $\mathbb{P}_k(X_0 = i, T_k \geq 1) = \mathbb{P}_k(X_0 = i) = 0$ since $X_0 = k \neqeq i$. Also, $\mathbb{P}_k(X_0 = i, T_k \geq 0) = \mathbb{P}_k(X_0 = i) = 0$. So
\begin{align*}
\sum_{r=0}^{\infty} \mathbb{P}_k(X_r = i, T_k \geq r+1) = \sum_{r=0}^{\infty} \mathbb{P}_k(X_r = i, T_k \geq r) = \sum_{r=1}^{\infty} \mathbb{P}_k(X_r = i, T_k \geq r) = \rho_i.
\end{align*}
**Case $i = k$.** The $r = 0$ term is $\mathbb{P}_k(X_0 = k, T_k \geq 1) = 1$ (since $X_0 = k$ under $\mathbb{P}_k$, and $T_k \geq 1$ by definition). For $r \geq 1$: $\{X_r = k, T_k \geq r+1\}$ requires the chain to be at $k$ at time $r$ but to have first returned to $k$ after time $r$, which means $T_k > r$; but $X_r = k$ implies $T_k \leq r$. This is a contradiction, so $\mathbb{P}_k(X_r = k, T_k \geq r+1) = 0$ for all $r \geq 1$. Hence
\begin{align*}
\sum_{r=0}^{\infty} \mathbb{P}_k(X_r = k, T_k \geq r+1) = 1 = \rho_k.
\end{align*}
[/proof]
Substituting the claim into the expression for $\rho_j$:
\begin{align*}
\rho_j = \sum_{i \in S} p_{i,j} \, \rho_i,
\end{align*}
which is exactly $\rho = \rho P$.
[guided]
The key idea is to relate $\rho_j$ (expected visits to $j$ during the excursion) to $\rho_i$ (expected visits to $i$) through the transition probabilities $p_{i,j}$. Intuitively: the chain can visit $j$ at time $m$ only by being at some state $i$ at time $m-1$ and transitioning to $j$. So visits to $j$ are "fed" by visits to $i$ through the transition $i \to j$.
We start from
\begin{align*}
\rho_j = \sum_{m=1}^{\infty} \mathbb{P}_k(X_m = j,\, T_k \geq m).
\end{align*}
Why can we condition on $X_{m-1}$ and apply the Markov property? The event $\{T_k \geq m\}$ means $X_r \neqeq k$ for $1 \leq r \leq m-1$, which depends only on $(X_1, \ldots, X_{m-1})$. The Markov property then gives $\mathbb{P}_k(X_m = j \mid X_{m-1} = i, T_k \geq m) = \mathbb{P}_k(X_m = j \mid X_{m-1} = i) = p_{i,j}$. Conditioning:
\begin{align*}
\rho_j = \sum_{i \in S} p_{i,j} \sum_{m=1}^{\infty} \mathbb{P}_k(X_{m-1} = i,\, T_k \geq m).
\end{align*}
After the re-index $r = m-1$, the inner sum becomes $\sum_{r=0}^{\infty} \mathbb{P}_k(X_r = i, T_k \geq r+1)$. The delicate point is showing this equals $\rho_i$. For $i \neqeq k$, the condition $T_k \geq r+1$ is redundant on $\{X_r = i\}$ (since $i \neqeq k$ means the chain has not returned to $k$ at time $r$, so $T_k \geq r$ is equivalent to $T_k \geq r+1$). The $r = 0$ term vanishes since $X_0 = k \neqeq i$. So the sum reduces to $\rho_i$.
For $i = k$, only the $r = 0$ term survives (it contributes $1$), and all $r \geq 1$ terms vanish because being at $k$ at time $r \geq 1$ means $T_k \leq r$, which contradicts $T_k \geq r+1$. This gives $\rho_k = 1$, consistent with part (1).
Therefore $\rho_j = \sum_{i \in S} p_{i,j} \rho_i$, i.e., $\rho = \rho P$.
[/guided]
[/step]
[step:Bound $\rho_i$ away from $0$ and $\infty$ using irreducibility]
Since $\rho = \rho P$, induction gives $\rho = \rho P^n$ for all $n \geq 1$, i.e., $\rho_j = \sum_{i \in S} p_{i,j}(n) \, \rho_i$ for all $n$ and $j$. In particular, extracting a single term from the sum:
\begin{align*}
\rho_i \geq \rho_k \, p_{k,i}(n) \quad \text{and} \quad \rho_k \geq \rho_i \, p_{i,k}(m).
\end{align*}
By irreducibility, choose $n \geq 1$ with $p_{k,i}(n) > 0$ and $m \geq 1$ with $p_{i,k}(m) > 0$. Since $\rho_k = 1$:
\begin{align*}
0 < p_{k,i}(n) \leq \rho_i \quad \text{and} \quad \rho_i \leq \frac{\rho_k}{p_{i,k}(m)} = \frac{1}{p_{i,k}(m)} < \infty.
\end{align*}
Hence $0 < \rho_i < \infty$ for every $i \in S$.
[guided]
The invariance relation $\rho = \rho P$ is a powerful structural fact: it means $\rho$ is an invariant measure (not necessarily normalised to $1$). To show $\rho_i$ is strictly positive and finite, we use the same technique that proved all entries of an invariant distribution are positive.
From $\rho = \rho P^n$, extracting the $k$-th term:
\begin{align*}
\rho_i = \sum_{j \in S} \rho_j \, p_{j,i}(n) \geq \rho_k \, p_{k,i}(n) = p_{k,i}(n),
\end{align*}
since $\rho_k = 1$. By irreducibility, $k \to i$, so $p_{k,i}(n) > 0$ for some $n$, giving $\rho_i > 0$.
For the upper bound, extract the $i$-th term from the sum for $\rho_k$:
\begin{align*}
1 = \rho_k = \sum_{j \in S} \rho_j \, p_{j,k}(m) \geq \rho_i \, p_{i,k}(m).
\end{align*}
By irreducibility, $i \to k$, so $p_{i,k}(m) > 0$ for some $m$, giving $\rho_i \leq 1/p_{i,k}(m) < \infty$.
Combining: $0 < p_{k,i}(n) \leq \rho_i \leq 1/p_{i,k}(m) < \infty$. This completes the proof.
[/guided]
[/step]
