Recurrent and transient states

Author

Parimal Parag

Updated

July 16, 2026

Recurrence and Transience

We will consider a random sequence X:\Omega\to\sX^{\Z_+} with initial state X_0 = x\in\sX and the kth hitting times to state y for all k \in \N denoted by \tau_k \triangleq \tau_X^{\set{y},k} inductively defined as \tau_k \triangleq \inf\set{n > \tau_{k-1}: X_n = y} where \tau_0 \triangleq 0. We define the inter-return time sequence H:\Omega\to\N^\N as H_k \triangleq H_X^{\set{y},k} = \tau_k - \tau_{k-1} for all k \in \N.

Definition 1. For a random sequence X: \Omega \to \sX^{\Z_+} with initial state X_0 = x,

the probability of hitting state y eventually is denoted by f_{xy} \triangleq P_x\set{\tau_1 < \infty}, and

the probability of first visit to state y at time n \in\N is denoted by f_{xy}^{(n)} \triangleq P_x\set{\tau_1 = n}. %,\quad n \in \N.

Remark 1. We can write the finiteness of hitting time \tau_1 as the disjoint union \set{\tau_1 < \infty} = \cup_{n \in \N}\set{\tau_1=n}. Therefore, f_{xy} = \sum_{n\in\N}f_{xy}^{(n)}.

Remark 2. If f_{xy} = P_x\set{\tau_1 < \infty} = 1 for all initial states x \in \sX, then \tau_1 is almost surely finite and hence a stopping time.

Definition 2. From the initial state x, the distribution

for the first hitting time to state y is called the first passage time distribution and denoted by ((f_{xy}^{(n)}: n \in \N), 1 - f_{xy}), and

for the first return time to state x is called the first recurrence time distribution and denoted by ((f_{xx}^{(n)}: n \in \N),1 - f_{xx}).

Definition 3. A state y\in\sX is called recurrent if f_{yy} = 1, and is called transient if f_{yy} < 1.

Definition 4. For any state y \in \sX, the mean recurrence time is denoted by \mu_{yy} \triangleq \E_y\tau_1.

Remark 3. The mean recurrence time for any transient state is infinite. For any recurrent state y \in \sX, we write \tau_1 = \tau_1\SetIn{\tau_1 < \infty} = \sum_{n \in \N}n\SetIn{\tau_1=n} almost surely, and the mean recurrence time is given by \mu_{yy} = \sum_{n \in \N}nf^{(n)}_{yy}.

Definition 5. For a recurrent state y \in \sX,

if the mean recurrence time is finite, then the state y is called positive recurrent, and

if the mean recurrence time is infinite, then the state y is called null recurrent.

Proposition 6. For a homogeneous discrete Markov chain X: \Omega \to \sX^{\Z_+}, we have

Proof. Proof. We can write the event of zero visits to state y as \set{N_y(\infty) = 0} = \set{\tau_1 = \infty}. Further, we can write the event of m visits to state y as Recall that H:\Omega\to\N^\N is an independent random sequence with subsequence (H_k: k \ge 2) identically distributed, with P_x\set{H_j = n} = P_y\set{\tau_1 =n} for all j \ge 2. Therefore, we get ◻

Corollary 7. For a homogeneous Markov chain X, we have P_x\set{N_y(\infty) < \infty} = \SetIn{f_{yy} < 1} + (1-f_{xy})\SetIn{f_{yy} = 1}.

Proof. Proof. We can write the event \set{N_y(\infty) < \infty} as disjoint union of events \set{N_y(\infty) = k}, to get the result. ◻

Remark 4. For a time homogeneous Markov chain X: \Omega \to \sX^{\Z_+}, we have

P_x\set{N_y(\infty) = \infty} = f_{xy}\SetIn{f_{yy}=1}, and

P_y\set{N_y(\infty) = \infty} = \SetIn{f_{yy}=1}.

Corollary 8. The mean number of visits to state y, starting from a state x is \E_xN_y(\infty) = \frac{f_{xy}}{1-f_{yy}}\SetIn{f_{yy} < 1} + \infty\SetIn{f_{xy} > 0, f_{yy} = 1}.

Remark 5. For any state y \in \sX, we have \E_yN_y(\infty) = \frac{f_{yy}}{1-f_{yy}}\SetIn{f_{yy}<1} + \infty\SetIn{f_{yy}=1}. That is, the mean number of visits to initial state y is finite iff the state y is transient.

Remark 6. In particular, this corollary implies the following consequences.

  1. A transient state is visited a finite amount of times almost surely. This follows from Corollary [cor:01law], since P_x\set{N_y(\infty) < \infty} = 1 for all transient states y \in \sX and any initial state x \in \sX.

  2. A recurrent state is visited infinitely often almost surely. This also follows from Corollary [cor:01law], since P_y\set{N_y(\infty) < \infty} = 0 for all recurrent states y \in \sX.

  3. In a finite state Markov chain, not all states may be transient.

    Proof. Proof. To see this, we assume that for a finite state space \sX, all states y \in \sX are transient. Then, we know that N_y(\infty) is finite almost surely for all states y \in \sX. It follows that, for any initial state x \in \sX It follows that \sum_{y \in \sX}N_y(\infty) is also finite almost surely for all states y \in \sX for finite state space \sX. However, we know that \sum_{y \in \sX}N_y(\infty) = \sum_{k \in \N}\sum_{y \in \sX}\indicator{X_k = y} = \infty. This leads to a contradiction. ◻

Proposition 9. For a homogeneous DTMC X: \Omega \to \sX^{\Z_+}, a state y\in\sX is recurrent iff \sum_{k\in\N} p_{yy}^{(k)} = \infty, and transient iff \sum_{k\in\N} p_{yy}^{(k)} < \infty.

Proof. Proof. Recall that if the mean recurrence time to a state y\in\sX is \E_yN_y(\infty) = \sum_{k\in\N}p_{yy}^{(k)} finite then the state is transient and infinite if the state is recurrent. ◻

Corollary 10. For a transient state y \in \sX, the following limits hold \lim_{n \to \infty}p_{xy}^{(n)} = 0, and \lim_{n \to \infty}\frac{\sum_{k = 1}^np_{xy}^{(k)}}{n} = 0.

Proof. Proof. For a transient state y \in \sX and any state x \in \sX, we have \E_xN_y(\infty) = \sum_{n \in \N}p_{xy}^{(n)} < \infty. Since the series sum is finite, it implies that the limiting terms in the sequence \lim_{n \to \infty}p_{xy}^{(n)} = 0. Further, we can write \sum_{k=1}^np_{xy}^{(k)} \le \E_xN_y(\infty) \le M for some M \in \N and hence \lim_{n \to \infty}\frac{\sum_{k = 1}^np_{xy}^{(k)}}{n} = 0. ◻

Lemma 11. For any state y \in \sX, let H:\Omega\to\N^\N be the sequence of almost surely finite inter-visit times to state y, and N_y(n) = \sum_{k=1}^n\indicator{X_k = y} be the number of visits to state y in n times. Then, N_y(n) +1 is a finite mean stopping time with respect to the sequence H.

Proof. Proof. We first observe that N_y(n) + 1 \le n+1 and hence has a finite mean for each n \in \N. Further, we observe that \set{N_y(n) + 1 = k} can be completely determined by observing H_1, \dots, H_k, since ◻

Theorem 12. Let x, y \in \sX be such that f_{xy} = 1 and y is recurrent. Then, \lim_{n \to \infty}\frac{\sum_{k=1}^np_{xy}^{(k)}}{n} = \frac{1}{\mu_{yy}}.

Proof. Proof. Let y \in \sX be recurrent. The proof consists of three parts. In the first two parts, we will show that starting from the state y, we have the limiting empirical average of mean number of visits to state y is \lim_{n \to \infty}\frac{1}{n}\E_yN_y(n) = \frac{1}{\mu_{yy}}. In the third part, we will show that for any starting state x \in \sX such that f_{xy}=1, we have the limiting empirical average of mean number of visits to state y is \lim_{n \to \infty}\frac{1}{n}\E_xN_y(n) = \frac{1}{\mu_{yy}}.

  • We observe that N_y(n)+1 is a stopping time with respect to inter-visit times H from Lemma [lem:StoppingTime]. Further, we have \sum_{j=1}^{N_y(n)+1}H_j > n. Applying Wald’s Lemma to the random sum \sum_{j=1}^{N_y(n)+1}H_j , we get \E_y(N_y(n)+1)\mu_{yy} > n. Taking limits, we obtain \lim\inf_{n \in \N}\frac{\sum_{k=1}^np_{yy}^{(k)}}{n} \ge \frac{1}{\mu_{yy}}.

  • Let X_0 = y and consider a fixed positive integer M\in \N. Then H is and we define truncated recurrence times \bar{H}:\Omega\to[M]^\N for all j \in \N as \bar{H}_j \triangleq M \wedge H_j. It follows that the sequence \bar{H} is and \bar{H}_j \le H_j for all j \in \N. We define the mean of the truncated recurrence times as \bar{\mu}_{yy} \triangleq \E_y\bar{H}_1. From the monotonicity of truncation, we get \bar{\mu}_{yy} \le \mu_{yy}.

    We define the random variable \bar{\tau}_k \triangleq \sum_{j=1}^k\bar{H}_j for all k \in \N, and \bar{\tau}_k \le \tau_k for all k \in \N. We can define the associated counting process that counts the number of truncated recurrences in first n steps as \bar{N}_y(n) \triangleq \sum_{k\in \N}\SetIn{\bar{\tau}_k \le n} for all n \in \N. We conclude that \bar{N}_y(n)+1 is a stopping time with respect to process \bar{H}, and \bar{N}_y(n) \ge N_y(n) sample path wise. Further, we have Applying Wald’s Lemma to stopping time N_y(n)+1 with respect to sequence H and stopping time \bar{N}_y(n)+1 with respect to sequence \bar{H}, and monotonicity of expectation, we get Taking limits, we obtain \lim\sup_{n \in \N}\frac{\sum_{k=1}^np_{yy}^{(k)}}{n} \le \frac{1}{\bar{\mu}_{yy}}. Letting M grow arbitrarily large, we obtain the upper bound.

  • Further, we observe that p_{xy}^{(k)} = \sum_{s=0}^{k-1}f_{xy}^{(k-s)}p_{yy}^{(s)}. Since 1 = f_{xy} = \sum_{k \in \N}f_{xy}^{(k)}, we have Since the series \sum_{k \in \N}f_{xy}^{(k)} converges, we get \lim_{n \to \infty}\frac{\sum_{k=1}^np_{xy}^{(k)}}{n} = \lim_{n \to \infty}\frac{\sum_{k=1}^np_{yy}^{(k)}}{n}.

 ◻