Weak convergence
Convergence in distribution
Definition 1 (convergence in distribution). A random sequence \(X:\Omega\to\R^\N\) defined on a probability space \((\Omega, \sF, P)\) converges in distribution to a random variable \(X_\infty:\Omega'\to\R\) defined on a probability space \((\Omega', \sF', P')\) if \(\lim_nF_{X_n}(x) = F_{X_\infty}(x)\) at all continuity points \(x\) of \(F_{X_\infty}\). Convergence in distribution is denoted by \(\lim_nX_n = X_\infty\) in distribution.
Proposition 2. Consider a random sequence\(X:\Omega\to\R^\N\) defined on a probability space \((\Omega, \sF, P)\) and a random variable \(X_\infty: \Omega'\to\R\) defined on another probability space \((\Omega', \sF', P')\). Then the following statements are equivalent.
\(\lim_nX_n = X_\infty\) in distribution.
\(\lim_n\E[g(X_n)] = \E[g(X_\infty)]\) for any bounded continuous Borel measurable function \(g:\R\to\R\).
Characteristic functions converge point-wise, i.e. \(\lim_n\Phi_{X_n}(u) = \Phi_{X_\infty}(u)\) for each \(u \in \R\).
Proof. Proof. Let \(X:\Omega\to\R^\N\) be a sequence of random variables and let \(X_\infty:\Omega'\to\R\) be a random variable. We will show that \((a) \implies (b) \implies (c) \implies (a)\).
Applying the bounded convergence theorem to any bounded continuous Borel measurable function \(g:\R\to\R\), we have \(\lim_n\int_{x \in \R} g(x)dF_{X_n}(x) = \int_{x \in \R} g(x)\lim_ndF_{X_n}(x)\).
Taking \(g(x) = e^{jux}\), we get the result.
The proof of this part is technical and is omitted.
◻
Example 3 (Convergence in distribution but not in probability). Consider a sequence of non-degenerate continuous random variables \(X:\Omega\to\R^\N\) and independent random variable \(Y: \Omega'\to\R\), all with the common distribution \(F_Y\). Then \(F_{X_n} = F_Y\) for all \(n \in \N\), and hence \(\lim_nX_n = Y\) in distribution. If the common distribution \(F_Y\) is zero mean Gaussian with variance \(\sigma^2\), then \(X_n - Y\) is zero mean Gaussian with variance \(2\sigma^2\). Therefore, for \(\epsilon < \sigma\sqrt{\pi}\) and all \(n \in \N\) It follows that \(P\set{\abs{X_n - Y} > \epsilon} \ge 1- \frac{\epsilon}{\sigma\sqrt{\pi}}\) for all \(n \in \N\), and hence \(\lim X_n \neq Y\) in probability.
Lemma 4 (Convergence in probability implies in distribution). Consider a sequence \(X:\Omega\to\R^\N\) of random variables and a random variable \(X_\infty: \Omega\to\R\) defined on a probability space \((\Omega, \sF, P)\), such that \(\lim_nX_n = X_\infty\) in probability, then \(\lim_nX_n = X_\infty\) in distribution.
Proof. Proof. We will show that all continuity points \(x\) of \(F_{X_\infty}\), we have \(\lim_{n\to\infty}F_{X_n}(x) = F_{X_\infty}(x)\). Fix \(\epsilon > 0\). Since \(x\) is a continuity point of non-decreasing function \(F_{X_\infty}\), choose \(\delta > 0\) such that \(F_{X_\infty}(x+\delta) - F_{X_\infty}(x-\delta) < \epsilon\). Therefore, it suffices to show that For the chosen \(\delta >0\), we consider the event \(A_n(\delta) \triangleq \set{\omega \in \Omega: \abs{X_n(\omega)-X_\infty(\omega)} > \delta} = \set{X_n \notin [X_\infty- \delta, X_\infty+\delta]}\in \sF,\) and define events \(A_{X_n}(x) \triangleq \set{X_n \le x}\) and \(A_{X_\infty}(x) \triangleq \set{X_\infty \le x}\). Then, we can write
2 &A_X_n(x)A_X_(x+) A_X_(x+),&&A_X_n(x)A_X_^c(x+) A_n(),
&A_X_(x-)A_X_n(x) A_X_n(x), &&A_X_(x-)A_X_n^c(x) A_n().
From the above set relations, law of total probability, and union bound, we have From the convergence in probability, we have \(\lim_nP(A_n(\delta)) = 0\), and the result follows. ◻
Theorem 5 (Central Limit Theorem). Consider an random sequence \(X:\Omega\to\R^\N\) defined on a probability space \((\Omega, \sF, P)\), with \(\E X_n = \mu\) and \(\Var(X_n) = \sigma^2\) for all \(n\in\N\). We define the \(n\)-sum as \(S_n\triangleq \sum_{i=1}^nX_i\) and consider a standard normal random variable \(Y:\Omega\to\R\) with density function \(f_Y(y)= \frac{1}{\sqrt{2\pi}}e^{-\frac{y^2}{2}}\) for all \(y \in \R\). Then,
Proof. Proof. The classical proof is using the characteristic functions. Let \(Z_i \triangleq \frac{X_i-\mu}{\sigma}\) for all \(i \in \N\), then the shifted and scaled \(n\)-sum is given by \(\frac{S_n-n\mu}{\sigma\sqrt{n}} = \frac{1}{\sqrt{n}}\sum_{i=1}^nZ_i\). We use the third equivalence in Proposition [Prop:ConvDistEquivalence] to show that the characteristic function of converges to the characteristic function of the standard normal. We define the characteristic functions
3 &_n(u) (ju), & &_Z_i(u) (juZ_i), & &_Y(u) (juY).
We can compute the characteristic function of the standard normal as Since the random sequence \(Z: \Omega\to \R^\N\) is a zero mean sequence, it follows that \(\Phi_{Z_1}^{(1)}(0) = j\E Z_1 = 0\) and \(\Phi_{Z_1}^{(2)}(0) = j^2\E Z_1^2 = - 1\). Using the Taylor expansion of the characteristic function \(\Phi_{Z_1}\), we have For any \(u \in \R\), taking limit \(n \in \N\), we get the result. ◻
Strong law of large numbers
Definition 6. For a random sequence \(X:\Omega\to\R^\N\) defined on a probability space \((\Omega,\sF,P)\) with bounded mean \(\E\abs{X_n} < \infty\) for all \(n \in \N\), we define the \(n\)-sum as \(S_n \triangleq \sum_{i=1}^nX_i\) and the empirical \(n\)-mean \(\frac{S_n}{n}\) for each \(n \in \N\). For each \(n \in \N\), we define event
Theorem 7 (\(L^4\) strong law of large numbers). Let \(X:\Omega\to\R^\N\) be a sequence of independent random variables defined on probability space \((\Omega,\sF,P)\) with bounded mean \(\E X_n\) for each \(n \in \N\) and uniformly bounded fourth central moment \(\sup_{n \in \N}\E(X_n-\E X_n)^4 \le B < \infty\). Then, the empirical \(n\)-mean converges to \(\lim_n\frac{\E S_n}{n}\) almost surely.
Proof. Proof. Recall that \(\E(S_n-\E S_n)^4 = \E(\sum_{i=1}^n(X_i-\E X_i))^4 = \sum_{i=1}^n\E(X_i-\E X_i)^4 + 3\sum_{i=1}^n\sum_{j \neq i}\E(X_i-\E X_i)^2\E(X_j-\E X_j)^2\). Recall that when the fourth moment is bounded, then so is second moment. Hence, \(\sup_{i\in \N}\E(X_i-\E X_i)^2 \le C\) for some \(C \in \R_+\). Therefore, from the Markov’s inequality, we have It follows that the \(\sum_{n \in \N}P(E_n) < \infty\), and hence by Borel Canteli Lemma, we have Since, the choice of \(\epsilon\) was arbitrary, the result follows. ◻
Theorem 8 (\(L^2\) strong law of large numbers). Let \(X:\Omega \to \R^\N\) be a sequence of pair-wise uncorrelated random variables defined on a probability space \((\Omega, \sF, P)\) with bounded mean \(\E X_n\) for all \(n \in \N\) and uniformly bounded variance \(\sup_{n\in\N}\Var(X_n) \le B < \infty\). Then, the empirical \(n\)-mean converges to \(\lim_n\frac{\E S_n}{n}\) almost surely.
Proof. Proof. For each \(n \in \N\), we define events \(F_n \triangleq E_{n^2}\), and From the Markov’s inequality and union bound, we have Therefore, \(\sum_{n \in \N}P(F_n) < \infty\) and \(\sum_{n \in \N}P(G_n) < \infty\), and hence by Borel Canteli Lemma, we have The result follows from the fact that for any \(k \in \N\), there exists \(n \in \N\) such that \(k \in \set{n^2, \dots, (n+1)^2-1}\) and hence ◻
Theorem 9 (\(L^1\) strong law of large numbers). Let \(X:\Omega\to\R^\N\) be a random sequence defined on a probability space \((\Omega,\sF,P)\) such that \(\sup_{n\in\N}\E\abs{X_n} \le B < \infty\). Then, the empirical \(n\)-mean converges to \(\lim_n\frac{\E S_n}{n}\) almost surely.