Expectation
Expectation
Example 1. Consider a probability space \((\Omega, \sF, P)\). We consider \(N\) trials of a random experiment, and define a random vector \(X: \Omega \to \sX^N\) such that \(X_i \triangleq \pi_i\circ X :\Omega \to\sX\) is a discrete random variable associated with the trial \(i \in [N]\). If the marginal distributions of random variables \((X_1, \dots, X_N)\) are identical with the common probability mass function \(P_{X_1}: \sX \to [0,1]\), then the empirical mean of random variable \(X_1\) can be written as For a random variable \(X_1: \Omega \to \sX\), we can define events \(E_{X_1}(x) \triangleq X_1^{-1}\set{x}\) for each value \(x \in \sX\). The probability mass function \(P_{X_1}:\sX \to [0,1]\) for the discrete random variable \(X_1\) can be estimated for each \(x \in \sX\) as the empirical probability mass function
Recall that a simple random variable \(X_1\) can be written as \(X_1 = \sum_{x \in \sX}x\Ind{E_{X_1}(x)}\), where \(E_{X_1} \triangleq (E_{X_1}(x) \in \sF: x \in \sX)\) is a finite partition of the sample space \(\Omega\) and \(P_{X_1}(x) = P(E_{X_1}(x))\). That is, we can write the empirical mean in terms of the empirical PMF as
This example motivates the following definition of mean for simple random variables.
Definition 2 (Expectation of simple random variable). The mean or expectation of a simple random variable \(X:\Omega \to \sX \subseteq \R\) defined on a probability space \((\Omega, \sF, P)\), is denoted by \(\E[X]\) and defined as
Remark 1. For an indicator random variable \(\Ind{A}\), we have \(\E\Ind{A} = P(A)\). That is, the expectation of an indicator function is the probability of the indicated set.
Remark 2. Since a simple random variable can be written as \(X = \sum_{x \in \sX}x\Ind{E_X(x)}\) where \(E_X(x) \triangleq X^{-1}\set{x}\) for all \(x\in\sX\), we can write the expectation of a simple random variable as an integral
Theorem 3. Consider a non-negative random variable \(X: \Omega \to \R_+\) defined on a probability space \((\Omega, \sF, P)\). There exists a sequence of non-decreasing non-negative simple random variables \(Y: \Omega \to \R_+^\N\) such that for all \(\omega \in \Omega\) Then \(\E[Y_n]\) is defined for each \(n \in \N\), the sequence \((\E[Y_n] \in \R_+: n \in \N)\) is non-decreasing, and the limit \(\lim_n\E[Y_n] \in \R_+\cup\set{\infty}\) exists. This limit is independent of the choice of the sequence and depends only on the probability space.
Proof. Proof. For each \(n \in \N\) and \(k \in \set{0, \dots, 2^{2n}-1}\), we define half-open sets \(B_{n,k} \triangleq (k2^{-n}, (k+1)2^{-n}]\). Then, the collection of sets \(B_n \triangleq (B_{n,k}: k \in \set{0, \dots, 2^{2n}-1})\) partitions the set \((0, 2^n]\) for each \(n \in \N\). Further, we observe that \(\cup_{n \in \N}(0, 2^n] = \R^+\) and that \(B_{n+1,2k}\cup B_{n+1,2k+1} = B_{n,k}\) for all \(n \in \N\) and \(k\).
For a non-negative random variable \(X: \Omega \to \R_+\), we define events \(A^X_{n,k} = X^{-1}(B_{n,k}) \in \sF\), and a sequence of simple non-negative random variables \(Y: \Omega\to\R_+^\N\) in the following fashion We observe that \(Y_n\) is a quantized version of \(X\), and its value is the left end-point \(k2^{-n}\) when \(X \in B_{n,k}\) for each \(k \in \set{0, \dots, 2^{2n}-1}\). Since \(\cup_{k=0}^{2^{2n}-1}A^X_{n,k} = X^{-1}(0, 2^n]\), it follows that we cover the positive real line as \(n\) grows larger and the step size grows smaller. Thus, the limiting random variable can take all possible non-negative real values. We observe that We see that \(Y_n(\omega) \le Y_{n+1}(\omega) \le X(\omega)\) and \(\lim_nY_n(\omega) = X(\omega)\) for all \(\omega \in \Omega\).
Since \(Y_n: \Omega \to \R\) is a simple random variable for all \(n \in \N\), the expectation \(\E[Y_n]\) is defined for all \(n\), and can be written as We observe that this expectation is completely specified by the distribution function \(F_X\), and we can write the limit ◻
Definition 4 (Expectation of a non-negative random variable). For a non-negative random variable \(X: \Omega \to \R\) defined on the probability space \((\Omega,\sF,P)\), consider the sequence of non-decreasing simple random variables \(Y: \Omega \to \R_+^\N\) such that \(\lim_nY_n = X\). The expectation of the non-negative random variable \(X\) is defined as
Remark 3. From the definition, it follows that \(\E[X] = \int_{\R_+} x dF_X(x)\).
Definition 5 (Expectation of a real random variable). For a real-valued random variable \(X\) defined on a probability space \((\Omega, \sF, P)\), we can define the following functions
2 &X_+ , & &X_- .
We can verify that \(X_+, X_-\) are non-negative random variables and hence their expectations are well defined. We observe that \(X(\omega) = X_+(\omega) - X_-(\omega)\) for each \(\omega \in \Omega\). If at least one of the \(\E[X_+]\) and \(\E[X_-]\) is finite, then the expectation of the random variable \(X\) is defined as
Theorem 6 (Expectation as an integral with respect to the distribution function). For a random variable \(X:\Omega\to\R\) defined on the probability space \((\Omega, \sF, P)\), the expectation is given by
Proof. Proof. It suffices to show this for a non-negative random variable \(X\), and the result follows from the definition of expectation of a non-negative random variable as the limit of expectation of approximating simple functions. ◻
Properties of Expectations
Theorem 7 (Properties). Let \(X: \Omega \to \R\) be a random variable defined on the probability space \((\Omega, \sF, P)\).
Linearity: Let \(a, b \in \R\) and \(X, Y\) be random variables defined on the probability space \((\Omega, \sF, P)\). If \(\E X, \E Y\), and \(a\E X + b\E Y\) are well defined, then \(\E(aX+bY)\) is well defined and
Monotonicity: If \(P\set{X \ge Y} = 1\) and \(\E[Y]\) is well defined with \(\E[Y] > -\infty\), then \(\E[X]\) is well defined and \(\E[X] \ge \E[Y]\).
Functions of random variables: Let \(g: \R \to \R\) be a Borel measurable function, then \(g(X)\) is a random variable with \(\E[g(X)] = \int_{x\in \R}g(x)dF(x)\).
Continuous random variables: Let \(f_X: \R \to [0, \infty)\) be the density function, then \(\E X = \int_{x \in \R}xf_X(x)dx\).
Discrete random variables: Let \(P_X: \sX \to [0,1]\) be the probability mass function, then \(\E X = \sum_{x \in \sX}xP_X(x)\).
Integration by parts: The expectation \(\E X = \int_{x \ge 0}(1-F_X(x))dx - \int_{x < 0}F_X(x)dx\) is well defined when at least one of the two parts is finite on the right hand side.
Proof. Proof. It suffices to show properties \((i)-(iii)\) for simple random variables.
Let \(X = \sum_{x \in \sX}x\mathbbm{1}_{E_X(x)}\) and \(Y = \sum_{y \in \sY}y\mathbbm{1}_{E_Y(y)}\) be simple random variables, then \((E_X(x)\cap E_Y(y) \in \sF: (x, y)\in \sX \times \sY)\) partition the sample space \(\Omega\). Hence, we can write \(aX + bY = \sum_{(x,y) \in \sX \times \sY}(ax+by)\SetIn{E_X(x)\cap E_Y(y)}\) and from linearity of sum it follows that
From the fact that \(X-Y \ge 0\) almost surely and linearity of expectation, it suffices to show that \(\E X \ge 0\) for non-negative random variable \(X\). It can easily be shown for simple non-negative random variables, and follows for general non-negative random variables by taking limits.
It suffices to show this holds true for simple random variables \(X: \Omega \to \sX \subset \R\). Since \(g: \R \to \R\) is Borel measurable, \(Y \triangleq g(X): \Omega\to\sY \triangleq g(\sX)\) is a random variable. It follows that, we can write the following disjoint union \(\sX = \cup_{y\in \sY}\cup_{x \in g^{-1}\set{y}}\set{x}\). Further, for each \(y \in \sY\), we have Since \(E_X(x)\) are disjoint for all \(x \in \sX\), we get \(P(E_Y(y)) = \sum_{x \in g^{-1}\set{y}}P(E_X(x))\). Using the above two facts, we can write the expectation
For continuous random variables, we have \(dF_X(x) = f_X(x)dx\) for all \(x \in \R\).
For discrete random variables \(X : \Omega \to \sX\), we have \(dF_X(x) = P_X(x)\) for all \(x \in \sX\) and zero otherwise.
We can write \(\E X = \int_{x <0}xdF_X(x)-\int_{x \ge 0}xd(1-F_X)(x)\). We apply integration by parts to the first term on the right, to get Similarly, we apply integration by parts to the second term on the right, to get
◻