Correlation

Correlation

Let \(X:\Omega\to\R\) and \(Y:\Omega\to\R\) be random variables defined on the same probability space \((\Omega,\sF,P)\).

Exercise 1. Show that the function \(g: \R^2 \to \R\) defined by \(g: (x,y) \mapsto xy\) is a Borel measurable function.

Definition 2 (Correlation). For two random variables \(X,Y\) defined on the same probability space, the correlation between these two random variables is defined as \(\E[XY]\). If \(\E[XY] = \E[X]\E[Y]\), then the random variables \(X,Y\) are called uncorrelated.

Lemma 3. If \(X, Y\) are independent random variables, then they are uncorrelated.

Proof. Proof. It suffices to show for \(X,Y\) simple and independent random variables. We can write \(X = \sum_{x\in \sX}x\Ind{A_X(x)}\) and \(Y = \sum_{y\in \sY}y\Ind{A_Y(y)}\). Therefore, ◻

Proof. Proof. If \(X, Y\) are independent random variables, then the joint distribution \(F_{X,Y}(x,y) = F_X(x)F_Y(y)\) for all \((x,y) \in \R^2\). Therefore, ◻

Example 4 (Uncorrelated dependent random variables). Let \(X: \Omega \to \R\) be a continuous random variable with even density function \(f_X:\R \to \R_+\), and \(g: \R \to \R_+\) be another even function that is increasing for \(y \in \R_+\). Then \(g\) is Borel measurable function and \(Y = g(X)\) is a random variable. Further, we can verify that \(X, Y\) are uncorrelated and dependent random variables.

To show dependence of \(X\) and \(Y\), we take positive \(x,y\) such that \(F_X(x) < 1\) and \(x > x_y\) where \(\set{x_y} = g^{-1}(y)\cap \R_+\). Then, we can write the set Hence, we can write the joint distribution at \((x,y)\) as

Since \(X\) has even density function, we have \(f_X(x) = f_X(-x)\) for all \(x \in \R\). Therefore, we have The last equality follows from the fact that \(g\) and \(f_X\) are even. Therefore, we have

Theorem 5 (AM greater than GM). For any two random variables \(X,Y\), the correlation is upper bounded by the average of the second moments, with equality iff \(X=Y\) almost surely. That is,

Proof. Proof. This follows from the linearity and monotonicity of expectations and the fact that \((X-Y)^2 \ge 0\) with equality iff \(X = Y\). ◻

Theorem 6 (Cauchy-Schwarz inequality). For any two random variables \(X,Y\), the correlation of absolute values of \(X\) and \(Y\) is upper bounded by the square root of product of second moments, with equality iff \(X = \alpha Y\) for any constant \(\alpha \in \R\). That is,

Proof. Proof. For two random variables \(X\) and \(Y\), we can define normalized random variables \(W \triangleq \frac{\abs{X}}{\sqrt{\E X^2}}\) and \(Z \triangleq \frac{\abs{Y}}{\sqrt{\E Y^2}}\), to get the result. ◻

Covariance

Definition 7 (Covariance). For two random variables \(X,Y\) defined on the same probability space, the covariance between these two random variables is defined as \(\cov(X,Y) \triangleq \E(X-\E X)(Y-\E Y)\).

Lemma 8. If the random variables \(X,Y\) are uncorrelated, then the covariance is zero.

Proof. Proof. We can write the covariance of uncorrelated random variables \(X, Y\) as ◻

Lemma 9. Let \(X: \Omega \to \R^n\) be an uncorrelated random vector and \(a = (a_1, \dots, a_n) \in \R^n\), then

Proof. Proof. From the linearity of expectation, we can write the variance of the linear combination as ◻

Definition 10 (Correlation coefficient). The ratio of covariance of two random variables \(X,Y\) and the square root of product of their variances is called the correlation coefficient and denoted by

Theorem 11 (Correlation coefficient). For any two random variables \(X,Y\), the absolute value of correlation coefficient is less than or equal to unity, with equality iff \(X = \alpha Y + \beta\) almost surely for constants \(\alpha = \sqrt{\frac{\Var(X)}{\Var(Y)}}\) and \(\beta = \E X - \alpha \E Y\).

Proof. Proof. For two random variables \(X\) and \(Y\), we can define normalized random variables \(W \triangleq \frac{X-\E X}{\sqrt{\Var(X)}}\) and \(Z \triangleq \frac{Y-\E Y}{\sqrt{\Var(Y)}}\). Applying the AM-GM inequality to random variables \(W, Z\), we get Recall that equality is achieved iff \(W = Z\) almost surely or equivalently iff \(X = \alpha Y + \beta\) almost surely. Taking \(U = -Y\), we see that \(-\cov(X,Y) \le \sqrt{\Var(X)\Var(Y)}\), and hence the result follows. ◻

\(L^p\) spaces

Definition 12. A pair \((p,q) \in \R^2\) where \(p, q \ge 1\) and \(\frac{1}{p}+\frac{1}{q} = 1\), i s called the conjugate pair, and the spaces \(L^p\) and \(L^q\) are called dual spaces.

Example 13. The dual of \(L^1\) space is \(L^\infty\). The space \(L^2\) is dual of itself, and called a Hilbert space.

Theorem 14 (H"older’s inequality). Consider a conjugate pair \((p,q)\) and random variables \(X \in L^p, Y \in L^q\). Then,

Proof. Proof. Consider a random variable \(Z: \Omega \to \set{p\ln v, q \ln w}\) with probability mass function \(\set{\frac{1}{p}, \frac{1}{q}}\). It follows from Jensen’s inequality applied to the convex function \(f(x) = e^x\) and the random variable \(Z\), that It follows that for any random variables \(V,W\), we have \(VW \le \frac{V^p}{p} + \frac{W^q}{q}\). Taking expectation on both sides, we get from the monotonicity of expectation that \(\E VW \le \frac{\E V^p}{p} +\frac{\E W^q}{q}.\) Taking \(V \triangleq \frac{\abs{X}}{\norm{X}_p}\) and \(W \triangleq \frac{\abs{Y}}{\norm{Y}_q}\), we get the result. ◻

Definition 15. For a pair of random variables \((X,Y) \in (L^p,L^q)\) for conjugate pair \((p,q)\), we can define inner product \(\inner{}:L^p\times L^q \to \R\) by

Remark 1. For \(X \in L^p\) and \(Y\in L^q\), the expectation \(\E\abs{XY}\) is finite from Hölder’s inequality. Therefore, the inner product \(\inner{X,Y} = \E[XY]\) is well defined and finite.

Remark 2. This inner product is well defined for the conjugate pair \((1,\infty)\).

Theorem 16 (Minkowski’s inequality). For \(1 \le p < \infty\), let \(X,Y\in L^p\) be two random variables defined on a probability space \((\Omega, \sF, P)\). Then, with inequality iff \(X = \alpha Y\) for some \(\alpha \ge 0\) or \(Y = 0\).

Proof. Proof. Since addition is a Borel measurable function, \(X+Y\) is a random variable. We first show that \(X+Y \in L^p\), when \(X,Y \in L^p\). To this end, we observe that \(g:\R_+\to\R_+\) defined by \(g(x)= x^p\) for all \(x \in \R_+\), is a convex function for \(p \ge 1\). From the convexity of \(g\), we have This implies that \(\abs{X+Y}^p \le 2^{p-1}(\abs{X}^p + \abs{Y}^p)\).

The inequality holds trivially if \(\norm{X+Y}_p = 0\). Therefore, we assume that \(\norm{X+Y}_p > 0\), without any loss of generality. Using the definition of \(\norm{}_p\), triangle inequality, and linearity of expectation we get From the Hölder’s inequality applied to conjugate pair \((p,q)\) to the two products on RHS, we get Recall that \(q = \frac{p}{p-1}\). Therefore, \(\norm{\abs{X+Y}^{p-1}}_q = (\E\abs{X+Y}^p)^{1-\frac{1}{p}}\) and the result follows. ◻

Remark 3. We have shown that the map \(\norm{}_p\) is a norm by proving the Minkowski’s inequality. Therefore, \(L^p\) is a normed vector space. We can define distance between two random variables \(X_1, X_2 \in L^p\) by the norm \(\norm{X_1-X_2}_p\).