Generating functions

Generating functions

Suppose that X:\Omega \to \R is a continuous random variable on the probability space (\Omega,\sF, P) with distribution function F_X: \R \to [0,1].

Characteristic function

Example 1. Let j \triangleq \sqrt{-1}, then we can show that h_u: \R \to \C defined by h_u(x) \triangleq e^{ju x} = \cos(u x) + j \sin(u x) is also Borel measurable for all u \in \R. Thus, h_u(X):\Omega \to \C is a complex valued random variable on this probability space.

Definition 2. For a random variable X: \Omega \to \R defined on the probability space (\Omega, \sF, P), the characteristic function \Phi_X: \R \to \C is defined by \Phi_X(u) \triangleq \E e^{j u X} for all u \in \R and j^2 = -1.

Remark 1. The characteristic function \Phi_X(u) is always finite, since \abs{\Phi_X(u)} = \abs{\E e^{ju X}} \le \E\abs{e^{ju X}} = 1.

Remark 2. For a discrete random variable X: \Omega \to \sX with PMF P_X:\sX\to [0,1], the characteristic function \Phi_X(u) %= \E[e^{juX}] = \sum_{x \in \sX}e^{ju x}P_X(x).

Remark 3. For a continuous random variable X: \Omega \to \R with density function f_X: \R \to \R_+, the characteristic function \Phi_X(u) = \int_{-\infty}^{\infty}e^{ju X}f_X(x)dx.

Example 3 (Gaussian random variable). For a Gaussian random variable X: \Omega \to \R with mean \mu and variance \sigma^2, the characteristic function \Phi_X is We observe that \abs{\Phi_X(u)} = e^{-u^2\sigma^2/2} has Gaussian decay with zero mean and variance 1/\sigma^2.

Theorem 4. If \E\abs{X}^N is finite for some integer N \in \N, then \Phi_X^{(k)}(u) is finite and continuous functions of u \in \R for all k \in [N]. Further, \Phi_X^{(k)}(0) = j^k\E[X^k] for all k \in [N].

Proof. Proof. Since \E\abs{X}^N is finite, then so is \E\abs{X}^k for all k\in [N]. Therefore, \E[X^k] exists and is finite. Exchanging derivative and the integration (which can be done since e^{ju x} is a bounded function with all derivatives), and evaluating the derivative at u=0, we get \Phi_X^{(k)}(0) = \E\left[\frac{d^ke^{ju X}}{du^k}\Big\rvert_{u=0}\right] %= \int_{-\infty}^{\infty}\frac{d^ke^{juX}}{du^k}\Big\rvert_{u=0}f_X(x)dx %= \int_{-\infty}^{\infty}j^kx^kf_X(x)dx %= \int_{-\infty}^{\infty}(j^kxe^{juX})\Big\rvert_{u=0}f_X(x)dx = j^k\E[X^k]. ◻

Theorem 5. Two random variables have the same distribution iff they have the same characteristic function.

Proof. Proof. It is easy to see the necessity and the sufficiency is difficult. ◻

Moment generating function

Example 6. A function g_t: \R \to \R_+ defined by g_t(x) \triangleq e^{t x} is monotone and hence Borel measurable for all t \in \R. Therefore, g_t(X): \Omega \to \R_+ is a positive random variable on this probability space.

Definition 7. For a random variable X: \Omega \to \R defined on the probability space (\Omega, \sF, P), the moment generating function M_X: \R \to \R_+ is defined by M_X(t) \triangleq \E e^{t X} for all t \in \R where M_X(t) is finite.

Remark 4. Characteristic function always exist, however are complex in general. Sometimes it is easier to work with moment generating functions, when they exist.

Lemma 8. For a random variable X, if the MGF M_X(t) is finite for some t \in \R, then M_X(t) = 1 + \sum_{n \in \N} \frac{t^n}{n!}\E[X^n].

Example 9 (Gaussian random variable). For a Gaussian random variable X: \Omega \to \R with mean \mu and variance \sigma^2, the moment generating function M_X is M_X(t) = \exp\Big(\frac{t^2\sigma^2}{2}+t\mu\Big).

Probability generating function

For a non-negative integer-valued random variable X: \Omega \to \sX \subseteq \Z_+, it is often more convenient to work with the z-transform of the probability mass function, called the probability generating function.

Definition 10. For a discrete non-negative integer-valued random variable X: \Omega \to \sX \subseteq \Z_+ with probability mass function P_X: \sX \to [0,1], the probability generating function \Psi_X: \C \to \C is defined by

Lemma 11. For a non-negative simple random variable X: \Omega \to \sX, we have \abs{\Psi_X(z)} \le 1 for all \abs{z} \le 1.

Proof. Proof. Let z \in \C with \abs{z}\le 1. Let P_X : \sX \to [0,1] be the probability mass function of the non-negative simple random variable X. Since any realization x \in \sX of random variable X is non-negative, we can write ◻

Theorem 12. For a non-negative simple random variable X:\Omega \to \sX with finite kth moment \E X^k, the k-th derivative of probability generating function evaluated at z=1 is the k-th order factorial moment of X. That is,

Remark 5. Moments can be recovered from kth order factorial moments. For example,

Theorem 13. Two non-negative integer-valued random variables have the same probability distribution iff their z-transforms are equal.

Proof. Proof. The necessity is clear. For sufficiency, we see that \Psi_X^{(k)} = \sum_{x \ge k}k!z^{x-k}P_X(x) and hence \Psi_X^{(k)}(0) = k!P_X(k). Further, interchanging the derivative and the summation (by dominated convergence theorem), we get the second result. ◻

Gaussian Random Vectors

Definition 14. For a random vector X: \Omega\to \R^n defined on a probability space (\Omega,\sF,P), we can define the characteristic function \Phi_X:\R^n\to\C by \Phi_X(u) \triangleq \E e^{j \inner{u,X}} where u \in \R^n.

Remark 6. If X:\Omega\to\R^n is an independent random vector, then \Phi_X(u) = \prod_{i=1}^n\Phi_{X_i}(u_i) for all u \in \R^n.

Definition 15. For a probability space (\Omega, \sF, P), Gaussian random vector is a continuous random vector X: \Omega \to \R^n defined by its density function where the vector \mu \in \R^n and the positive definite matrix \Sigma \in \R^{n \times n}.

Remark 7. For a Gaussian random vector with vector \mu = (\mu_1, \dots, \mu_1) for some real scalar \mu_1 and matrix \Sigma = \sigma^2I for some positive \sigma^2 \in \R_+, we can write its density as f_X(x) = \frac{1}{(2\pi\sigma^2)^{n/2}}\exp\left(-\frac{1}{2}\sum_{i=1}^n\frac{(x_i-\mu_1)^2}{\sigma^2}\right)\text{ for all }x \in \R^n. It follows that X is an random vector with each component being a Gaussian random variable with mean \mu_1 and variance \sigma^2. The characteristic function \Phi_X of an Gaussian random vector X:\Omega\to \R^n parametrized by (\mu_1,\sigma^2) is given by \Phi_X(u) = \prod_{i=1}^n\Phi_{X_i}(u_i) = \exp\Big(-\frac{\sigma^2}{2}\sum_{i=1}^nu_i^2 +j\mu_1\sum_{i=1}^nu_i\Big).

Lemma 16. For an zero mean unit variance Gaussian vector Z:\Omega\to\R^n, vector \alpha \in \R^n, and scalar \mu \in \R, the affine combination Y \triangleq \mu + \inner{\alpha, Z} is a Gaussian random variable.

Proof. Proof. From the linearity of expectation and the fact that Z is a zero mean vector, we get \E Y = \mu. Further, from the linearity of expectation and the fact that \E[ZZ^T] = I, we get To show that Y is Gaussian, it suffices to show that \Phi_Y(u) = \exp(-\frac{u^2\sigma^2}{2}+ju\mu) for any u \in \R. Recall that Z is an independent random vector with individual components being identically zero mean unit variance Gaussian. Therefore, \Phi_{Z_i}(u) = \exp(-\frac{u^2}{2}), and we can compute the characteristic function of Y as ◻

Theorem 17. A random vector X: \Omega\to\R^n is Gaussian with parameters (\mu, \Sigma) iff Z \triangleq \Sigma^{-\frac{1}{2}}(X-\mu) is an zero mean unit variance Gaussian random vector.

Proof. Proof. Let X = \mu + \Sigma^{\frac{1}{2}}Z for an zero mean unit variance Gaussian random vector Z:\Omega\to\R^n, then we will show that X is a Gaussian random vector by transformation of random vector densities. Since the (i,j)th component of the Jacobian matrix J(x) is given by J_{ij}(x) = \frac{\partial x_j}{\partial z_i} = \Sigma^{\frac{1}{2}}_{i,j} for all i,j\in [n], we can write the Jacobian matrix J(x) = \Sigma^{\frac{1}{2}}, Since the density of Z is f_Z(z) = \frac{1}{\sqrt{(2\pi)^n}}\exp(-\frac{1}{2}z^Tz), and from the transformation of random vectors, we get

Conversely, we can show that if X is a Gaussian random vector, then Z = \Sigma^{-\frac{1}{2}}(X-\mu) is an zero mean unit variance Gaussian random vector by transformation of random vectors. ◻

Remark 8. A random vector X:\Omega\to\R^n with mean \mu \in \R^n and covariance matrix \Sigma \in \R^{n\times n} is Gaussian iff X can be written as X = \mu + \Sigma^{\frac{1}{2}}Z, for an Gaussian random vector Z: \Omega \to \R^n with mean 0 and variance 1. It follows that \E X = \mu and \Sigma = \E(X-\mu)(X-\mu)^T.

Remark 9. We observe that the components of the Gaussian random vector X = \mu + A Z for A = \Sigma^{\frac{1}{2}} are Gaussian random variables with mean \mu_i and variance \sum_{k=1}^nA_{i,k}^2 = (AA^T)_{i,i} = \Sigma_{i,i}, since each component X_i = \mu_i + \sum_{k=1}^nA_{i,k}Z_k is an affine combination of zero mean unit variance random variables.

Remark 10. For any u \in \R^n, we compute the characteristic function \Phi_X from the distribution of Z as

Lemma 18. If the components of the Gaussian random vector are uncorrelated, then they are independent.

Proof. Proof. If a Gaussian vector is uncorrelated, then the covariance matrix \Sigma is diagonal. It follows that we can write f_X(x) = \prod_{i=1}^nf_{X_i}(x_i) for all x\in\R^n. ◻