Independence
Law of Total Probability
Exercise 1 (Countably infinite coin tosses). Consider a sequence of coin tosses, such that the sample space is \Omega = \left\{H,T\right\}^\mathbb{N}. For set of outcomes E_n \triangleq \left\{\omega \in \Omega: \omega_n = H\right\}, we consider an event space generated by \mathscr{F}\triangleq \sigma(\left\{E_n: n\in \mathbb{N}\right\}). Let \mathscr{F}_n be the event space generate by the first n coin tosses, i.e. \mathscr{F}_n \triangleq \sigma(\left\{E_i: i \in [n]\right\}). Let A_n be the set of outcomes corresponding to at least one head in first n outcomes A_n \triangleq \left\{\omega \in \Omega: \omega_i = H \text{ for some } i \in [n]\right\} = \cup_{i=1}^nE_i \in \mathscr{F}, and B_n be the set of outcomes corresponding to first head at the nth outcome B_n \triangleq \left\{\omega \in \Omega: \omega_1 = \dots = \omega_{n-1}= T, \omega = H\right\} = \cap_{i=1}^{n-1}E_i^c\cap E_n \in \mathscr{F}.
Show that \mathscr{F}= \sigma(\left\{\mathscr{F}_n: n \in \mathbb{N}\right\}).
Show that \sigma(\left\{A_n: n \in \mathbb{N}\right\}) \subseteq \mathscr{F} and \sigma(\left\{B_n: n\in \mathbb{N}\right\}) \subseteq \mathscr{F}.
Theorem 2 (Law of total probability). For a probability space (\Omega, \mathscr{F}, P), consider a sequence of events B\in \mathscr{F}^\mathbb{N} that partitions the sample space \Omega, i.e. B_m \cap B_n = \emptyset for all m \neq n, and \cup_{n \in \mathbb{N}}B_n = \Omega. Then, for any event A \in \mathscr{F}, we have \begin{equation*} P(A) = \sum_{n \in \mathbb{N}}P(A \cap B_n). \end{equation*}
Proof. Proof. We can expand any event A \in \mathscr{F} in terms of any partition B of the sample space \Omega as \begin{equation*} A = A \cap \Omega = A \cap (\cup_{n \in \mathbb{N}}B_n) = \cup_{n \in \mathbb{N}}(A \cap B_n). \end{equation*} From the mutual disjointness of the events B \in \mathscr{F}^\mathbb{N}, it follows that the sequence (A \cap B_n \in \mathscr{F}: n \in \mathbb{N}) is mutually disjoint. The result follows from the countable additivity of probability of disjoint events. ◻
Example 3 (Countably infinite coin tosses). Consider the sample space \Omega = \left\{H,T\right\}^\mathbb{N} and event space \mathscr{F} generated by sequence E \in \mathscr{F}^\mathbb{N} defined in Exercise 1. We observe that any event A \in \mathscr{F}_n can be written as \begin{equation*} A = \cup_{\omega \in A}\left\{\omega\right\} = \cup_{\omega \in A}\cap_{i=1}^n(\left\{\omega \in E_i\right\}\cup\left\{\omega \notin E_i\right\}). \end{equation*}
Independence
Definition 4 (Independence of events). For a probability space (\Omega, \mathscr{F}, P), a family of events A \in \mathscr{F}^I is said to be independent, if for any finite set F \subseteq I, we have \begin{equation*} P(\cap_{i \in F}A_i) = \prod_{i \in F}P(A_i). \end{equation*}
Remark 1. The certain event \Omega and the impossible event \emptyset are always independent to every event A \in \mathscr{F}.
Example 5 (Two coin tosses). Consider two coin tosses, such that the sample space is \Omega = \left\{HH, HT, TH, TT\right\}, and the event space is \mathscr{F}= \mathcal{P}(\Omega). It suffices to define a probability function P: \mathscr{F}\to [0,1] on the sample space. We define one such probability function P, such that \begin{equation*} P(\left\{HH\right\}) = P(\left\{HT\right\}) = P(\left\{TH\right\}) =P(\left\{TT\right\}) = \frac{1}{4}. \end{equation*} Let event E_1 \triangleq \left\{HH, HT\right\} and E_2 \triangleq \left\{HH, TH\right\} correspond to getting a head on the first or the second toss respectively.
From the defined probability function, we obtain the probability of getting a tail on the first or the second toss is \frac{1}{2}, and identical to the probability of getting a head on the first or the second toss. That is, P(E_1) = P(E_2) = \frac{1}{2} and the intersecting event E_1\cap E_2 = \left\{HH\right\} with the probability P(E_1 \cap E_2) = \frac{1}{4}. That is, for events E_1, E_2 \in \mathscr{F}, we have \begin{equation*} P(E_1 \cap E_2) = P(E_1)P(E_2). \end{equation*} That is, events E_1 and E_2 are independent.
Example 6 (Countably infinite coin tosses). Consider the outcome space \Omega = \left\{H,T\right\}^\mathbb{N} and event space \mathscr{F} generated by the sequence E defined in Exercise 1. We define a probability function P:\mathscr{F}\to [0,1] by P(\cap_{i\in F}E_i)= p^{\left\lvert F\right\rvert} for any finite subset F \subseteq \mathbb{N}. By definition, E \in \mathscr{F}^\mathbb{N} is a sequence of independent events. Consider A, B \in \mathscr{F}^\mathbb{N}, where A_n \triangleq \cup_{i=1}^nE_i and B_n \triangleq \cap_{i=1}^{n-1}E_i^c\cap E_n \in \mathscr{F} for all n\in\mathbb{N}. It follows that P(A_n) = 1-(1-p)^n and P(B_n) = p(1-p)^{n-1} for n \in \mathbb{N}.
For any \omega \in \Omega, we can define the number of heads in first n trials by k_n(\omega) \triangleq\sum_{i=1}^n\mathbbm{1}_{\left\{H\right\}}(\omega_i) = \sum_{i=1}^n\mathbbm{1}_{\left\{\omega \in E_i\right\}}. For any general event A \in \mathscr{F}_n = \sigma(\left\{E_i: i \in [n]\right\}), we can write \begin{equation*} P(A) = \sum_{\omega \in A} \prod_{i=1}^n\Big[P\left\{\omega \in E_i\right\} + P\left\{\omega \in E_i^c\right\}\Big] = \sum_{\omega \in A}p^{k_n(\omega)}(1-p)^{n-k_n(\omega)}. \end{equation*}
Example 7 (Counter example). Consider a probability space (\Omega, \mathscr{F}, P) and the events A_1, A_2, A_3 \in \mathscr{F}. The condition P(A_1\cap A_2\cap A_3) = P(A_1)P(A_2)P(A_3) is not sufficient to guarantee independence of the three events. In particular, we see that if
2 &P(A_1A_2A_3) = P(A_1)P(A_2)P(A_3),&&P(A_1A_2A_3^c) P(A_1)P(A_2)P(A_3^c),
then P(A_1\cap A_2) = P(A_1\cap A_2\cap A_3) + P(A_1\cap A_2\cap A_3^c) \neq P(A_1)P(A_2).
Definition 8. A family of collections of events (\mathscr{A}_i \subseteq \mathscr{F}: i \in I) is called independent, if for any finite set F \subseteq I and A_i \in \mathscr{A}_i for all i \in F, we have \begin{equation*} P(\cap_{i \in F}A_i) = \prod_{i \in F}P(A_i). \end{equation*}
Conditional Probability
Consider N trials of a random experiment over an outcome space \Omega and an event space \mathscr{F}. Let \omega_n \in \Omega denote the outcome of the experiment of the nth trial. Consider two events A, B \in \mathscr{F} and denote the number of times event A and event B occurs by N(A) and N(B) respectively. We denote the number of times both events A and B occurred by N(A\cap B). Then, we can write these numbers in terms of indicator functions as
3 &N(A) = _n=1^N_{_n A}, &&N(B) = _n=1^N_{_n B}, &&N(AB) = _n=1^N_{_n A B}.
We denote the relative frequency of events A, B, A\cap B in N trials by \frac{N(A)}{N}, \frac{N(B)}{N}, \frac{N(A\cap B)}{N} respectively. We can find the relative frequency of events A, on the trials where B occurred as \begin{equation*} \frac{\frac{N(A\cap B)}{N}}{\frac{N(B)}{N}} = \frac{N(A \cap B)}{N(B)}. \end{equation*} Inspired by the relative frequency, we define the conditional probability function conditioned on events.
Definition 9. Fix an event B \in \mathscr{F} such that P(B) > 0, we can define the conditional probability P(\cdot | B): \mathscr{F}\to [0,1] of any event A \in \mathscr{F} conditioned on the event B as \begin{equation*} P(A|B) = \frac{P(A\cap B)}{P(B)}. \end{equation*}
Lemma 10 (Conditional probability). For any event B \in \mathscr{F} such that P(B) > 0, the conditional probability P(\cdot | B): \mathscr{F}\to [0,1] is a probability measure on space (\Omega, \mathscr{F}).
Proof. Proof. We will show that the conditional probability satisfies all three axioms of a probability measure.
For all events A \in \mathscr{F}, we have P(A|B) \geqslant 0 since P(A \cap B) \geqslant 0.
For an infinite sequence of mutually disjoint events (A_i \in \mathscr{F}: i \in \mathbb{N}) such that A_i \cap A_j = \emptyset for all i \neq j, we have P(\cup_{i \in \mathbb{N}}A_i|B) = \sum_{i \in \mathbb{N}}P(A_i|B). This follows from disjointness of the sequence (A_i\cap B \in \mathscr{F}: i \in \mathbb{N}).
Since \Omega \cap B = B, we have P(\Omega |B) = 1.
◻
Remark 2. For two independent events A, B \in \mathscr{F} such that P(A\cap B) > 0, we have P(A|B) = P(A) and P(B|A) = P(B). If either P(A) = 0 or P(B) = 0, then P(A \cap B) = 0.
Remark 3. For any partition B of the sample space \Omega, if P(B_n) > 0 for all n \in \mathbb{N}, then from the law of total probability and the definition of conditional probability, we have \begin{equation*} P(A) = \sum_{n \in \mathbb{N}}P(A|B_n)P(B_n). \end{equation*}
Conditional Independence
Definition 11 (Conditional independence of events). For a probability space (\Omega, \mathscr{F}, P), a family of events A \in \mathscr{F}^I is said to be conditionally independent given an event C \in \mathscr{F} such that P(C) > 0, if for any finite set F \subseteq I, we have \begin{equation*} P(\cap_{i \in F}A_i|C) = \prod_{i \in F}P(A_i|C). \end{equation*}
Remark 4. Let C \in \mathscr{F} be an event such that P(C) > 0. Two events A, B \in \mathscr{F} are said to be conditionally independent given event C, if \begin{equation*} P(A \cap B|C) = P(A|C) P(B|C). \end{equation*} If the event C = \Omega, it implies that A, B are independent events.
Remark 5. Two events may be independent, but not conditionally independent and vice versa.
Example 12. Consider two independent events A, B \in \mathscr{F} such that P(A\cap B) > 0 and P(A \cup B) < 1. Then the events A and B are not conditionally independent given A\cup B. To see this, we observe that \begin{equation*} P(A \cap B | A\cup B) = \frac{P( (A \cap B)\cap (A \cup B))}{P(A\cup B)} = \frac{P(A \cap B)}{P(A\cup B)} = \frac{P(A)P(B)}{P(A \cup B)} = {P(A|A \cup B)}{P(B)}. \end{equation*} We further observe that P(B|A\cup B) = \frac{P(B)}{P(A\cup B)} \neq P(B) and hence P(A \cap B | A\cup B) \neq P(A|A\cup B)P(B|A\cup B).
Example 13. Consider two non-independent events A, B \in \mathscr{F} such that P(A) > 0. Then the events A and B are conditionally independent given A. To see this, we observe that \begin{equation*} P(A \cap B | A) = \frac{P(A \cap B)}{P(A)} = P(B|A)P(A|A). \end{equation*}