Foundations of Probability

The goal of this section is to introduce the key definitions and concepts encountered in probability theory. An interesting historical note: the widespread use of probability in the scientific community did not occur until the early $20^{\text {th}}$ century, and this was in part facilitated by Andrei Kolmogorov's 1933 publication on the "Foundations of the Theory of Probability", which reduced the whole of probability at that time to a few simple axioms (described in section 1.3).

General Probability Relationships

Inclusion-Exclusion Formula

Two events $(A, B)$ :

\quad P(A \cup B)=P(A)+P(B)-P(A, B)

Three events $(A, B, C)$ :

\quad P(A \cup B \cup C)=P(A)+P(B)+P(C)-P(A, B)-P(A, C)-P(B, C)+P(A, B, C)

Conditional Probability

P(A \mid B)=\frac{P(A, B)}{P(B)}

Chain Rule

\begin{aligned} P(A, B) & =P(A \mid B) P(B) \quad \text { Condition } P(A, B) \text { on } \mathrm{B} \\ P(A, B, C) & =P(A \mid B, C)[P(B, C)] \quad \text { Condition on } P(A, B, C) \text { on }(\mathrm{B}, \mathrm{C}) \\ & =P(A \mid B, C)[P(B \mid C) P(C)] \quad \text { Condition } P(B, C) \text { on } \mathrm{C} \end{aligned}

Total Law

P(B)=\sum_{i=1}^{n} P\left(B \mid A_{i}\right) P\left(A_{i}\right)

Bayes' Theorem

\begin{aligned} P(B, A) & =P(A, B) \\ P(B \mid A) P(A) & =P(A \mid B) P(B) \quad \text { Condition on } A \text { on left and } B \text { on right } \\ P(B \mid A) & =\frac{P(A \mid B) P(B)}{P(A)} \quad \text { Bayes' Theorem - changes condition event } \\ & =\frac{P(A \mid B) P(B)}{\sum_{i=1}^{n} P\left(A \mid B_{i}\right) P\left(B_{i}\right)} \quad \text { Express } P(A) \text { using Total Law } \end{aligned}

Independence

P(A, B)=P(A) P(B)

Covariance & Correlation Experiments, Sample spaces, and Events