Multivariate Distributions

Up to this point we have mostly focused on single (univariate) random variables (i.e. $f_{X}(x)$ ). However, in certain experiments it might be appropriate to explore the relationships between several random variables, such as the relationship between blood cholesterol and heart disease. In this section, we will mostly focus on the bivariate case (e.g. $\left.f_{X, Y}(x, y)\right)$ - that is, pairs of random variables $(X, Y)$ - as the main concepts can easily be extended to the general multivariate case.

Joint Cumulative Distribution Function

We begin by defining the joint cumulative distribution function (CDF), which we recall from our analysis of single (univariate) random variables is the same for discrete and continuous random variables:

The joint cumulative distribution function (joint CDF) of the random variables $X$ and $Y$ is given by the function

Recall that the comma here represents the intersection between $X$ and $Y$ (i.e. $X \cap Y$ ). The function $F_{X, Y}(x, y)$ evaluated at a point $(x, y)$ is the probability that the random variable $X$ takes a value less than or equal to $x$, and the random variable $Y$ takes a value less than or equal to $y$.

It should be noted that this definition also applies to mixed distributions (i.e. when $X$ and $Y$ are a mixture of discrete and continuous random variables). Joint CDFs have the following properties:

1. $F_{X, Y}(x, y) \in[0,1]$ for all $(x, y) \in \mathbb{R}$
2. $F_{X, Y}(x, y) \rightarrow 0$ as $x \rightarrow-\infty$ or $y \rightarrow-\infty$
3. $F_{X, Y}(x, y) \rightarrow F_{X}(x)$ as $y \rightarrow \infty$ and $F_{X, Y}(x, y) \rightarrow F_{Y}(y)$ as $x \rightarrow \infty$

Property 3 stems from the fact that if $y \rightarrow \infty$, then the event $Y \leq y$ occurs with probability 1 (and vice versa for the case where $x \rightarrow \infty)$.

Joint Discrete Random Variables

Suppose that the random variables $X, Y$ are both discrete. It is straightforward to generalise the univariate definition of the PMF to the bivariate (and multivariate) case:

The joint probability mass function (joint PMF) of the discrete random variables $X$ and $Y$ is given by the function

The function $f_{X, Y}(x, y)$ evaluated at a point $(x, y)$ is the probability that the random variable $X$ takes the value $x$, and the random variable $Y$ takes the value $y$.

If we know the joint $\operatorname{PMF} f_{X, Y}(x, y)$, we can compute the joint $\operatorname{CDF} F_{X, Y}(x, y)$ by summation of the appropriate probabilities:

As in the univariate case, the total probability (i.e. summation over all values for $x$ and $y$ ) must equal 1 :

We can display the values of the joint PMF in a two-dimensional table, with the rows representing the values of $X$ and the columns representing the values of $Y$.

Example: Joint PMF for Two Fair Dice

Let's consider our favourite PMF example where we roll two fair dice. We will define $X$ to be the usual sum of the two dice (see the Examples in Section 2.2 and the PMF shown in Figure 3), and $Y$ to be the larger of the two numbers (i.e. $Y=\max \left(\omega_{1}, \omega_{2}\right)$ ).

We can compute each of the entries for $f_{X, Y}(x, y)$ by fixing a value for $Y=y$ (the largest number on either die), fixing a value for $X=x$ (the sum of the two die), and then seeing the there exists an outcome $\omega$ such that $x=y+\omega$.

Let's see how this applies to a few scenarios for $Y=3$ :

By enumerating over all $X$ and $Y$, we can compute the entire joint PMF $f_{X, Y}(x, y)$ as:

From the joint PMF we can directly compute a number of joint queries:

1. $F_{X, Y}(4,4)=\frac{6}{36}$
2. $F_{X, Y}(12,6)=1$
3. $P(X=6,3<Y \leq 6)=\frac{4}{36}$

Joint Continuous Random Variables

Let us now consider the case where random variables $X$ and $Y$ are both continuous.

The random variables $X$ and $Y$ are jointly continuous if there exists a function $f_{X, Y}(x, y)$ and a joint probability density function (joint PDF) of $X, Y$, with the following property:

for any region $A$. Integrating the PDF over a region $A$ gives the probability that $X, Y$ take values within that region. The order of integration is not important and is interchangeable (as we shall see in the example below).

If the region $A$ of interest is rectangular, we have:

The total probability must be equal to 1 , so if we integrate the joint PDF everywhere, we obtain:

The joint CDF and PDF are linked in the usual way, but we need to integrate/differentiate twice,

Example: A Simple Joint PDF for Continuous RVs

Consider the PDF for two continuous random variables, $X$ and $Y$, given by:

(a) Find the value of $k$ so that $f_{X, Y}(x, y)$ is a valid joint PDF.

Solution:

The expression above is a joint PDF if and only if $\iint f_{X, Y}(x, y) d x d y=1$. So integrating the PDF over the correct range $([0,1])$, and integrating over $x$ first we have:

Thus $k=6$. The complete expression for $f_{X, Y}$ then becomes:

(b) Compute $P(X>Y)$.

Solution:

As we mentioned in the definition above, we need to integrate over the area $A$ defined by $X>Y$, and the order of integration (i.e. integrating over $x$ or $y$ first) should not matter. However, for nonrectangular areas (i.e. there is a dependence between $x$ and $y$ ) we must be careful in determining our range of integration.

Trick for Determining Integration Bounds for Non-rectangular Areas:

1.) First construct a table that specifies the bounds on each of the variables. For this particular example we have:

2.) Now we integrate the first variable over the range specified in the table above. The two possibilities for $x$ and $y$ are summarised in red below:

3.) Specifying the outer integration range for the second variable is a bit trickier, as we can no longer use the first variable in the range (it will have already been removed from the expression due to the first integration). In the case, if any of first variables appear in the range of the outer integration, we set them to the values of their corresponding bounds (as read from the table), which is shown in blue below:

Let us now compute $P(X>Y)$ by integrating over $y$ first and then $x$ :

We leave it to the reader to prove we get the same answer when integrating over $x$ first and then $y$.

Marginal Distributions

We introduced the concept of marginalisation in Section 2.1.1, and we saw that it is essentially the Total Law of probability applied to random variables. Here we provide a more formal definition of marginalisation for discrete and continuous random variables and utilise examples to illustrate.

The marginal probability mass functions (marginal PMFs) of the discrete random variables $X$ and $Y$ are the functions

The marginal of $X$ gives us the probability distribution of the random variable $X$ alone, ignoring any information about $Y$, and vice versa.

Example: Marginalisation of the Joint PMF for Two Dice

Consider again our example of a joint PMF $f_{X, Y}(x, y)$, where $X$ was defined to be sum of the two dice and $Y$ to be the larger of the two numbers. Using the above definition, we can compute $f_{X}(x)$ by summing over all $y$ (i.e. over all columns) for a given $x$; the resulting values are shown in the far right column below. Likewise, we can compute $f_{Y}(y)$ by summing over all $x$ (i.e. over all rows) for a given $y$; the resulting values are shown in the bottom row below:

Notice that the resulting $f_{X}(x)$ and $f_{Y}(y)$ are indeed PMFs as they satisfy the two requirements (i.e. they are bounded between zero and one, and they sum to one). The univariate PMFs $f_{X}(x)$ and $f_{Y}(y)$ are commonly written in the margins of the joint PMF table (as shown above), and are known as the marginals (hence the term "Marginalisation").

Pause and Reflect 1: Recall we defined marginalisation in Section 2.1.1 as $P(X)=\sum_{\forall y} P(X, y)$. We have simply re-expressed this as $f_{X}(x)=\sum_{\forall y} f_{X, Y}(x, y)$ using PMF functions for discrete random variables; these two expressions are identical.

Pause and Reflect 2: Does $f_{X}(x)$ looks familiar? Check out Figure 3 and the table above it.

Pause and Reflect 3: Note that it is generally not possible to go the other way around; that is, to reconstruct the full joint PMF table $f_{X, Y}(x, y)$ from $f_{X}(x)$ and $f_{Y}(y)$ alone (unless the random variables are independent, which will be discussed in the next section).

In an analogous fashion we can define marginalisation for continuous random variables as follows:

The marginal probability density functions (marginal PDFs) of the continuous random variables $X$ and $Y$ are the functions

This is essentially the same as the previous definition; in a continuous setting, getting rid of a variable requires integration rather than summation. Let's consider a few examples to illustrate marginalising joint PDFs.

Example: Marginalisation of Joint PDFs for Continuous RVs

1.) Find $f_{X}(x)$ for the following joint PDF:

2.) Find the marginal density functions $f_{X}(x)$ and $f_{Y}(y)$ for the following joint PDF:

Independence of Two Random Variables

Though we have previously encountered independent random variables, we can now give a formal definition.

For the random variables $X$ and $Y$, if the events $X \leq x$ and $Y \leq y$ are independent for all $x, y \in \mathbb{R}$, then

and we say that $X$ and $Y$ are independent random variables.

If $X, Y$ are independent, it is straightforward to show that $f_{X, Y}(x, y)=f_{X}(x) f_{Y}(y)$, which for discrete random variables is exactly what we had in Chapter $1(P(A, B)=P(A) P(B)$ if $A$ and $B$ are independent). We will exploit this property to determine if $X$ and $Y$ are independent, where we will be left with functions $f_{X}(x)$ and $f_{Y}(y)$ rather than numbers (e.g. $P(A)$ and $P(B)$ ).

Example: Independence of a Simple Joint PDF

1.) Consider the previous example for $f_{X, Y}(x, y)=6 x^{2} y$ (for $x, y \in[0,1]$ ); are $X$ and $Y$ independent?

Solution:

From the definition of conditional independence, $X$ and $Y$ are independent if and only if $f_{X, Y}(x, y)=$ $f_{X}(x) f_{Y}(y)$; therefore we need to compute $f_{X}$ and $f_{Y}$. This can be done using marginalisation:

Combining the terms to compute $f_{X}(x) f_{Y}(x)=3 x^{2} \cdot 2 y=6 x^{2} y$

2.) How about for the joint PDF defined by $f_{X, Y}(x, y)=\frac{3}{2} x^{3} y^{2}$ in the example above? Do the marginal PDFs show that $X$ and $Y$ independent?

Conditional Distributions

Now suppose that we know the value taken by the random variable $X$, and we wish to work out how this affects the distribution of $Y$. In the discrete case, this is a simple conditional probability calculation:

In this expression, the RHS is the ratio of the joint PMF and the marginal PMF of $X$. The LHS is a conditional PMF, and we can use the same definition for the general case.

The conditional probability mass/density function (conditional PMF/PDF) of the random variables $X$ and $Y$ is the function

and similarly for $f_{Y \mid X}(\cdot \mid x)$. The function $f_{Y \mid X}(\cdot \mid x)$ gives the distribution of the random variable $Y$ conditional on the event $X=x$.

We have previously mentioned that $f_{X, Y}(x, y)=f_{X}(x) f_{Y}(y)$ if $X, Y$ are independent. In this case the conditional PMF/PDF reduces to:

This is not surprising; if the random variables are independent, the value of $X$ does not contain any information about the value of $Y$, so the conditional is equal to the marginal (compare to similar derivation in Section 1.7).

Expectation for Multivariate Distributions

Now that we can compute the conditional PMFs and PDFs $f_{X \mid Y}(x \mid y)$ (or $f_{Y \mid X}(y \mid x)$ ), we can define expectations of conditional distributions:

The conditional expectation of the random variable $X$ given $Y=y$ is defined as:

Conditional expectation allows us to compute the average value of $X$ if we already know the value of $Y$.

To compute the expectation of a joint function of two random variables, we sum/integrate at every point $(x, y)$ the value of the function multiplied by the joint PMF/PDF:

As we have shown already, expectation is a linear operator, so for random variables $X, Y$ and constants $a, b, c \in \mathbb{R}$, we have

Example: Expectation of a Joint PDF

Consider an electrical circuit with two resistors $\left(R_{X}, R_{Y}\right)$ wired in parallel. Let's define continuous random variables $X$ and $Y$ to be the resistance for each resistor, which varies between 10 and 20 ohms according to the following PDF:

The resistance $R$ of a parallel circuit is traditionally defined as:

Given that the resistances are distributed according to the above joint PDF, what is the expected resistance of the circuit?

Solution:

Covariance

It is often useful to characterise the nature of the dependence between two random variables. Here we will just consider the covariance, as defined below.

The covariance of two random variables $X$ and $Y$ is defined as

Covariance is a measure of linear association between two random variables.

Positive covariance indicates that large values of $X$ tend to be associated with large values of $Y$. The higher the covariance, the stronger the relationship. Conversely, if $\operatorname{Cov}[X, Y]<0$, then large values of $X$ tend to be associated with small values of $Y$, and vice versa. A covariance near zero indicates that there is no simple linear relationship between the two variables.

Consider the case where $X$ and $Y$ are independent random variables:

which results in $\operatorname{Cov}[X, Y]=0$ (and similarly for the discrete case). However, we must be careful to note that the converse does not hold! That is, if $\operatorname{Cov}[X, Y]=0$ we cannot conclude that $X$ and $Y$ are necessarily independent.

Example: Covariance of the Joint PMF for Two Dice

For one last time recall the joint PMF $f_{X, Y}(x, y)$, where $X$ was defined to be the sum of two dice and $Y$ to be the larger of the two numbers. We have also computed the marginal functions $f_{X}(x)$ and $f_{Y}(y)$ in an earlier example. The covariance between discrete random variables $X$ and $Y$ is:

Recall we have already computed $E[X]=7$ (expectation for the sum of two dice). Thus all we need is $E[X Y]$ and $E[Y]$ :

This result for the covariance makes intuitive sense, since as the value for the larger of the two numbers increases (i.e. $Y$ increases), the value of the sum of the two dice $(X)$ is also expected to increase.

For random variables $X, Y, Z$ and constants $a, b, c, d$, we can use the properties of expectation $(E[a X]=$ $a E[X]$ and $E[a]=a$ for some constant $a$ ) to derive the following relationships:

1. $\operatorname{Cov}[X, a]=0$
2. $\operatorname{Cov}[a X+b, c Y+d]=a c \operatorname{Cov}[X, Y]$
3. $\operatorname{Cov}[X+Y, Z]=\operatorname{Cov}[X, Z]+\operatorname{Cov}[Y, Z]$

We leave it to the reader to prove these properties.

We say that covariance is a bilinear operator, in the sense that it is linear in both its inputs. Notice that $\operatorname{Cov}[X, X]=\operatorname{Var}[X]$, so the second of the properties above explains why $\operatorname{Var}[a X+b]=a^{2} \operatorname{Var}[X]$. The third property implies that

A deficiency of covariance is that it depends on the units of measurement. For example, if $X$ and $Y$ are time measurements and we change the units from minutes to seconds, the covariance becomes $\operatorname{Cov}[60 X, 60 Y]=3600 \operatorname{Cov}[X, Y]$.