Interval Estimation

Up until now we have been using Maximum Likelihood Estimation to generate point estimates for our model parameters $\theta$. In particular, we derived expressions for the estimators $\hat{\theta}$ as a function of random sample $D$. You might have noticed by now that most of our maximum likelihood estimators for distributions considered thus far have been taking on similar functional forms:

That is, the estimators above are equal to the sample mean $\bar{X}=\frac{1}{n} \sum_{i=1}^{n} D_{i}$. This is somewhat intuitive, as we had previously shown the true value of these model parameters to be equal to the mean value of the population distribution $E[X]$ (see Table 1 ). Thus, the best approximation to $E[X]$ would be to compute the mean from a random sample $D=\left(D_{1}, D_{2}, \ldots, D_{n}\right)$ drawn from the underlying distribution for $X$.

However, we had discussed earlier that $D$ itself is a random variable that will change with each sampling trial; therefore the value of the estimate will also change for each random sample. Fortunately, for the estimates listed above we already know how the sample means $\bar{X}$ is distributed over many sampling trials (e.g. see sampling distribution of sample means in Figure 29), and we use these sampling distributions to quantify our uncertainty for any estimate we compute.

Sample Mean for Normal Random Variables

We begin by considering the exact solution where our random samples $D$ are drawn from a Normal distribution. Recall from our discussion on the Normal distribution in Chapter 3 that a Z-transformation of a Normal random variable follows a standard Normal distribution:

We had also shown that the sum of Normal random variables is itself a Normal random variable (note that this is exact and not a CLT approximation). Thus, if a maximum likelihood estimator corresponds to the mean $\bar{X}=\frac{1}{n} \sum_{i=1}^{n} D_{i}$ of a sample $\left(D_{1}, D_{2}, \ldots, D_{n}\right)$ from a Normal distribution (as is the case for $\hat{\mu})$, then we can precisely state that our sampling distribution for $\bar{X}(=\hat{\mu})$ is:

Where $\mu$ and $\sigma^{2}$ are the mean and variance of the underlying Normal population distribution, respectively. Thus, we would like to use this sampling distribution to understand how good our estimate $\hat{\mu}=\bar{X}$ is of the true population mean $\mu$. For simplicity sake, let's say that we are given the population variance $\sigma^{2}$. Then we can rearrange the above inequalities to deduce bounds around the unknown parameter $\mu$ based on an estimate $\bar{X}$ :

Thus, by choosing bounds $a$ and $b$, we can integrate the standard normal on the righthand side to compute the probability that our unknown model parameter $\mu$ is contained within the interval $\left[\bar{X}+a \cdot \frac{\sigma}{\sqrt{n}}, \bar{X}+b \cdot \frac{\sigma}{\sqrt{n}}\right]$; we refer to this interval as the confidence interval.

In order to use our sampling distribution to compute the confidence interval, we need to first specify a probability that the interval we construct using an estimate $\bar{X}$ will contain $\mu$; we define this as our confidence level:

Confidence Level: $(1-\alpha)=$ Probability that a confidence interval computed from $\bar{X}$ will contain the true parameter value $\mu$

It is immediately apparent from the above expression that once we have selected a value for the confidence level (i.e. by selecting an $\alpha$ ), then the corresponding values for $a$ and $b$ are also fixed since the standard normal curve is symmetric about $Z=\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$. To see this, consider the confidence intervals around $Z=\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ for confidence levels of $0.95(\alpha=0.05)$ and $0.99(\alpha=0.01)$, as shown in Figure 30. Due to the symmetry of the standard normal curve, $b=-a$ and we can see in Figure 30 that $b=1.96$ for $\alpha=0.05$ and $b=2.575$ for $\alpha=0.01$.

Computing Confidence Interval for Standard Normal using Lookup Tables

A convenient way to compute the confidence interval for a given confidence level is to use the standard Normal CDF lookup tables (see right side of Figure 30). Once we have specified our confidence level $(1-\alpha)$, we just search the standard Normal CDF lookup table for the corresponding $z_{\alpha / 2}(=b=-a)$ such that:

So for the following scenarios we have:

Table 2: Using the standard Normal CDF lookup table to compute confidence interval for a given confidence level $=(1-\alpha)$.

We can therefore equivalently express our confidence interval for $\mu$ as:

NOTE: It is often helpful to think of $z_{\alpha / 2}$ as how many units of standard error $\left(\frac{\sigma}{\sqrt{n}}\right)$ we allow the confidence interval to be. Thus, the confidence interval is $\bar{X} \pm z_{\alpha / 2} \times$ Standard Error.

Figure 30: Computing confidence intervals for parameter estimates that correspond to the mean of a sample of Normal random variables. First we specify the confidence level $(1-\alpha)$, which is represented here as the shaded area under the curve centred around our estimate. For a) we have specified the confidence level to be $95 \%$, corresponding to $\alpha=0.05$; likewise for c) we have specified the confidence level to be $99 \%$, corresponding to $\alpha=0.01$. To find the corresponding confidence interval for a given confidence level, we then use the standard Normal CDF lookup table (right side). For $\alpha=0.05$ in b), we find the bound to be $z_{\alpha / 2}=1.96$; similarly for $\alpha=0.01$ in $\mathbf{d}$ ), we find the bound to be $z_{\alpha / 2}=2.575$.

Sample Mean for Normal Random Variables, Unknown Population Variance $\left(\sigma^{2}\right)$

Thus far we have considered the case where the population variance $\sigma^{2}$ was a known quantity. However, this parameter is seldom known so it is not precisely correct to say that $\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ follows a standard normal distribution. Indeed, in practice we can only ascertain information about $\sigma^{2}$ by looking at the sample $D$ available to us. Recall that we derived an expression for the sample variance $S^{2}$ by correcting for the bias our MLE estimator for $\hat{\sigma^{2}}$ as:

Thus we can use $S$ as an approximation for $\sigma$ in computing the confidence intervals. But how does our uncertainty in $\sigma$ manifest itself in the calculation of these confidence intervals? Intuitively we can appreciate that for large enough sample sizes, $S$ will be a fairly accurate approximation of $\sigma$. But how do we deal with small sample sizes (i.e. $n<100$ )?

The credit for recognising that $\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ and $\frac{(\bar{X}-\mu)}{S / \sqrt{n}}$ do not follow the same distribution goes to William Sealy Gossett (who graduated from Oxford in 1899 with First Class degrees in Chemistry and Maths). As an employee at the Guinness brewery, Gossett worked on the task of making the art of brewery more scientific. However, for many of his experiments, the sample sizes were only on the order of 4 or 5 , and he knew that there was no possibly way of knowing the exact value of the true population variance $\sigma^{2}$ in his statistical calculations.

In 1908 Gossett published a new PDF that describes the distribution of $\frac{(\bar{X}-\mu)}{S / \sqrt{n}}$. At that time, Guinness forbid any employees to publish papers for confidentiality reasons, so Gossett published these findings under the pen name of "Student". Thus, the resulting PDF was referred to as the Student $t$-distribution (the ' $t$ ' in this name refers to its use in test statistics, which we will analyse in the next chapter).

Formally speaking, for a random sample $D$ of a Normal population distribution with mean $\mu$ and variance $\sigma^{2}$, sampling distribution of the sample mean $\bar{X}$ follows a Student $t$-distribution with $n-1$ degrees of freedom $\left(t_{n-1}\right)$ :

Note here that we define the transform as $T$ instead of $Z$ to distinguish it from the Z-transformation. The Student $t$-distribution, $t_{n-1}$, has a few interesting and important properties, which are illustrated graphically in Figure 31:

1. It is bell-shaped and symmetric about zero, like the Normal distribution, but has "heavier tails" that approach zero probability at a slower rate.
2. Its dispersion varies according the size of the sample, $n$; the smaller the sample size, the greater the dispersion (to reflect the uncertainty in estimating the true variance). 3. As $n$ approaches $\infty, t_{n-1}$ converges to the standard Normal distribution $N(0,1)$ (approximately $n>100$ is close enough to use $N(0,1))$.

Figure 31: Difference between the Student t-distribution, $t_{n-1}$, and standard Normal distribution, $N(0,1)$. As the sample size increases from $n=6$ to $n=21$, we can see that $t_{n-1}$ becomes closer to the standard Normal distribution.

We can analogously define our confidence interval in terms of the Student $t$-distribution for a Normal sample $D$ with unknown population variance $\sigma^{2}$ and sample size $n$ :

Where we note in these tests that the Normal sample $D$ is used to compute both the sample mean and the sample standard deviation:

The corresponding confidence interval bound $t_{\alpha / 2, n-1}$ can be found using a lookup table for the upper percentile of Student $t$-distributions given a corresponding value for $\alpha$ and $n$ in a very similar manner to the standard Normal tables. Indeed, since $t_{n-1}$ is a symmetric distribution, we only need to need to find the value for the bound for the upper tail for a given confidence level $1-\alpha: P\left(T \geq t_{\alpha / 2, n-1}\right)$ (where in contrast for the standard Normal we used the $\left.\operatorname{CDF} P\left(Z \leq z_{\alpha / 2}\right)\right)$.

We illustrate this calculation graphically in Figure 32 for a confidence level of $95 \%(\alpha=0.05)$ and sample sizes $n=6$ and $n=20$. Table 3 below also summarises the confidence interval bounds for additional $\alpha$ values and sample sizes (compare to Table 2):

Table 3: Using the Student $t$-distribution lookup table to compute confidence interval for a given $\alpha$. Note that the 'degrees of freedom' is equal to $n-1$.

Again, it is often helpful to think of $t_{\alpha / 2, n-1}$ as how many units of standard error $\left(\frac{S}{\sqrt{n}}\right)$ we allow the confidence interval to be. Thus, we can write the confidence interval as $\bar{X} \pm t_{\alpha / 2, n-1} \times$ Standard Error, just as we did in the case for the Normal population with known variance. However, we will generally expect a larger confidence interval (i.e. $t_{\alpha / 2, n-1} \geq z_{\alpha / 2}$ ) for a given $\alpha$ due to the additional uncertainty associated with estimating $S$ from the sample data.

For instance, we see in Table 3 for $\alpha=0.05$ that as $n$ increases and our estimation of $S$ from the sample data becomes more accurate (i.e. $S \sim \sigma$ ), $t_{0.025, n-1}$ approaches the standard Normal bound of $z_{0.025}=1.96$. Likewise as $n$ increases for $\alpha=0.01, t_{0.005, n-1}$ approaches the standard Normal bound of $z_{0.005}=2.575$. Using shorthand notation we can write the confidence interval for the estimate as:

Figure 32: Computing confidence intervals for parameter estimates that correspond to the mean of a sample of Normal random variables. First we specify the confidence level $(1-\alpha)$, which corresponds to the area under the curve centred around our estimate. For a) we have specified the area under the curve to be $95 \%$, corresponding to $\alpha=0.05$; likewise for c) we have specified the area under the curve to be $99 \%$, corresponding to $\alpha=0.01$. To find the corresponding values for the bounds around our estimate for a given confidence level, we then use the standard Normal lookup table. For $\alpha=0.05$ in b), we find the bound to be $z_{\alpha / 2}=1.96$; similarly for $\alpha=0.01$ in $\mathbf{d}$ ), we find the bound to be $z_{\alpha / 2}=2.575$.

Sample Mean for Non-Normal Distributions

So far we have considered how to compute confidence intervals for our sample means for a random sample from a Normal distribution. In the case where the population variance $\sigma^{2}$ was known:

And in the case where the population variance is unknown (and must be estimated using the sample variance $\left.S^{2}\right)$ :

But we noted that for large enough $n$ (e.g. $n>100$ ) that the Student $t$-distribution converges to the standard Normal $N(0,1)$, and as a result $z_{\alpha / 2} \approx t_{\alpha / 2, n-1}$ (i.e. the interval becomes the same).

However, we also seen for large sample sizes $n$ that the sample means $\bar{X}$ for a non-Normal population distribution follows a sampling distribution that is Normally distributed. For instance, Figure 29 shows that the sampling distribution of the sample means $\bar{W}$, as computed from random samples of the exponential population distribution $f_{W}(w \mid \hat{\lambda}=0.1)$ - (and therefore $\left.E[W]=\frac{1}{\hat{\lambda}} \equiv \mu, \sqrt{\operatorname{Var}[W]}=\sqrt{\frac{1}{\hat{\lambda}^{2}}} \equiv \sigma\right)$, is itself a Normal distribution given by $N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)$.

Indeed this was the main result of the Central Limit Theorem at the end of Chapter 3: the sample means $\bar{X}$, computed from random samples of any population distribution with mean $\mu$ and variance $\sigma^{2}$, is Normally distributed for large sample sizes $n$ :

And since the CLT only applies to large $n$, we can use the sample standard deviation $S$ to approximate $\sigma^{2}$ without introducing too much further variability into our confidence intervals:

Confidence Intervals for General Sample Statistics

We can generally define the procedure for constructing confidence intervals for any sample statistic as follows:

Let $D_{1}, \ldots, D_{n}$ be a random sample $D$ from a population with an unknown parameter $\theta$. Given a confidence level $(1-\alpha)$, and if $l(D), u(D)$ are computed from sample statistics with the property that:

then we say that $[l(D), u(D)]$ is a $100(1-\alpha) \%$ confidence interval $(\mathrm{CI})$ for $\theta$. Thus, the confidence interval contains the true value of the parameter $\theta$ with some known probability $(1-\alpha)$.

To illustrate, consider an example for computing the confidence interval for $\mu$ for a random sample $D$ from a Normal distribution and a confidence level of $95 \%(\alpha=0.05)$. With the help of Figure 30 and Table 2 we find that: