Interval Estimation

Up until now we have been using Maximum Likelihood Estimation to generate point estimates for our model parameters $\theta$. In particular, we derived expressions for the estimators $\hat{\theta}$ as a function of random sample $D$. You might have noticed by now that most of our maximum likelihood estimators for distributions considered thus far have been taking on similar functional forms:

That is, the estimators above are equal to the sample mean $\bar{X}=\frac{1}{n} \sum_{i=1}^{n} D_{i}$. This is somewhat intuitive, as we had previously shown the true value of these model parameters to be equal to the mean value of the population distribution $E[X]$ (see Table 1 ). Thus, the best approximation to $E[X]$ would be to compute the mean from a random sample $D=\left(D_{1}, D_{2}, \ldots, D_{n}\right)$ drawn from the underlying distribution for $X$.

However, we had discussed earlier that $D$ itself is a random variable that will change with each sampling trial; therefore the value of the estimate will also change for each random sample. Fortunately, for the estimates listed above we already know how the sample means $\bar{X}$ is distributed over many sampling trials (e.g. see sampling distribution of sample means in Figure 29), and we use these sampling distributions to quantify our uncertainty for any estimate we compute.

Sample Mean for Normal Random Variables

We begin by considering the exact solution where our random samples $D$ are drawn from a Normal distribution. Recall from our discussion on the Normal distribution in Chapter 3 that a Z-transformation of a Normal random variable follows a standard Normal distribution:

We had also shown that the sum of Normal random variables is itself a Normal random variable (note that this is exact and not a CLT approximation). Thus, if a maximum likelihood estimator corresponds to the mean $\bar{X}=\frac{1}{n} \sum_{i=1}^{n} D_{i}$ of a sample $\left(D_{1}, D_{2}, \ldots, D_{n}\right)$ from a Normal distribution (as is the case for $\hat{\mu})$, then we can precisely state that our sampling distribution for $\bar{X}(=\hat{\mu})$ is:

Where $\mu$ and $\sigma^{2}$ are the mean and variance of the underlying Normal population distribution, respectively. Thus, we would like to use this sampling distribution to understand how good our estimate $\hat{\mu}=\bar{X}$ is of the true population mean $\mu$. For simplicity sake, let's say that we are given the population variance $\sigma^{2}$. Then we can rearrange the above inequalities to deduce bounds around the unknown parameter $\mu$ based on an estimate $\bar{X}$ :

Thus, by choosing bounds $a$ and $b$, we can integrate the standard normal on the righthand side to compute the probability that our unknown model parameter $\mu$ is contained within the interval $\left[\bar{X}+a \cdot \frac{\sigma}{\sqrt{n}}, \bar{X}+b \cdot \frac{\sigma}{\sqrt{n}}\right]$; we refer to this interval as the confidence interval.

In order to use our sampling distribution to compute the confidence interval, we need to first specify a probability that the interval we construct using an estimate $\bar{X}$ will contain $\mu$; we define this as our confidence level:

Confidence Level: $(1-\alpha)=$ Probability that a confidence interval computed from $\bar{X}$ will contain the true parameter value $\mu$

It is immediately apparent from the above expression that once we have selected a value for the confidence level (i.e. by selecting an $\alpha$ ), then the corresponding values for $a$ and $b$ are also fixed since the standard normal curve is symmetric about $Z=\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$. To see this, consider the confidence intervals around $Z=\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ for confidence levels of $0.95(\alpha=0.05)$ and $0.99(\alpha=0.01)$, as shown in Figure 30. Due to the symmetry of the standard normal curve, $b=-a$ and we can see in Figure 30 that $b=1.96$ for $\alpha=0.05$ and $b=2.575$ for $\alpha=0.01$.

Computing Confidence Interval for Standard Normal using Lookup Tables

A convenient way to compute the confidence interval for a given confidence level is to use the standard Normal CDF lookup tables (see right side of Figure 30). Once we have specified our confidence level $(1-\alpha)$, we just search the standard Normal CDF lookup table for the corresponding $z_{\alpha / 2}(=b=-a)$ such that:

So for the following scenarios we have:

Table 2: Using the standard Normal CDF lookup table to compute confidence interval for a given confidence level $=(1-\alpha)$.

We can therefore equivalently express our confidence interval for $\mu$ as:

NOTE: It is often helpful to think of $z_{\alpha / 2}$ as how many units of standard error $\left(\frac{\sigma}{\sqrt{n}}\right)$ we allow the confidence interval to be. Thus, the confidence interval is $\bar{X} \pm z_{\alpha / 2} \times$ Standard Error.

Figure 30: Computing confidence intervals for parameter estimates that correspond to the mean of a sample of Normal random variables. First we specify the confidence level $(1-\alpha)$, which is represented here as the shaded area under the curve centred around our estimate. For a) we have specified the confidence level to be $95 \%$, corresponding to $\alpha=0.05$; likewise for c) we have specified the confidence level to be $99 \%$, corresponding to $\alpha=0.01$. To find the corresponding confidence interval for a given confidence level, we then use the standard Normal CDF lookup table (right side). For $\alpha=0.05$ in b), we find the bound to be $z_{\alpha / 2}=1.96$; similarly for $\alpha=0.01$ in $\mathbf{d}$ ), we find the bound to be $z_{\alpha / 2}=2.575$.

Sample Mean for Normal Random Variables, Unknown Population Variance $\left(\sigma^{2}\right)$

Thus far we have considered the case where the population variance $\sigma^{2}$ was a known quantity. However, this parameter is seldom known so it is not precisely correct to say that $\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ follows a standard normal distribution. Indeed, in practice we can only ascertain information about $\sigma^{2}$ by looking at the sample $D$ available to us. Recall that we derived an expression for the sample variance $S^{2}$ by correcting for the bias our MLE estimator for $\hat{\sigma^{2}}$ as:

Thus we can use $S$ as an approximation for $\sigma$ in computing the confidence intervals. But how does our uncertainty in $\sigma$ manifest itself in the calculation of these confidence intervals? Intuitively we can appreciate that for large enough sample sizes, $S$ will be a fairly accurate approximation of $\sigma$. But how do we deal with small sample sizes (i.e. $n<100$ )?

The credit for recognising that $\frac{(\bar{X}-\mu)}{\sigma / \sqrt{n}}$ and $\frac{(\bar{X}-\mu)}{S / \sqrt{n}}$ do not follow the same distribution goes to William Sealy Gossett (who graduated from Oxford in 1899 with First Class degrees in Chemistry and Maths). As an employee at the Guinness brewery, Gossett worked on the task of making the art of brewery more scientific. However, for many of his experiments, the sample sizes were only on the order of 4 or 5 , and he knew that there was no possibly way of knowing the exact value of the true population variance $\sigma^{2}$ in his statistical calculations.

In 1908 Gossett published a new PDF that describes the distribution of $\frac{(\bar{X}-\mu)}{S / \sqrt{n}}$. At that time, Guinness forbid any employees to publish papers for confidentiality reasons, so Gossett published these findings under the pen name of "Student". Thus, the resulting PDF was referred to as the Student $t$-distribution (the ' $t$ ' in this name refers to its use in test statistics, which we will analyse in the next chapter).

Formally speaking, for a random sample $D$ of a Normal population distribution with mean $\mu$ and variance $\sigma^{2}$, sampling distribution of the sample mean $\bar{X}$ follows a Student $t$-distribution with $n-1$ degrees of freedom $\left(t_{n-1}\right)$ :

Note here that we define the transform as $T$ instead of $Z$ to distinguish it from the Z-transformation. The Student $t$-distribution, $t_{n-1}$, has a few interesting and important properties, which are illustrated graphically in Figure 31:

1. It is bell-shaped and symmetric about zero, like the Normal distribution, but has "heavier tails" that approach zero probability at a slower rate.
2. Its dispersion varies according the size of the sample, $n$; the smaller the sample size, the greater the dispersion (to reflect the uncertainty in estimating the true variance). 3. As $n$ approaches $\infty, t_{n-1}$ converges to the standard Normal distribution $N(0,1)$ (approximately $n>100$ is close enough to use $N(0,1))$.

Figure 31: Difference between the Student t-distribution, $t_{n-1}$, and standard Normal distribution, $N(0,1)$. As the sample size increases from $n=6$ to $n=21$, we can see that $t_{n-1}$ becomes closer to the standard Normal distribution.

We can analogously define our confidence interval in terms of the Student $t$-distribution for a Normal sample $D$ with unknown population variance $\sigma^{2}$ and sample size $n$ :

Where we note in these tests that the Normal sample $D$ is used to compute both the sample mean and the sample standard deviation:

The corresponding confidence interval bound $t_{\alpha / 2, n-1}$ can be found using a lookup table for the upper percentile of Student $t$-distributions given a corresponding value for $\alpha$ and $n$ in a very similar manner to the standard Normal tables. Indeed, since $t_{n-1}$ is a symmetric distribution, we only need to need to find the value for the bound for the upper tail for a given confidence level $1-\alpha: P\left(T \geq t_{\alpha / 2, n-1}\right)$ (where in contrast for the standard Normal we used the $\left.\operatorname{CDF} P\left(Z \leq z_{\alpha / 2}\right)\right)$.

We illustrate this calculation graphically in Figure 32 for a confidence level of $95 \%(\alpha=0.05)$ and sample sizes $n=6$ and $n=20$. Table 3 below also summarises the confidence interval bounds for additional $\alpha$ values and sample sizes (compare to Table 2):

Table 3: Using the Student $t$-distribution lookup table to compute confidence interval for a given $\alpha$. Note that the 'degrees of freedom' is equal to $n-1$.

Again, it is often helpful to think of $t_{\alpha / 2, n-1}$ as how many units of standard error $\left(\frac{S}{\sqrt{n}}\right)$ we allow the confidence interval to be. Thus, we can write the confidence interval as $\bar{X} \pm t_{\alpha / 2, n-1} \times$ Standard Error, just as we did in the case for the Normal population with known variance. However, we will generally expect a larger confidence interval (i.e. $t_{\alpha / 2, n-1} \geq z_{\alpha / 2}$ ) for a given $\alpha$ due to the additional uncertainty associated with estimating $S$ from the sample data.

For instance, we see in Table 3 for $\alpha=0.05$ that as $n$ increases and our estimation of $S$ from the sample data becomes more accurate (i.e. $S \sim \sigma$ ), $t_{0.025, n-1}$ approaches the standard Normal bound of $z_{0.025}=1.96$. Likewise as $n$ increases for $\alpha=0.01, t_{0.005, n-1}$ approaches the standard Normal bound of $z_{0.005}=2.575$. Using shorthand notation we can write the confidence interval for the estimate as:

Figure 32: Computing confidence intervals for parameter estimates that correspond to the mean of a sample of Normal random variables. First we specify the confidence level $(1-\alpha)$, which corresponds to the area under the curve centred around our estimate. For a) we have specified the area under the curve to be $95 \%$, corresponding to $\alpha=0.05$; likewise for c) we have specified the area under the curve to be $99 \%$, corresponding to $\alpha=0.01$. To find the corresponding values for the bounds around our estimate for a given confidence level, we then use the standard Normal lookup table. For $\alpha=0.05$ in b), we find the bound to be $z_{\alpha / 2}=1.96$; similarly for $\alpha=0.01$ in $\mathbf{d}$ ), we find the bound to be $z_{\alpha / 2}=2.575$.

Sample Mean for Non-Normal Distributions

So far we have considered how to compute confidence intervals for our sample means for a random sample from a Normal distribution. In the case where the population variance $\sigma^{2}$ was known:

And in the case where the population variance is unknown (and must be estimated using the sample variance $\left.S^{2}\right)$ :

But we noted that for large enough $n$ (e.g. $n>100$ ) that the Student $t$-distribution converges to the standard Normal $N(0,1)$, and as a result $z_{\alpha / 2} \approx t_{\alpha / 2, n-1}$ (i.e. the interval becomes the same).

However, we also seen for large sample sizes $n$ that the sample means $\bar{X}$ for a non-Normal population distribution follows a sampling distribution that is Normally distributed. For instance, Figure 29 shows that the sampling distribution of the sample means $\bar{W}$, as computed from random samples of the exponential population distribution $f_{W}(w \mid \hat{\lambda}=0.1)$ - (and therefore $\left.E[W]=\frac{1}{\hat{\lambda}} \equiv \mu, \sqrt{\operatorname{Var}[W]}=\sqrt{\frac{1}{\hat{\lambda}^{2}}} \equiv \sigma\right)$, is itself a Normal distribution given by $N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)$.

Indeed this was the main result of the Central Limit Theorem at the end of Chapter 3: the sample means $\bar{X}$, computed from random samples of any population distribution with mean $\mu$ and variance $\sigma^{2}$, is Normally distributed for large sample sizes $n$ :

And since the CLT only applies to large $n$, we can use the sample standard deviation $S$ to approximate $\sigma^{2}$ without introducing too much further variability into our confidence intervals:

Confidence Intervals for General Sample Statistics

We can generally define the procedure for constructing confidence intervals for any sample statistic as follows:

Let $D_{1}, \ldots, D_{n}$ be a random sample $D$ from a population with an unknown parameter $\theta$. Given a confidence level $(1-\alpha)$, and if $l(D), u(D)$ are computed from sample statistics with the property that:

then we say that $[l(D), u(D)]$ is a $100(1-\alpha) \%$ confidence interval $(\mathrm{CI})$ for $\theta$. Thus, the confidence interval contains the true value of the parameter $\theta$ with some known probability $(1-\alpha)$.

To illustrate, consider an example for computing the confidence interval for $\mu$ for a random sample $D$ from a Normal distribution and a confidence level of $95 \%(\alpha=0.05)$. With the help of Figure 30 and Table 2 we find that:

Thus $\mathbf{9 5 \%}$ confidence interval for $\mu$ is given as:

Again, this means that the true population parameter $\mu$ will be within this interval $95 \%$ of the time. In other words, if we were to conduct 100 sampling trials for $D$, then the computed confidence interval would contain $\mu 95$ times on average.

Example: Sampling Distribution of Sample Variance for a Normal Population

Thus far our sample statistic of interest has been the sample mean $\bar{X}$, since it has been equal to most of our Maximum Likelihood Estimators for the probability models considered in the course:

However, we saw that the MLE of the variance of a Normal population distribution $\left(\sigma^{2}\right)$ was equal to the sample variance $S^{2}$ :

This sample variance $S^{2}$ is a statistic we have not analysed in much detail, but remember that for all statistics we can compute their corresponding sampling distributions.

Let us consider the example covered in lecture, where we sampled 10 random UG student heights $(n=10)$ per day for a year (365 days, or 365 sampling trials). We had used this data to compute the sampling distribution for the sample means (i.e. the distribution of mean heights computed each day), as shown in Figure 33a. We had also shown that this sampling distribution was very accurately approximated by a Normal distribution of $\bar{X}$ using the fact that $E[\bar{X}]=\mu$ and $\operatorname{Var}[\bar{X}]=\sigma^{2} / n$ (i.e the standard error), where $\mu$ is the population mean and $\sigma^{2}$ is the population variance. Generally we do not know $\mu$ and $\sigma$, but temporarily assuming we had these values allowed us to perform a Z-transformation on $\bar{X}$ (Figure 33b) and then derive the confidence interval for $\mu$ as a function of $\bar{X}$ and some multiple of the standard error (see previous sections for details).

In an analogous fashion, using this data can also construct a sampling distribution for the sample variance $S^{2}$, which is shown in Figure 33c. We can see that the sampling distribution for the sample variance generally skews to the right, and so one could guess that a Normal distribution (or any other symmetric distribution) would not be a good model. However, it turns out that if we transform $S^{2}$ by diving by $\sigma^{2} /(n-1)$ then this sampling distribution is modelled by the chi-squared distribution:

Let $D_{1}, \ldots, D_{n}$ be a random sample $D$ from a Normal population with mean $\mu$ and variance $\sigma^{2}$. The ratio of the sampling distribution for the sample variance $S^{2}$ over $\sigma^{2} /(n-1)$ follows a chi-squared distribution of $n-1$ degrees of freedom.

As we did for the Normal and Student $t$-distributions, for a given confidence level $(1-\alpha)$ we can define a confidence interval for $\frac{(n-1) S^{2}}{\sigma^{2}}$ in terms of bounds $\chi_{1-\alpha / 2, n-1}^{2}$ and $\chi_{\alpha / 2, n-1}^{2}$ as follows:

In this case we are interested in constructing a confidence interval for the unknown population variance $\sigma^{2}$, so we take the reciprocal of the above expression:

Finally multiplying each term by $(n-1) S^{2}$ gives us our confidence interval for the unknown population variance $\sigma^{2}$ :

Figure 33: a) Sampling distribution of sample means $\bar{X}$ of 10 student heights for 365 sampling trials. We see that it is normally distributed, and well modelled by $N(E[\bar{X}], \sqrt{\operatorname{Var}[\bar{X}]})$. b) Z-transformation of sampling distribution of sample means to standardise calculation of confidence intervals for unknown population mean $\mu$. c) Sampling distribution of sample variance $S^{2}$ of 10 student heights for 365 sampling trials. We see that it can be modelled by some asymmetric distribution, as it skews to the right. d) The ratio of $S^{2}$ to $\sigma^{2} /(n-1)$ is modelled by a chi-squared distribution for $(n-1)$ degrees of freedom $\left(\chi_{n-1}^{2}\right)$, which will be used to compute confidence intervals for unknown population variance $\sigma^{2}$.

Computing Chi-squared Confidence Bounds

Note that bounds $\chi_{\alpha / 2, n-1}^{2}$ and $\chi_{1-\alpha / 2, n-1}^{2}$ are not equal and opposite, as the chi-squared distribution is not symmetric (as is evident in Figure 33d). To illustrate, let us compute these bounds for the chi-squared distribution in Figure $33\left(\chi_{9}^{2}\right)$ for a $95 \%$ confidence level $(\alpha=0.05)$. Figure 34 a shows where these bounds are for areas of $\alpha / 2$ under the curve of the left and right tails of $\chi_{9}^{2}$.

As with the Student $t$-distribution, we are often given lookup tables in the form of $P(Y>y)=p$ for the chi-squared distribution, where $Y \sim \chi_{n-1}^{2}$. Therefore, to compute the bounds we need to find the following for the lower bound $\chi_{1-\alpha / 2, n-1}^{2}$ (as shown in Figure 34b):

Thus the lower bound is $\chi_{0.975,9}^{2}=2.7$. Analogously we can compute the upper bound $\chi_{\alpha / 2, n-1}^{2}$ (shown in Figure 34c):

And we find the corresponding upper bound is $\chi_{0.025,9}^{2}=19.023$.

Figure 34: a) $95 \%$ confidence interval for chi-squared distribution and $n=10: \chi_{9}^{2}$. b) How to compute lower bound, $\chi_{1-\alpha / 2, n-1}^{2}$, using chi-squared lookup table. c) How to compute upper bound, $\chi_{\alpha / 2, n-1}^{2}$, using chi-squared lookup table.