Test Statistic and the Null Distribution

Returning to the potentially-biased coin example, imagine that we have tossed said coin 20 times. It is quite evident that the number of coin flips which come up heads could provide some information regarding the fairness of the coin. That is, we would expect a fair coin to result in approximately 10 heads and 10 tails, while a biased coin would "significantly" deviate from this number (what is regarded as 'significant' will be considered shortly).

We can therefore use the number of coin flips that come up heads as a test statistic. Note that this was also our MLE estimator for the Bernoulli distribution in the previous chapter: $\hat{\theta}=\frac{N(H)}{N(H)+N(T)}=\frac{N(H)}{n}$; indeed, a test statistic is frequently related to an estimator of a population parameter of interest. The distribution of the test statistic under the null hypothesis is called the null distribution. The test statistic, $T$, is a single measure of some property of a sample (e.g. the sample means) that we can use to make assumptions about the null (and alternative) hypotheses.

The null distribution is the probability distribution of the test statistic, given that the null hypothesis is true.

In most cases, we cannot model the null distribution exactly and have to resort to approximate methods; we will discuss these methods in detail later in the chapter. However, in our coin flip example, we can express the null distribution exactly as we know the test statistic ( $T \equiv$ the number of heads) follows as Binomial distribution:

Figure 35: The null distribution for 20 flips of a coin $\left(H_{0} \equiv P(\right.$ heads $\left.)=0.5\right)$.