Statistics, the art and science of learning from data by A. Agresti (fourth edition) – Chapter 9 summary

STEPS FOR PERFORMING A SIGNIFICANCE TEST
A hypothesis is a statement about the population. A significance test is a method for using data to summarize the evidence about a hypothesis. The null hypothesis (H0) is a statement that the parameter takes a particular value (e.g: probability of getting a baby girl: p = 0.482). The alternative hypothesis (Ha) states that the parameter falls in some alternative range of values. A significance test has five steps:

1. Assumptions
   Each significance test has certain assumptions or has certain condition under which it applies (e.g: an assumption is the assumption that random sampling has been used).
2. Hypotheses
   Each significance test has two hypotheses about a population parameter. The null hypothesis and the alternative hypothesis.
3. Test statistic
   The parameter to which the hypotheses refer has a point estimate. A test statistic describes how far that point estimate falls from the parameter value given in the null hypothesis. This is usually measured in number of standard errors between the point estimate and the parameter.
4. P-value
   A probability summary of the evidence against the null hypothesis is used to interpret a test statistic. The P-value is the probability that the test statistic equals the observed value or a value even more extreme. It is calculated by presuming that the null hypothesis is true.
5. Conclusion
   The conclusion of the significance test reports the P-value and interprets what is says about the question that motivated the test.

SIGNIFICANCE TESTS ABOUT PROPORTIONS
The steps for the significance test are the same for proportions. The biggest assumption made here is that the sample size is large enough that the sampling distribution is approximately normal. The hypotheses are the following for significance tests about proportions:

 and or

This is called a one-sided alternative hypothesis, because it has values falling only on one side of the null hypothesis value. A two-sided alternative hypothesis has the form of:

The test statistic of a significance test about proportions is:

 or

The P-value of a test statistic of a significance test about proportions is the left- or right-tail probability of a test statistic value even more extreme than observed. Smaller P-values indicate stronger evidence against the null hypothesis, because the data would be more unusual if the null hypothesis were true. In a two-sides test, the P-value is the probability of a single tail doubled. The significance level is a number such that we reject H0 if the P-value is less than or equal to that number. The most common significance level is 0.05. If the data provide evidence to reject H0 and accept Ha, the data is called statistically significant. If Ha is rejected, this does not mean that H0 is accepted. H0 is not rejected. If H0 is rejected, that does mean that Ha is accepted. In the case of a small sample, a two-sided significance test should be used.

SIGNIFICANCE TEST ABOUT MEANS
The steps for the significance test are the same for means. The biggest assumption made here is that the population distribution is approximately normal. The hypotheses are the following for significance test about means:

 and  or

A two-sided hypothesis for the mean has the form of:

The test statistic of a significance test about means is:

sample mean-null hypothesis mean standard error of sample mean or

The test statistic of significance test about means is called the t statistic and used the t-distribution. Two-sided inferences using the t-distribution are robust against violations of the normal population assumption.

DECISIONS AND TYPES OF ERRORS IN SIGNIFICANCE TEST
There are two types of potential errors:

1. Type I error
   This occurs when H0 is true and H0 is rejected.
2. Type II error
   This occurs when H0 is false and H0 is not rejected.

There are four possible of results in a test:

The significance level is the probability of a type I error. The rejection region is the collection of test statistic values for which a test rejects H0.

LIMITATIONS OF SIGNIFICANCE TESTS
A significance test merely indicates whether the particular parameter value in H0 is plausible. If H0 is rejected, it tells us that H0 is plausible, but the significance test does not tell us which values are plausible. There are several possible misinterpretations of significance tests:

1. “Do not reject H0” does not mean “accept H0”.
2. Statistical significance does not mean practical significance.

3. The P-value cannot be interpreted as the probability that H0 is true.
4. Some tests may be statistically significant by chance.
5. True effects may not be as large as initial estimates reported by the media

THE LIKELIHOOD OF A TYPE II ERROR
The probability of a type I error is the significance level of the test. Each value in Ha has its own probability of a type II error. The probability of a type II error increases when the true parameter value moves closer to H0. The probability of a type II error decreases as the parameter value moves farther into the Ha values and away from the H0 value and as the sample size increases.

The probability of rejecting H0 when it is false is called the power of the test. The formula for the power of the test is the following:

The higher the power, the better. The best tests have high power and small significance levels.