Lesson 5 of 7 · 15 min

What 95% confidence really means

Use repeated sampling to see that intervals vary from sample to sample, that most but not all of them capture p, and how to word the meaning of a confidence level correctly.

What you will be able to do

  • Explain that confidence intervals vary between samples.
  • Interpret a confidence level as the long-run proportion of intervals that contain pp.
  • Recognise incorrect interpretations, and use the binomial distribution to count intervals that capture pp.

Suppose 40% of all Australian adults have a gym membership, so p=0.4p = 0.4, but nobody knows it. Twenty research teams each survey their own random sample of 200 adults and each calculates a 95% confidence interval. Will all twenty intervals contain 0.4? Run the experiment yourself.

Interactive

Many samples, many intervals

Draw 100 intervals. Count how many miss pp, then change the confidence level to 80% and to 99%. What happens to the widths and to the number that miss?

Each sample gives a different p^\hat{p}, so each interval has a different centre (and a slightly different width, because p^\hat{p} appears in the standard error). Most intervals cover pp. A few, from samples whose p^\hat{p} happened to land far from pp, miss it. Here is one set of 20 from a simulation.

Twenty simulated 95% intervals from samples of 200 with p=0.4p = 0.4. Nineteen contain pp; interval 16 (red) misses it. In the long run about 1 in 20 miss.

Key idea

The meaning of the confidence level

A 95% confidence level describes the method, not one interval. If we took many random samples of the same size and built a 95% interval from each, about 95% of those intervals would contain pp and about 5% would not.

Any one calculated interval, such as (0.318,0.452)(0.318, 0.452), either contains pp or it doesn't. We just don't know which, and we are "95% confident" because the method succeeds 95% of the time.

A 95% confidence interval for the proportion of Queensland Year 12 students who work part-time is (0.47,0.55)(0.47, 0.55). Which statement is correct?

Choose one answer

Common mistake

Wordings that lose the mark

  • "There is a 95% chance that pp is in (0.47,0.55)(0.47, 0.55)." The interval is already calculated and pp is fixed.
  • "95% of the population lies in the interval." The interval is about one proportion, not individuals.
  • "The interval contains 95% of sample proportions." It describes pp, not other p^\hat{p} values.

Safe wording: "We are 95% confident that the proportion of all … is between 0.47 and 0.55, meaning that about 95% of intervals built this way from repeated samples would contain the true proportion."

Fifty research teams each take an independent random sample and calculate a 90% confidence interval for the same population proportion pp. How many of the fifty intervals would you expect to not contain pp?

Worked example

How many of 100 intervals capture p?

  • Complex unfamiliar
  • Technology active

A health department runs the same survey in 100 different regions with the same true proportion pp, taking an independent random sample in each and calculating a 95% confidence interval for pp.

(a) How many of the intervals would you expect to contain pp?

(b) Find the probability that at least 93 of the intervals contain pp.

(c) A manager says, "With 95% intervals, all 100 should contain pp." Comment.

5 steps

Try it yourself first, then check the working one step at a time.

Method

  1. The number of intervals containing pp is Bin(k,C)\mathrm{Bin}(k, C).

  2. Use the binomial mean and cdf.

Watch out for

  • Computing P(X>93)P(X > 93) instead of P(X≥93)P(X \ge 93): "at least 93" includes 93.

Exam tip

When asked to explain variation between intervals

Say three things: different random samples give different values of p^\hat{p}; each interval is centred on its own p^\hat{p}, so the intervals differ; and because the method has confidence level CC, about CC of them (most, but not all) contain pp. Link it to the context and the numbers in the question.

Summary

  • Different random samples give different p^\hat{p} values, so their confidence intervals differ.

  • A confidence level CC means about CC of all intervals built this way from repeated samples contain pp: most, but not all.

  • A single calculated interval either contains pp or not; never say "there is a 95% probability that pp is in this interval".

  • The number of kk independent intervals that contain pp is Bin(k,C)\mathrm{Bin}(k, C).

  • Higher confidence gives wider intervals that miss less often.

Finished the lesson?

Try the 2 quick checks above and the lesson completes itself, or mark it complete now.

Lesson 5 of 70% read

Save your progress with a free account