Lesson 5 of 7 · 15 min
What 95% confidence really means
Use repeated sampling to see that intervals vary from sample to sample, that most but not all of them capture p, and how to word the meaning of a confidence level correctly.
What you will be able to do
- Explain that confidence intervals vary between samples.
- Interpret a confidence level as the long-run proportion of intervals that contain .
- Recognise incorrect interpretations, and use the binomial distribution to count intervals that capture .
Suppose 40% of all Australian adults have a gym membership, so , but nobody knows it. Twenty research teams each survey their own random sample of 200 adults and each calculates a 95% confidence interval. Will all twenty intervals contain 0.4? Run the experiment yourself.
Draw 100 intervals. Count how many miss , then change the confidence level to 80% and to 99%. What happens to the widths and to the number that miss?
Each sample gives a different , so each interval has a different centre (and a slightly different width, because appears in the standard error). Most intervals cover . A few, from samples whose happened to land far from , miss it. Here is one set of 20 from a simulation.
Twenty simulated 95% intervals from samples of 200 with . Nineteen contain ; interval 16 (red) misses it. In the long run about 1 in 20 miss.
Key idea
The meaning of the confidence level
A 95% confidence level describes the method, not one interval. If we took many random samples of the same size and built a 95% interval from each, about 95% of those intervals would contain and about 5% would not.
Any one calculated interval, such as , either contains or it doesn't. We just don't know which, and we are "95% confident" because the method succeeds 95% of the time.
A 95% confidence interval for the proportion of Queensland Year 12 students who work part-time is . Which statement is correct?
Common mistake
Wordings that lose the mark
- "There is a 95% chance that is in ." The interval is already calculated and is fixed.
- "95% of the population lies in the interval." The interval is about one proportion, not individuals.
- "The interval contains 95% of sample proportions." It describes , not other values.
Safe wording: "We are 95% confident that the proportion of all … is between 0.47 and 0.55, meaning that about 95% of intervals built this way from repeated samples would contain the true proportion."
Fifty research teams each take an independent random sample and calculate a 90% confidence interval for the same population proportion . How many of the fifty intervals would you expect to not contain ?
Worked example
How many of 100 intervals capture p?
A health department runs the same survey in 100 different regions with the same true proportion , taking an independent random sample in each and calculating a 95% confidence interval for .
(a) How many of the intervals would you expect to contain ?
(b) Find the probability that at least 93 of the intervals contain .
(c) A manager says, "With 95% intervals, all 100 should contain ." Comment.
Step 1: Model the count
Each interval contains with probability 0.95, independently, so the number that do is .
Each region is a trial with two outcomes (captures or not), the same success probability, and independent samples.
Step 2: Expected number
.
Step 3: Probability of at least 93
(binomial cdf).
Step 4: Probability that all 100 contain p
Step 5: Comment
The manager is wrong. A 95% confidence level means about 95% of intervals contain , so we expect about 5 to miss. All 100 containing has probability only about 0.006.
Try it yourself first, then check the working one step at a time.
Answer
(a) 95 (b) (c) Wrong: we expect about 5 to miss ; the probability that none miss is about 0.006.
Method
The number of intervals containing is .
Use the binomial mean and cdf.
Watch out for
Computing instead of : "at least 93" includes 93.
Exam tip
When asked to explain variation between intervals
Say three things: different random samples give different values of ; each interval is centred on its own , so the intervals differ; and because the method has confidence level , about of them (most, but not all) contain . Link it to the context and the numbers in the question.
Summary
Different random samples give different values, so their confidence intervals differ.
A confidence level means about of all intervals built this way from repeated samples contain : most, but not all.
A single calculated interval either contains or not; never say "there is a 95% probability that is in this interval".
The number of independent intervals that contain is .
Higher confidence gives wider intervals that miss less often.
Finished the lesson?
Try the 2 quick checks above and the lesson completes itself, or mark it complete now.