Lesson 2 of 7 · 16 min
Bias and procedures that ensure randomness
Recognise the common sources of bias in samples and use random numbers, systematic sampling and stratified sampling to choose a sample fairly.
What you will be able to do
- Name and explain the common sources of bias in a sample.
- Select a simple random sample using a calculator or spreadsheet.
- Use systematic sampling with a random start and proportional stratified sampling.
In 1936 the American magazine Literary Digest mailed straw-poll ballots to about 10 million people, drawn mostly from phone books, car registrations and its own subscriber list. Over 2 million replied, and the poll predicted a landslide win for Alf Landon. Franklin Roosevelt won 46 of the 48 states.
Two million responses, and still badly wrong. In the 1930s, people with phones and cars were wealthier than average, and the people who chose to reply were different again. A sample can be huge and still biased.
Definition
Bias
A sampling method is biased if it systematically over- or under-represents part of the population, so its estimates tend to be off in the same direction every time it is used. Bias comes from the method, so a bigger sample doesn't remove it.
| Source | What happens | Example |
|---|---|---|
| Convenience (selection) bias | The sample is whoever is easiest to reach. | Surveying shoppers at one Gold Coast shopping centre about public transport. |
| Self-selection (voluntary response) bias | People choose whether to take part; those with strong views are more likely to. | A radio station's online poll about a new stadium. |
| Undercoverage | The sampling frame leaves out part of the population. | Phoning landlines only, which misses many younger adults. |
| Non-response bias | Many of those selected don't respond, and they differ from those who do. | A long paper survey that busy shift workers rarely return. |
| Response bias | The question or setting pushes people towards an answer. | "Do you agree that our hard-working nurses deserve a pay rise?" |
A council wants to know whether residents support a new dog off-leash area. It posts a link to a survey on its Facebook page and analyses the 1450 responses. Which source of bias is most clearly present?
The cure for bias is to take the choice away from people and give it to chance. Three procedures do this.
Selecting a simple random sample
Get a sampling frame
A complete list of the population, such as a school's enrolment list or the electoral roll.
Number every member
Label the members .
Generate random numbers
On a calculator use
RanInt#(1, N)(Casio) orrandInt(1, N)(TI); in a spreadsheet use=RANDBETWEEN(1, N).Ignore repeats
Keep generating until you have different numbers.
Select those members
The members with those numbers form the sample.
Definition
Systematic and stratified sampling
Systematic sampling. To choose from a list of , let . Choose a random starting number from 1 to , then take every th member after it. It is quick, and random as long as the list has no pattern that repeats every places.
Stratified sampling. Split the population into groups (strata) such as year levels or regions, and take a simple random sample from each, in proportion to its size:
It guarantees each group is fairly represented.
A Townsville school has 400 Year 10, 360 Year 11 and 440 Year 12 students. A stratified random sample of 90 students is chosen in proportion to year level. How many Year 11 students should be in the sample?
Worked example
Select a simple random sample and compare methods
A Cairns surf life saving club has 800 members and wants to survey 40 of them about training times.
(a) Describe how to select a simple random sample of 40 members using a calculator.
(b) Describe how to select a systematic sample of 40 members from the membership list.
(c) The club president suggests surveying the 40 members at Sunday morning training instead. Explain why this may be biased.
Step 1: Set up the sampling frame
Use the club's membership list of all 800 members and number them 1 to 800.
Every member must be on the list, or some can never be chosen.
Step 2: Generate random numbers
Use
RanInt#(1, 800)(Casio) orrandInt(1, 800)(TI) repeatedly. Ignore any number already chosen, until there are 40 different numbers.Step 3: Select the sample
The 40 members with those numbers form a simple random sample. Every group of 40 members was equally likely to be chosen.
Step 4: Find the sampling interval
, so the systematic sample takes every 20th member.
Step 5: Choose a random start
Generate a random integer from 1 to 20, say 13. The sample is members : that is 40 members.
The random start gives every member the same chance, , of selection.
Step 6: Explain the bias
Members who come to Sunday morning training are the ones for whom the current time suits. Members who can't make that time — perhaps because of work or family — are left out, so the survey would likely overstate support for the current training times. This is convenience (selection) bias.
Try it yourself first, then check the working one step at a time.
Answer
Number the 800 members, generate random integers from 1 to 800 ignoring repeats until 40 are chosen. For a systematic sample, take every 20th member from a random start between 1 and 20. Surveying Sunday trainers is a convenience sample that leaves out members for whom that time doesn't suit, so it is biased towards the current times.
Method
Number the sampling frame to .
Simple random sample: random integers from 1 to , ignoring repeats.
Systematic sample: ; random start from 1 to ; then every th.
To explain bias, say who is left out and which way that pushes the result.
Watch out for
Forgetting to say what happens to repeated random numbers.
Starting a systematic sample at member 1 every time: the start must be random.
Common mistake
Bigger isn't better if the method is biased
Increasing the sample size reduces the random variation in a statistic, but it does nothing about bias. A random sample of 1000 beats a self-selected sample of 100 000.
Ten repeats of each method when the true proportion is . Random sampling scatters around (random variation). The biased method is consistent, but consistently wrong.
Worked example
Evaluate the design of a council survey
A regional Queensland council wants to estimate the proportion of its 24 000 households that would use a kerbside green-waste bin. It mails a survey to every household in the three suburbs closest to the council offices, with the question
"Would you support a green-waste bin to help protect our environment?"
Of the 3000 surveys mailed, 900 are returned, and 81 % of those say yes.
(a) Identify three sources of bias in this survey and explain the likely effect of each.
(b) The council has a list of all 24 000 households and knows that 9600 are in town, 10 800 in suburbs and 3600 on rural blocks. Design a better procedure to select a sample of 400 households.
Step 1: Look at who could be chosen
Only three suburbs, all near the council offices, are sampled. That is undercoverage: rural households, which may already compost or burn green waste, have no chance of selection. The estimate could be pushed either way, but it can't be trusted to represent the whole council area.
Step 2: Look at who replied
Only replied. That is non-response bias: people who care about green waste are more likely to return the survey, so the 81 % probably overstates support.
Step 3: Look at the question
Linking the bin to "protect our environment" invites agreement. That is response bias from a leading question, which again pushes the yes proportion up. It also doesn't mention any cost.
A neutral wording, such as "Would you use a green-waste bin at a cost of $X per year?", asks about use rather than general approval.
Step 4: Choose a procedure that covers everyone
Stratify by location so each area is fairly represented. In proportion: town , suburbs , rural .
Step 5: Select randomly within each stratum
Number the households in each area, and use a random number generator to choose 160 town, 180 suburban and 60 rural households, ignoring repeats.
Step 6: Reduce non-response and response bias
Ask a neutral question that includes the cost, and follow up households that don't reply (a reminder letter, then a phone call or door knock) so the response rate is high.
Try it yourself first, then check the working one step at a time.
Answer
(a) Undercoverage (only three nearby suburbs, so rural households are excluded), non-response bias (only 30 % replied, probably the most interested) and response bias (a leading question). The last two probably push the 81 % too high. (b) A stratified random sample of 160 town, 180 suburban and 60 rural households chosen with random numbers, a neutral question that mentions the cost, and follow-up of non-respondents.
Method
Ask three questions: who could be chosen? who actually responded? how was the question asked?
For each source of bias, say which way it pushes the estimate.
Fix the frame with a full list, the selection with random numbers (stratified if groups differ), and the response with neutral wording and follow-up.
Watch out for
Saying "the sample is too small". 900 is plenty; the problem is how it was chosen.
Naming a bias without explaining its effect on the estimate.
Exam tip
Describing a procedure
"Describe a procedure to select a random sample" wants a method another person could follow exactly. Name the sampling frame, how members are numbered, the random number command (with its range), how repeats are handled and how many are chosen. "Pick people at random" earns nothing.
Summary
Bias is a systematic error from the sampling method. A larger sample doesn't remove it.
Watch for convenience bias, self-selection, undercoverage, non-response and response (wording) bias.
Simple random sample: number the sampling frame to , generate random integers, ignore repeats.
Systematic sample: every th member, , from a random start between 1 and .
Stratified sample: a random sample from each group, in proportion to the group's size.
Finished the lesson?
Try the 2 quick checks above and the lesson completes itself, or mark it complete now.