Lesson 2: Bias and procedures that ensure randomness
Evaluate the design of a council survey
SkillsSpotting bias in samplesProcedures that ensure randomness
Worked example
A regional Queensland council wants to estimate the proportion of its 24 000 households that would use a kerbside green-waste bin. It mails a survey to every household in the three suburbs closest to the council offices, with the question
"Would you support a green-waste bin to help protect our environment?"
Of the 3000 surveys mailed, 900 are returned, and 81 % of those say yes.
(a) Identify three sources of bias in this survey and explain the likely effect of each.
(b) The council has a list of all 24 000 households and knows that 9600 are in town, 10 800 in suburbs and 3600 on rural blocks. Design a better procedure to select a sample of 400 households.
Step 1: Look at who could be chosen
Only three suburbs, all near the council offices, are sampled. That is undercoverage: rural households, which may already compost or burn green waste, have no chance of selection. The estimate could be pushed either way, but it can't be trusted to represent the whole council area.
Step 2: Look at who replied
Only replied. That is non-response bias: people who care about green waste are more likely to return the survey, so the 81 % probably overstates support.
Step 3: Look at the question
Linking the bin to "protect our environment" invites agreement. That is response bias from a leading question, which again pushes the yes proportion up. It also doesn't mention any cost.
A neutral wording, such as "Would you use a green-waste bin at a cost of $X per year?", asks about use rather than general approval.
Step 4: Choose a procedure that covers everyone
Stratify by location so each area is fairly represented. In proportion: town , suburbs , rural .
Step 5: Select randomly within each stratum
Number the households in each area, and use a random number generator to choose 160 town, 180 suburban and 60 rural households, ignoring repeats.
Step 6: Reduce non-response and response bias
Ask a neutral question that includes the cost, and follow up households that don't reply (a reminder letter, then a phone call or door knock) so the response rate is high.
Answer
(a) Undercoverage (only three nearby suburbs, so rural households are excluded), non-response bias (only 30 % replied, probably the most interested) and response bias (a leading question). The last two probably push the 81 % too high. (b) A stratified random sample of 160 town, 180 suburban and 60 rural households chosen with random numbers, a neutral question that mentions the cost, and follow-up of non-respondents.
Try a similar one
A fresh question of the same kind, checked as you answer, with hints and a full solution.
Method
Ask three questions: who could be chosen? who actually responded? how was the question asked?
For each source of bias, say which way it pushes the estimate.
Fix the frame with a full list, the selection with random numbers (stratified if groups differ), and the response with neutral wording and follow-up.
Watch out for
Saying "the sample is too small". 900 is plenty; the problem is how it was chosen.
Naming a bias without explaining its effect on the estimate.