Lesson 2 of 7 · 16 min

Bias and procedures that ensure randomness

Recognise the common sources of bias in samples and use random numbers, systematic sampling and stratified sampling to choose a sample fairly.

What you will be able to do

  • Name and explain the common sources of bias in a sample.
  • Select a simple random sample using a calculator or spreadsheet.
  • Use systematic sampling with a random start and proportional stratified sampling.

In 1936 the American magazine Literary Digest mailed straw-poll ballots to about 10 million people, drawn mostly from phone books, car registrations and its own subscriber list. Over 2 million replied, and the poll predicted a landslide win for Alf Landon. Franklin Roosevelt won 46 of the 48 states.

Two million responses, and still badly wrong. In the 1930s, people with phones and cars were wealthier than average, and the people who chose to reply were different again. A sample can be huge and still biased.

Definition

Bias

A sampling method is biased if it systematically over- or under-represents part of the population, so its estimates tend to be off in the same direction every time it is used. Bias comes from the method, so a bigger sample doesn't remove it.

Common sources of bias
SourceWhat happensExample
Convenience (selection) biasThe sample is whoever is easiest to reach.Surveying shoppers at one Gold Coast shopping centre about public transport.
Self-selection (voluntary response) biasPeople choose whether to take part; those with strong views are more likely to.A radio station's online poll about a new stadium.
UndercoverageThe sampling frame leaves out part of the population.Phoning landlines only, which misses many younger adults.
Non-response biasMany of those selected don't respond, and they differ from those who do.A long paper survey that busy shift workers rarely return.
Response biasThe question or setting pushes people towards an answer."Do you agree that our hard-working nurses deserve a pay rise?"

A council wants to know whether residents support a new dog off-leash area. It posts a link to a survey on its Facebook page and analyses the 1450 responses. Which source of bias is most clearly present?

Choose one answer

The cure for bias is to take the choice away from people and give it to chance. Three procedures do this.

Selecting a simple random sample

  1. Get a sampling frame

    A complete list of the population, such as a school's enrolment list or the electoral roll.

  2. Number every member

    Label the members 1,2,…,N1, 2, \dots, N.

  3. Generate random numbers

    On a calculator use RanInt#(1, N) (Casio) or randInt(1, N) (TI); in a spreadsheet use =RANDBETWEEN(1, N).

  4. Ignore repeats

    Keep generating until you have nn different numbers.

  5. Select those members

    The members with those numbers form the sample.

Definition

Systematic and stratified sampling

Systematic sampling. To choose nn from a list of NN, let k=Nnk = \frac{N}{n}. Choose a random starting number from 1 to kk, then take every kkth member after it. It is quick, and random as long as the list has no pattern that repeats every kk places.

Stratified sampling. Split the population into groups (strata) such as year levels or regions, and take a simple random sample from each, in proportion to its size:

number from a stratum=stratum sizeN×n.\text{number from a stratum} = \frac{\text{stratum size}}{N} \times n.

It guarantees each group is fairly represented.

A Townsville school has 400 Year 10, 360 Year 11 and 440 Year 12 students. A stratified random sample of 90 students is chosen in proportion to year level. How many Year 11 students should be in the sample?

Worked example

Select a simple random sample and compare methods

  • Simple familiar

A Cairns surf life saving club has 800 members and wants to survey 40 of them about training times.

(a) Describe how to select a simple random sample of 40 members using a calculator.

(b) Describe how to select a systematic sample of 40 members from the membership list.

(c) The club president suggests surveying the 40 members at Sunday morning training instead. Explain why this may be biased.

6 steps

Try it yourself first, then check the working one step at a time.

Method

  1. Number the sampling frame 11 to NN.

  2. Simple random sample: random integers from 1 to NN, ignoring repeats.

  3. Systematic sample: k=Nnk = \frac{N}{n}; random start from 1 to kk; then every kkth.

  4. To explain bias, say who is left out and which way that pushes the result.

Watch out for

  • Forgetting to say what happens to repeated random numbers.

  • Starting a systematic sample at member 1 every time: the start must be random.

Common mistake

Bigger isn't better if the method is biased

Increasing the sample size reduces the random variation in a statistic, but it does nothing about bias. A random sample of 1000 beats a self-selected sample of 100 000.

Ten repeats of each method when the true proportion is p=0.4p = 0.4. Random sampling scatters around pp (random variation). The biased method is consistent, but consistently wrong.

Worked example

Evaluate the design of a council survey

  • Complex unfamiliar
  • Technology free

A regional Queensland council wants to estimate the proportion of its 24 000 households that would use a kerbside green-waste bin. It mails a survey to every household in the three suburbs closest to the council offices, with the question

"Would you support a green-waste bin to help protect our environment?"

Of the 3000 surveys mailed, 900 are returned, and 81 % of those say yes.

(a) Identify three sources of bias in this survey and explain the likely effect of each.

(b) The council has a list of all 24 000 households and knows that 9600 are in town, 10 800 in suburbs and 3600 on rural blocks. Design a better procedure to select a sample of 400 households.

6 steps

Try it yourself first, then check the working one step at a time.

Method

  1. Ask three questions: who could be chosen? who actually responded? how was the question asked?

  2. For each source of bias, say which way it pushes the estimate.

  3. Fix the frame with a full list, the selection with random numbers (stratified if groups differ), and the response with neutral wording and follow-up.

Watch out for

  • Saying "the sample is too small". 900 is plenty; the problem is how it was chosen.

  • Naming a bias without explaining its effect on the estimate.

Exam tip

Describing a procedure

"Describe a procedure to select a random sample" wants a method another person could follow exactly. Name the sampling frame, how members are numbered, the random number command (with its range), how repeats are handled and how many are chosen. "Pick people at random" earns nothing.

Summary

  • Bias is a systematic error from the sampling method. A larger sample doesn't remove it.

  • Watch for convenience bias, self-selection, undercoverage, non-response and response (wording) bias.

  • Simple random sample: number the sampling frame 11 to NN, generate random integers, ignore repeats.

  • Systematic sample: every kkth member, k=Nnk = \frac{N}{n}, from a random start between 1 and kk.

  • Stratified sample: a random sample from each group, in proportion to the group's size.

Finished the lesson?

Try the 2 quick checks above and the lesson completes itself, or mark it complete now.

Lesson 2 of 70% read

Save your progress with a free account