Lesson 6: Cultural bias in intelligence testing
Find and explain a biased test item
SkillsCultural bias in testingReliability and validity of tests
Worked example
A test developer trialled five items of a new reasoning test with two groups of Year 7 students from different cultural backgrounds. The groups were matched on their total scores on the rest of the test. The table shows the percentage of each group who answered each item correctly (illustrative data).
| Item | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Group A (%) | 72 | 65 | 58 | 80 | 44 |
| Group B (%) | 70 | 63 | 31 | 78 | 46 |
Item 3 asks which of four winter sports uses a toboggan.
(a) Identify the item most likely to be culturally biased, using the data. (b) Explain why it is biased, and evaluate what the developer should do.
Step 1: Find the differences
Differences (A − B): item 1, 2; item 2, 2; item 3, 27; item 4, 2; item 5, −2 percentage points.
Step 2: Identify the outlier
Items 1, 2, 4 and 5 differ by about 2 points, as expected for groups matched on ability. Item 3 differs by 27 points.
Step 3: Explain why matching matters
Because the groups are matched on overall ability, a fair item should be passed about equally often by both. A large gap means item 3 measures something besides reasoning.
Step 4: Explain the bias
Item 3 depends on knowledge of winter sports, which is more common in some cultures and climates. It rewards experience, not reasoning, so it lowers the test's construct validity for Group B.
Step 5: Evaluate the fix
The developer should remove or rewrite item 3 so it tests the same reasoning with content both groups know, then trial it again. Removing items like this reduces bias, but cannot make the test culture-free: the test's format, language and norms can still favour one group.
Answer
(a) Item 3 (27-point gap; the others differ by about 2). (b) It depends on cultural knowledge of winter sports rather than reasoning, so it is biased against Group B and lowers construct validity. It should be removed or rewritten and re-trialled, though no test can be made completely culture-free.
Try a similar one
A fresh question of the same kind, checked as you answer, with hints and a full solution.
Method
With groups matched on ability, compare pass rates item by item.
A large gap on one item suggests bias in that item.
Explain what besides the ability the item measures.
Evaluate the remedy and its limits.
Watch out for
Choosing the hardest item (item 5) instead of the one with the biggest gap between groups.
Treating a gap between matched groups as a difference in ability.