Lesson 6 of 8 · 17 min
Cultural bias in intelligence testing
Evaluate how far intelligence tests are culturally biased, how test makers reduce bias, and how well each theory of intelligence applies across social and cultural contexts.
What you will be able to do
- Explain the main sources of cultural bias in intelligence tests.
- Explain how test makers and psychologists try to reduce bias, and the limits of culture-fair tests.
- Evaluate the degree to which intelligence tests are culturally biased.
- Discuss how well theories of intelligence apply across social and cultural contexts.
Here is an invented test item: "What should you fit to a car's tyres before driving on a snowy mountain road?" A student from Hobart who skis might know at once. A student from Cairns, just as quick a thinker, might never have heard of snow chains. The item rewards experience of a particular way of life, not reasoning, and that is the core of cultural bias.
Definition
Cultural bias
Cultural bias is a systematic difference in how a test works for people from different cultures, so that their scores reflect familiarity with the test's language, knowledge and ways of doing things as well as the ability it claims to measure.
Cultural bias is a validity problem. A biased test is not measuring the same construct for everyone who sits it, so a score means something different for one group than for another. It can still be perfectly reliable: it can underestimate the same person by the same amount every time.
| Source | What happens | Example |
|---|---|---|
| Language | The test is given in a language or dialect the person does not speak fluently, or its wording favours one dialect | A child who speaks Aboriginal English or Torres Strait Creole at home sits a vocabulary subtest in Standard Australian English |
| Content and knowledge | Items assume knowledge more common in one culture | Questions about snow chains, a particular sport or a particular holiday |
| Format and familiarity | Some people have far more practice with timed, written, multiple-choice puzzles | Students who have sat many school tests are "test-wise" |
| Norms | The norm group does not include people like the person tested | Comparing a recent arrival with norms from people raised and schooled in Australia |
| Testing situation | Answering a stranger's questions, alone and against the clock, is normal in some cultures and uncomfortable or impolite in others | A child taught not to show off, or not to answer an adult's question directly |
| Values | The test rewards what one culture counts as intelligent, such as speed | A culture that values careful, unhurried answers loses marks on timed subtests |
Differences in average scores. When two groups' average scores differ, psychologists treat it first as a question about the test, not about the people: does each item, and the test as a whole, work the same way in both groups? One way to check is to compare people from the two groups who have the same overall score. If they still differ on one item, that item is behaving differently in the two groups and is probably biased. Test makers use this check to remove biased items before a test is published.
The item results of two groups matched on overall ability. A fair item is passed equally often by both.
Worked example
Find and explain a biased test item
A test developer trialled five items of a new reasoning test with two groups of Year 7 students from different cultural backgrounds. The groups were matched on their total scores on the rest of the test. The table shows the percentage of each group who answered each item correctly (illustrative data).
| Item | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| Group A (%) | 72 | 65 | 58 | 80 | 44 |
| Group B (%) | 70 | 63 | 31 | 78 | 46 |
Item 3 asks which of four winter sports uses a toboggan.
(a) Identify the item most likely to be culturally biased, using the data. (b) Explain why it is biased, and evaluate what the developer should do.
Step 1: Find the differences
Differences (A − B): item 1, 2; item 2, 2; item 3, 27; item 4, 2; item 5, −2 percentage points.
Step 2: Identify the outlier
Items 1, 2, 4 and 5 differ by about 2 points, as expected for groups matched on ability. Item 3 differs by 27 points.
Step 3: Explain why matching matters
Because the groups are matched on overall ability, a fair item should be passed about equally often by both. A large gap means item 3 measures something besides reasoning.
Step 4: Explain the bias
Item 3 depends on knowledge of winter sports, which is more common in some cultures and climates. It rewards experience, not reasoning, so it lowers the test's construct validity for Group B.
Step 5: Evaluate the fix
The developer should remove or rewrite item 3 so it tests the same reasoning with content both groups know, then trial it again. Removing items like this reduces bias, but cannot make the test culture-free: the test's format, language and norms can still favour one group.
Try it yourself first, then check the working one step at a time.
Answer
(a) Item 3 (27-point gap; the others differ by about 2). (b) It depends on cultural knowledge of winter sports rather than reasoning, so it is biased against Group B and lowers construct validity. It should be removed or rewritten and re-trialled, though no test can be made completely culture-free.
Method
With groups matched on ability, compare pass rates item by item.
A large gap on one item suggests bias in that item.
Explain what besides the ability the item measures.
Evaluate the remedy and its limits.
Watch out for
Choosing the hardest item (item 5) instead of the one with the biggest gap between groups.
Treating a gap between matched groups as a difference in ability.
Which of these is the clearest evidence that a particular test item is culturally biased?
Reducing bias
- Culture-reduced tests use non-verbal items. Raven's Progressive Matrices (1938) asks the person to choose the piece that completes a pattern of shapes; Cattell's Culture Fair Intelligence Test (1949) uses similar puzzles.
- Non-verbal scores: the SB5 gives a nonverbal IQ and the WISC-V a nonverbal index, built from tasks that need little spoken language.
- Local norms, such as the Australian and New Zealand edition of the WISC-V.
- Testing in the person's first language, with a trained interpreter where needed, and translated tests that have been re-normed.
- A culturally competent psychologist who builds rapport, explains the purpose, consults with the person's family and community, and interprets the score alongside other evidence.
Note
Culture-fair is not culture-free
Even pattern puzzles are a cultural product. Children who have done years of school worksheets have practised exactly this kind of abstract, timed problem. Scores on Raven's matrices rose more over the twentieth century than scores on almost any other test (the Flynn effect), which shows how much they depend on experience. No test can measure intelligence free of culture.
Note
Assessing First Nations peoples
Most standardised tests were developed and normed with populations that included few Aboriginal peoples and Torres Strait Islander peoples, and many First Nations children grow up speaking more than one language or dialect. Guidance for Australian psychologists stresses cultural consultation, involving family and community, considering language, and treating a score from a standardised test with great caution. A low score in these conditions may say more about the test's fit than about the person.
Do theories of intelligence apply across cultures?
Theories carry their culture's idea of intelligence with them. When researchers have asked people in different communities what makes someone intelligent, many answers in African and Asian communities included being socially responsible, cooperative, respectful and wise, not only quick at solving puzzles. A theory built on Western school tasks may not describe what intelligence means in those communities.
| Approach | Applies well because… | Applies less well because… |
|---|---|---|
| Psychometric | Positive correlations among mental tests (g) are found in many countries | Tests must be translated, adapted and re-normed; they reward school-type, timed tasks |
| Information processing | Speed and working memory tasks need little language or knowledge | Not every culture values speed; people less familiar with laboratory tasks may perform below their ability |
| Sternberg | Practical intelligence is defined as success in your own context, so it expects intelligence to look different in different places | Its measures of tacit knowledge must be rebuilt for each context |
| Gardner | Values abilities, such as interpersonal and naturalist, that many cultures prize | Which abilities count as intelligences is itself a cultural choice |
| Goleman | Recognising and managing emotions matters everywhere | Rules about which emotions to show, and how, differ between cultures, so the "correct" emotional response differs too |
Social contexts matter as well as cultural ones. Access to schooling, books, nutrition and health care differs with family income and with remoteness, and each affects test performance. A test normed on mostly urban, schooled children may disadvantage children with less access to schooling, whatever their culture.
Which theory most directly expects that what counts as intelligent behaviour will differ from one cultural context to another?
Exam tip
Evaluating the degree of bias
"Evaluate the degree" wants a judgement with evidence on both sides. Biased: language and content favour the culture of the test makers; norms may not represent the person; the testing situation and the value placed on speed differ between cultures. Less biased than they were: items are screened for bias, there are non-verbal scores and local norms, and skilled psychologists interpret scores in context. Judgement: current tests are much less biased than early ones, but no test is culture-free, so a score from someone outside the norm group must be interpreted with care.
Summary
Cultural bias is a validity problem: the test measures cultural familiarity as well as the ability it claims to measure.
Bias comes from language, content, format, norms, the testing situation and the values a test rewards.
An item is probably biased if groups matched on overall ability pass it at different rates; a difference in averages is a reason to check the test, not evidence about any group's ability.
Culture-reduced tests (Raven's matrices), non-verbal scores, local norms, first-language testing and culturally competent psychologists reduce bias, but no test is culture-free.
Each theory reflects its culture's idea of intelligence; Sternberg's practical intelligence is the most explicitly contextual.
Finished the lesson?
Try the 2 quick checks above and the lesson completes itself, or mark it complete now.