Lesson 5 of 8 · 18 min
Reliability and validity of IQ and EQ tests
Discuss how reliable and valid IQ and EQ tests are, using test–retest, internal consistency, content, construct and criterion evidence, and judge when a score can mislead.
What you will be able to do
- Define reliability and validity and name their main types.
- Interpret a reliability coefficient, including a test–retest correlation.
- Discuss the degree to which IQ tests are reliable and valid.
- Consider whether IQ and EQ tests can be misleading or inaccurate.
Imagine a bathroom scale. If it shows a different weight each time you step on it, it is unreliable. If it always shows exactly 3 kg too much, it is reliable but not valid. A psychological test is judged the same way, and the judgement matters most when one score decides a school placement, a diagnosis or a job.
Definition
Reliability and validity
Reliability is the consistency of a test: whether it gives the same results when it is repeated, scored by another person or split into parts.
Validity is whether a test measures what it claims to measure, so that conclusions drawn from its scores are justified.
One fictional person sits the same test many times. Random error scatters the scores; systematic error shifts them all one way.
Reliability
Reliability is reported as a reliability coefficient, a correlation between two sets of scores from 0 (no consistency) to 1 (perfect). For tests used to make decisions about individuals, psychologists want 0.9 or above.
| Type | Question it answers | How it is measured |
|---|---|---|
| Test–retest | Are scores consistent over time? | The same people sit the same test twice, weeks apart; correlate the two sets of scores |
| Internal consistency (split-half) | Do all the items measure the same thing? | Correlate scores on two halves of the test, such as odd and even items |
| Inter-rater | Do different scorers agree? | Two trained psychologists score the same responses; compare their scores |
| Parallel forms | Do two versions of a test give the same result? | The same people sit both versions; correlate the scores |
Illustrative data: ten adults sat the same IQ test twice. The points lie close to a straight line, so test–retest reliability is high.
Worked example
Judge the test–retest reliability of an EQ questionnaire
Eight students completed a self-report emotional intelligence questionnaire (scored out of 50), then completed it again four weeks later (illustrative data).
| Student | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| First score | 34 | 41 | 29 | 38 | 45 | 31 | 36 | 40 |
| Retest score | 36 | 39 | 33 | 35 | 44 | 30 | 38 | 37 |
(a) Use a calculator to find Pearson's r, to 2 decimal places. (b) What type of reliability does r measure here? (c) The p-value for this r is 0.004. Evaluate the questionnaire's reliability.
Step 1: Enter the pairs and find r
Enter first scores as and retest scores as in the calculator's two-variable statistics. .
Step 2: Name the reliability
The same people took the same test on two occasions, so r is a test–retest reliability coefficient.
Step 3: Describe and test r
is a strong positive correlation, and , so it is statistically significant: students who scored high the first time tended to score high again.
Step 4: Compare with the standard for decisions about individuals
Psychologists look for 0.9 or above before using a test to make decisions about individuals. 0.88 is close but just below.
Step 5: Note the limits of the evidence
Only 8 students from one school took part, so the estimate is imprecise. And reliability is not validity: a consistent self-report score could still measure how students see themselves rather than their actual emotional skills.
Try it yourself first, then check the working one step at a time.
Answer
(a) . (b) Test–retest reliability. (c) A strong, significant positive correlation shows good consistency over four weeks, just under the 0.9 usually wanted for decisions about individuals; with only 8 students the estimate is uncertain, and reliability does not show the questionnaire is valid.
Method
Same test, same people, two occasions: test–retest.
Find r, describe it, check p against 0.05.
Compare with 0.9 for individual decisions; comment on sample size and validity.
Watch out for
Calling a high r "valid". It shows consistency, not accuracy.
Saying the scores are reliable because the means are similar (36.75 and 36.5). Reliability is about whether each person keeps their position, which r measures.
How reliable are IQ tests? The full-scale IQs of the WAIS-IV, WISC-V and SB5 have reliability coefficients above 0.9, by internal consistency and test–retest, which is very high for any psychological measure. Single subtests are less reliable, so a psychologist interprets the full-scale and index scores, not one subtest. Two things lower reliability in practice:
- Practice effects: sitting the same test again soon raises scores by a few points, because the tasks are familiar.
- The testing situation: tiredness, illness, anxiety, low motivation or an inexperienced tester add random error on the day.
That error is why reports give a confidence interval: a full-scale IQ of 104 might be reported as "99 to 109 with 95% confidence".
Two psychologists independently score the same child's answers to the vocabulary subtest and give the same scaled score. What does this show?
Validity
A test can be perfectly consistent and still measure the wrong thing, so validity is the harder question.
| Type | Question it answers | Evidence for IQ tests |
|---|---|---|
| Content validity | Do the items cover the whole of what is being measured? | Wechsler scales sample verbal, spatial, reasoning, memory and speed tasks, but leave out creativity, practical and emotional abilities |
| Construct validity | Does the test measure the construct (intelligence), not something else? | Subtests correlate with one another as g predicts; but scores also reflect education, language and familiarity with tests |
| Criterion validity: concurrent | Do scores agree with another measure taken at the same time? | Scores on the WAIS-IV and SB5 correlate highly with each other |
| Criterion validity: predictive | Do scores predict a later outcome? | IQ predicts school results (correlation about 0.5), years of education and job training success |
The evidence makes IQ tests valid measures of the abilities they sample, especially those needed for school, and moderately good predictors of educational outcomes. A correlation of about 0.5 with school results is strong for psychology, but it leaves most of the differences in school results to motivation, teaching, health, family and other abilities. The tests are less valid as measures of intelligence in the wider sense that Gardner, Sternberg and Goleman describe.
Researchers find that students' WISC-V scores in Year 5 correlate with their Year 9 NAPLAN numeracy results. What type of validity does this provide evidence for?
Can IQ and EQ tests mislead?
A test can be reliable and valid on average and still mislead about one person, or be used for something it was not designed for.
Ways a score can mislead
Measurement error near a cut-off
A score of 72 with a confidence interval of 67 to 78 could fall either side of 70. Decisions near a cut-off need other evidence.
Old norms
Average scores rose about 3 points per decade in the twentieth century (the Flynn effect), so a test scored against norms collected 20 years earlier can overestimate a person's IQ by about 6 points.
A single number for an uneven profile
A full-scale IQ of 100 can come from very high verbal and very low processing speed scores. The total hides what matters for support.
A norm group that does not represent the person
Norms from one country, language or culture may not fit a person from another. The next lesson is about this.
The testing situation
Anxiety, fatigue, illness or low motivation lower a score on the day.
EQ tests are more open to these problems. Self-report questionnaires have good internal consistency, but they measure how emotionally skilled people believe they are, and people can present themselves well when a job is at stake. Their scores overlap heavily with personality traits such as extraversion and emotional stability, so their construct validity is questioned. Ability tests of EI are harder to fake, but deciding the "correct" emotional response needs an expert or majority judgement, which may not suit every culture, and some of their parts have lower reliability. EQ scores add only a little to the prediction of work performance once IQ and personality are known.
Select all the reasons a self-report EQ questionnaire used for job selection could give a misleading result.
Common mistake
Reliable is not the same as valid
A test can be reliable without being valid: a vocabulary test in English gives consistent scores for a recent arrival who speaks little English, and consistently underestimates their reasoning. A test cannot be valid without being reliable, because a score that changes at random cannot be accurate.
Exam tip
Discussing "the degree to which"
Answer with a degree, not yes or no: "IQ tests are highly reliable (full-scale coefficients above 0.9) and valid for predicting school results (r ≈ 0.5), but their validity as measures of intelligence in the broader sense is limited because…". Name each type of reliability or validity you use.
Summary
Reliability is consistency (test–retest, internal consistency, inter-rater, parallel forms); validity is measuring what the test claims (content, construct, criterion: concurrent and predictive).
Full-scale IQs on current scales have reliability coefficients above 0.9; subtests are less reliable.
IQ tests predict school results moderately well (r ≈ 0.5) but sample only some abilities.
Scores can mislead through measurement error near cut-offs, old norms (the Flynn effect), uneven profiles, unrepresentative norms and the testing situation.
Self-report EQ can be faked and overlaps with personality; ability-based EQ is harder to fake but its "correct" answers rest on judgement.
Finished the lesson?
Try the 3 quick checks above and the lesson completes itself, or mark it complete now.