Lesson 7 of 8 · 20 min
Reliability and validity of diagnosis
Calculate per cent agreement between clinicians, tell reliability from validity, explain what improves each, and analyse research on diagnosis, including Rosenhan (1973) and an agreement experiment.
What you will be able to do
- Distinguish inter-rater and test-retest reliability of diagnosis, and calculate per cent agreement.
- Explain what makes a diagnosis valid, and why a reliable diagnosis may not be valid.
- Explain how explicit criteria, structured interviews and training improve reliability.
- Analyse research on diagnosis, including its design, data, conclusions and ethics.
A diagnosis is only useful if two things are true. Different clinicians should reach the same diagnosis for the same person, and that diagnosis should be right: it should describe a real condition the person has, and say something useful about what will help them. The first is reliability. The second is validity. You met both ideas for IQ tests in the last topic; here they apply to the people who diagnose.
Definition
Reliability of diagnosis
Reliability is the consistency of diagnosis.
- Inter-rater reliability: two or more clinicians, assessing the same person independently, give the same diagnosis.
- Test-retest reliability: the same person, assessed again a short time later (before anything has really changed), receives the same diagnosis.
The simplest measure is per cent agreement: .
Worked example
Calculate per cent agreement between two clinicians
Two clinicians independently assess the same 40 fictional people for an anxiety disorder. Their judgements are shown in the table.
| Clinician B: disorder | Clinician B: no disorder | |
|---|---|---|
| Clinician A: disorder | 14 | 5 |
| Clinician A: no disorder | 3 | 18 |
(a) Calculate the per cent agreement. (b) State what this measures. (c) Suggest one way to improve it.
Step 1: Find the agreed cases
The clinicians agree when both say "disorder" (14) or both say "no disorder" (18): . The other are disagreements; check: .
Agreeing that someone does not have a disorder is just as much agreement as agreeing that they do.
Step 2: Calculate the percentage
Step 3: Interpret
This measures inter-rater reliability: how consistently two clinicians diagnose the same people. It says nothing about whether their diagnoses were correct.
Step 4: Improve it
Use explicit criteria with a structured interview, so both clinicians ask the same questions and score answers the same way, and train them together.
Try it yourself first, then check the working one step at a time.
Answer
(a) 80%. (b) Inter-rater reliability. (c) For example, a structured interview based on explicit criteria.
Method
Agreed cases = both "disorder" + both "no disorder".
Agreement = agreed ÷ total × 100%.
Name it as reliability, not validity.
Watch out for
Using only the 14 "both disorder" cases: .
Calling 80% agreement evidence that the diagnoses are valid.
Two clinicians independently assess the same 25 fictional people for a depressive disorder. They both diagnose 9 people, both find no disorder in 12, and disagree about the rest. Calculate their per cent agreement.
What lowers reliability. Vague criteria ("seems persistently low") leave clinicians to fill the gaps with their own judgement. In an unstructured interview, each clinician asks different questions and hears different things. Clinicians differ in training and in what they expect to see. Many disorders share symptoms (poor sleep and poor concentration appear in depression and in anxiety disorders), people often have more than one disorder (comorbidity), and a person may describe their experiences differently on different days.
What raises it. Since DSM-III in 1980, manuals have given explicit criteria: how many symptoms, for how long, and how much they must interfere. A structured interview asks every person the same questions in the same order and scores the answers against those criteria. Training clinicians together reduces differences between them. Even so, when the DSM-5 criteria were field-tested in 2013 with clinicians' usual unstructured interviews, agreement was good or very good for most of the 23 diagnoses tested, but questionable or unacceptable for 9 of them (Regier et al. 2013).
Change how clear the criteria are, and whether the clinicians share a bias. Agreement is reliability; the follow-up checks validity.
Definition
Validity of diagnosis
Validity is whether a diagnosis is accurate and meaningful: whether the category describes a real, distinct condition, and whether it is right for this person. A valid diagnosis helps explain the person's difficulties and predicts their likely course and response to treatment (prognosis). A diagnosis that later assessments confirm, and that leads to treatment that works, is evidence of validity.
Reliable is not the same as valid. Two clinicians who share the same mistaken belief can agree every time and both be wrong. Reliability is necessary for validity (a diagnosis that changes with whoever makes it cannot be accurate), but it is not sufficient.
What threatens validity. Categories with blurry edges, where one disorder shades into another or into ordinary experience. Heterogeneity: depression needs any five of nine symptoms, so two people with the same diagnosis can share as few as one symptom, and may have different causes, courses and needs. Comorbidity, which raises the question of whether two disorders are really separate. And bias, which you examine in the next lesson.
Two psychiatrists independently assess 30 fictional patients and agree on the diagnosis for 28 of them. A year later, careful follow-up shows that 10 of the agreed diagnoses were wrong. What does this show about the diagnoses?
Worked example
Agreement is not accuracy
In a fictional study, two psychiatrists who trained together at the same clinic independently assess 20 people referred for low mood. They agree on 18 of the 20 diagnoses. A year later, an independent team carries out a full follow-up assessment and finds that 5 of those 18 agreed diagnoses were wrong.
(a) Calculate the per cent agreement. (b) Calculate the percentage of all 20 people whose agreed diagnosis was confirmed by the follow-up. (c) Explain what the results show about the reliability and validity of the diagnoses, and suggest a reason.
Step 1: Per cent agreement
Step 2: Confirmed diagnoses
Of the 18 agreed diagnoses, were confirmed. As a percentage of all 20 people: .
Step 3: Reliability
90% agreement is high: the diagnoses were reliable.
Step 4: Validity
But more than a quarter of the agreed diagnoses (, 28%) were wrong, and only 65% of people had a confirmed agreed diagnosis, so validity was much weaker.
Step 5: Suggest a reason
The psychiatrists trained together, so they may share the same habits or biases (for example, a tendency to diagnose depression whenever mood is low). A shared bias makes clinicians agree with each other without making them right.
Reliability can be high when errors are shared. Validity needs a check against something independent, such as a follow-up.
Try it yourself first, then check the working one step at a time.
Answer
(a) 90%. (b) 65%. (c) The diagnoses were reliable but not very valid: the psychiatrists agreed, but many agreed diagnoses were wrong, possibly because of a bias they shared from training together.
Method
Agreement measures reliability; confirmation by an independent follow-up measures validity.
Look for a reason errors could be shared.
Watch out for
Concluding the diagnoses were valid because the psychiatrists agreed.
Dividing 13 by 18 when the question asks for a percentage of all 20 people.
Research on diagnosis: Rosenhan (1973). The most famous study of diagnosis is David Rosenhan's "On being sane in insane places". Eight people with no history of disorder, including Rosenhan, sought admission to twelve psychiatric hospitals in five US states. Each reported hearing an unclear voice saying words such as "empty", "hollow" and "thud", and otherwise behaved normally and told the truth about their lives. All were admitted and, except in one case, the diagnosis was schizophrenia; each was eventually discharged with schizophrenia "in remission". Their stays ranged from 7 to 52 days (19 days on average). Staff did not detect them, although other patients often did.
In a follow-up, staff at another hospital were told that pseudopatients would try to gain admission over three months. Of 193 patients admitted, staff judged 41 with high confidence to be pseudopatients, and at least one psychiatrist suspected 23. Rosenhan had sent none.
What the study suggested
Diagnosis was unreliable and lacked validity: clinicians could not tell people without a disorder from those with one, and once a label was given, normal behaviour (such as note-taking) was read as a symptom. Expectations changed diagnoses in both directions.
Criticisms and ethics
Hospital staff were deceived and did not consent. Faking a symptom says little about diagnosing people who seek help honestly. Diagnostic criteria were vague in 1973 and are explicit now. And Cahalan (2019) found records that cast doubt on whether the study was carried out as reported, so treat its findings with caution.
Classify the variables, then pick the null and alternative hypotheses. The worked example below analyses data from this design.
Worked example
Analyse an experiment on structured interviews
A researcher randomly allocated 20 pairs of trainee clinicians to two groups. In each pair, the two trainees independently diagnosed the same 20 video-recorded, fictional case interviews. Ten pairs used a structured interview checklist; ten pairs used their usual unstructured approach. The researcher recorded each pair's per cent agreement.
| Group | Per cent agreement for each pair |
|---|---|
| Structured | 90, 85, 95, 80, 90, 85, 90, 95, 85, 80 |
| Unstructured | 70, 65, 75, 60, 80, 70, 65, 75, 70, 60 |
An unpaired t-test gave , .
(a) Identify the independent and dependent variables and the experimental design. (b) Calculate the mean and standard deviation of each group, to 2 decimal places. (c) Draw a conclusion about the hypothesis that structured interviews increase agreement. (d) Evaluate one limitation of the study.
Mean per cent agreement between pairs of trainee clinicians (error bars: ±1 standard deviation). Fictional data.
Step 1: Variables and design
IV: the interview method (structured checklist or unstructured). DV: each pair's per cent agreement on the 20 cases. Different pairs did each condition, randomly allocated: an independent groups design.
Step 2: Means
Structured: . Unstructured: .
Step 3: Standard deviations
Using the sample standard deviation (): structured , unstructured percentage points.
On a calculator, use the sample standard deviation (), which divides by .
Step 4: Conclusion
The structured group's mean agreement was 18.5 percentage points higher, the error bars do not overlap, and (indeed ). So we reject the null hypothesis: the structured checklist was associated with significantly higher agreement, supporting the hypothesis.
Step 5: Evaluate a limitation
Agreement measures reliability only: pairs could agree on wrong diagnoses, so the study does not show that structured interviews make diagnosis more valid. Also, trainees diagnosing recorded fictional cases may not behave like experienced clinicians with real clients, which limits external validity. A follow-up study could compare diagnoses with an expert panel's, with real clinicians.
Try it yourself first, then check the working one step at a time.
Answer
(a) IV: interview method; DV: per cent agreement; independent groups. (b) Structured 87.50% (s = 5.40); unstructured 69.00% (s = 6.58). (c) Significant difference (): reject the null hypothesis; structured interviews increased agreement. (d) For example, agreement is reliability, not validity, and trainees with recorded cases limit generalisation.
Method
Name IV, DV and design from how participants were allocated.
Calculate means and sample standard deviations; keep units (percentage points).
Use to decide about the null hypothesis, then state the conclusion in words.
Limit the conclusion to what the DV measured, and to whom it generalises.
Watch out for
Calling it repeated measures because every pair saw the same 20 cases. Each pair did only one condition.
Saying the study proves structured interviews make diagnoses correct.
Which ethical principle did Rosenhan's (1973) study most clearly breach in its treatment of hospital staff?
Exam tip
Say which one the evidence shows
In a reliability or validity question, name the concept the evidence is about. Agreement between clinicians, or with oneself on retest, is reliability. Agreement with a later, more complete assessment, or a correct prediction of outcome, is evidence of validity. Then say what would improve it: explicit criteria, structured interviews, training, or a follow-up to check accuracy.
Summary
Reliability is consistency: inter-rater (different clinicians agree) and test-retest (the same diagnosis on re-assessment).
Per cent agreement = cases agreed ÷ total cases × 100%, counting both "disorder" and "no disorder" agreements.
Explicit criteria, structured interviews and training raise reliability; vague criteria, unstructured interviews and overlapping symptoms lower it.
Validity is accuracy and meaning: a valid diagnosis explains the person's difficulties and predicts course and treatment response.
Reliability is necessary but not sufficient for validity: clinicians who share a bias can agree and both be wrong.
Rosenhan (1973) suggested diagnosis lacked reliability and validity, but its ethics, its method and its reporting have been strongly criticised.
Finished the lesson?
Try the 3 quick checks above and the lesson completes itself, or mark it complete now.