Lesson 7 of 8 · 20 min

Reliability and validity of diagnosis

Calculate per cent agreement between clinicians, tell reliability from validity, explain what improves each, and analyse research on diagnosis, including Rosenhan (1973) and an agreement experiment.

What you will be able to do

  • Distinguish inter-rater and test-retest reliability of diagnosis, and calculate per cent agreement.
  • Explain what makes a diagnosis valid, and why a reliable diagnosis may not be valid.
  • Explain how explicit criteria, structured interviews and training improve reliability.
  • Analyse research on diagnosis, including its design, data, conclusions and ethics.

A diagnosis is only useful if two things are true. Different clinicians should reach the same diagnosis for the same person, and that diagnosis should be right: it should describe a real condition the person has, and say something useful about what will help them. The first is reliability. The second is validity. You met both ideas for IQ tests in the last topic; here they apply to the people who diagnose.

Definition

Reliability of diagnosis

Reliability is the consistency of diagnosis.

  • Inter-rater reliability: two or more clinicians, assessing the same person independently, give the same diagnosis.
  • Test-retest reliability: the same person, assessed again a short time later (before anything has really changed), receives the same diagnosis.

The simplest measure is per cent agreement: agreement=cases agreedtotal cases×100%\text{agreement} = \dfrac{\text{cases agreed}}{\text{total cases}} \times 100\%.

Worked example

Calculate per cent agreement between two clinicians

  • Simple familiar

Two clinicians independently assess the same 40 fictional people for an anxiety disorder. Their judgements are shown in the table.

Clinician B: disorderClinician B: no disorder
Clinician A: disorder145
Clinician A: no disorder318

(a) Calculate the per cent agreement. (b) State what this measures. (c) Suggest one way to improve it.

4 steps

Try it yourself first, then check the working one step at a time.

Method

  1. Agreed cases = both "disorder" + both "no disorder".

  2. Agreement = agreed ÷ total × 100%.

  3. Name it as reliability, not validity.

Watch out for

  • Using only the 14 "both disorder" cases: 1440=35%\frac{14}{40} = 35\%.

  • Calling 80% agreement evidence that the diagnoses are valid.

Two clinicians independently assess the same 25 fictional people for a depressive disorder. They both diagnose 9 people, both find no disorder in 12, and disagree about the rest. Calculate their per cent agreement.

in %

What lowers reliability. Vague criteria ("seems persistently low") leave clinicians to fill the gaps with their own judgement. In an unstructured interview, each clinician asks different questions and hears different things. Clinicians differ in training and in what they expect to see. Many disorders share symptoms (poor sleep and poor concentration appear in depression and in anxiety disorders), people often have more than one disorder (comorbidity), and a person may describe their experiences differently on different days.

What raises it. Since DSM-III in 1980, manuals have given explicit criteria: how many symptoms, for how long, and how much they must interfere. A structured interview asks every person the same questions in the same order and scores the answers against those criteria. Training clinicians together reduces differences between them. Even so, when the DSM-5 criteria were field-tested in 2013 with clinicians' usual unstructured interviews, agreement was good or very good for most of the 23 diagnoses tested, but questionable or unacceptable for 9 of them (Regier et al. 2013).

Interactive

Two clinicians, one set of cases

Change how clear the criteria are, and whether the clinicians share a bias. Agreement is reliability; the follow-up checks validity.

Definition

Validity of diagnosis

Validity is whether a diagnosis is accurate and meaningful: whether the category describes a real, distinct condition, and whether it is right for this person. A valid diagnosis helps explain the person's difficulties and predicts their likely course and response to treatment (prognosis). A diagnosis that later assessments confirm, and that leads to treatment that works, is evidence of validity.

Reliable is not the same as valid. Two clinicians who share the same mistaken belief can agree every time and both be wrong. Reliability is necessary for validity (a diagnosis that changes with whoever makes it cannot be accurate), but it is not sufficient.

What threatens validity. Categories with blurry edges, where one disorder shades into another or into ordinary experience. Heterogeneity: depression needs any five of nine symptoms, so two people with the same diagnosis can share as few as one symptom, and may have different causes, courses and needs. Comorbidity, which raises the question of whether two disorders are really separate. And bias, which you examine in the next lesson.

Two psychiatrists independently assess 30 fictional patients and agree on the diagnosis for 28 of them. A year later, careful follow-up shows that 10 of the agreed diagnoses were wrong. What does this show about the diagnoses?

Choose one answer

Worked example

Agreement is not accuracy

  • Complex familiar

In a fictional study, two psychiatrists who trained together at the same clinic independently assess 20 people referred for low mood. They agree on 18 of the 20 diagnoses. A year later, an independent team carries out a full follow-up assessment and finds that 5 of those 18 agreed diagnoses were wrong.

(a) Calculate the per cent agreement. (b) Calculate the percentage of all 20 people whose agreed diagnosis was confirmed by the follow-up. (c) Explain what the results show about the reliability and validity of the diagnoses, and suggest a reason.

5 steps

Try it yourself first, then check the working one step at a time.

Method

  1. Agreement measures reliability; confirmation by an independent follow-up measures validity.

  2. Look for a reason errors could be shared.

Watch out for

  • Concluding the diagnoses were valid because the psychiatrists agreed.

  • Dividing 13 by 18 when the question asks for a percentage of all 20 people.

Research on diagnosis: Rosenhan (1973). The most famous study of diagnosis is David Rosenhan's "On being sane in insane places". Eight people with no history of disorder, including Rosenhan, sought admission to twelve psychiatric hospitals in five US states. Each reported hearing an unclear voice saying words such as "empty", "hollow" and "thud", and otherwise behaved normally and told the truth about their lives. All were admitted and, except in one case, the diagnosis was schizophrenia; each was eventually discharged with schizophrenia "in remission". Their stays ranged from 7 to 52 days (19 days on average). Staff did not detect them, although other patients often did.

In a follow-up, staff at another hospital were told that pseudopatients would try to gain admission over three months. Of 193 patients admitted, staff judged 41 with high confidence to be pseudopatients, and at least one psychiatrist suspected 23. Rosenhan had sent none.

What the study suggested

Diagnosis was unreliable and lacked validity: clinicians could not tell people without a disorder from those with one, and once a label was given, normal behaviour (such as note-taking) was read as a symptom. Expectations changed diagnoses in both directions.

Criticisms and ethics

Hospital staff were deceived and did not consent. Faking a symptom says little about diagnosing people who seek help honestly. Diagnostic criteria were vague in 1973 and are explicit now. And Cahalan (2019) found records that cast doubt on whether the study was carried out as reported, so treat its findings with caution.

Interactive

Design an agreement experiment

Classify the variables, then pick the null and alternative hypotheses. The worked example below analyses data from this design.

Worked example

Analyse an experiment on structured interviews

  • Complex unfamiliar
  • Technology active

A researcher randomly allocated 20 pairs of trainee clinicians to two groups. In each pair, the two trainees independently diagnosed the same 20 video-recorded, fictional case interviews. Ten pairs used a structured interview checklist; ten pairs used their usual unstructured approach. The researcher recorded each pair's per cent agreement.

GroupPer cent agreement for each pair
Structured90, 85, 95, 80, 90, 85, 90, 95, 85, 80
Unstructured70, 65, 75, 60, 80, 70, 65, 75, 70, 60

An unpaired t-test gave t(18)=6.87t(18) = 6.87, p<0.001p < 0.001.

(a) Identify the independent and dependent variables and the experimental design. (b) Calculate the mean and standard deviation of each group, to 2 decimal places. (c) Draw a conclusion about the hypothesis that structured interviews increase agreement. (d) Evaluate one limitation of the study.

Mean per cent agreement between pairs of trainee clinicians (error bars: ±1 standard deviation). Fictional data.

5 steps

Try it yourself first, then check the working one step at a time.

Method

  1. Name IV, DV and design from how participants were allocated.

  2. Calculate means and sample standard deviations; keep units (percentage points).

  3. Use p<0.05p < 0.05 to decide about the null hypothesis, then state the conclusion in words.

  4. Limit the conclusion to what the DV measured, and to whom it generalises.

Watch out for

  • Calling it repeated measures because every pair saw the same 20 cases. Each pair did only one condition.

  • Saying the study proves structured interviews make diagnoses correct.

Which ethical principle did Rosenhan's (1973) study most clearly breach in its treatment of hospital staff?

Choose one answer

Exam tip

Say which one the evidence shows

In a reliability or validity question, name the concept the evidence is about. Agreement between clinicians, or with oneself on retest, is reliability. Agreement with a later, more complete assessment, or a correct prediction of outcome, is evidence of validity. Then say what would improve it: explicit criteria, structured interviews, training, or a follow-up to check accuracy.

Summary

  • Reliability is consistency: inter-rater (different clinicians agree) and test-retest (the same diagnosis on re-assessment).

  • Per cent agreement = cases agreed ÷ total cases × 100%, counting both "disorder" and "no disorder" agreements.

  • Explicit criteria, structured interviews and training raise reliability; vague criteria, unstructured interviews and overlapping symptoms lower it.

  • Validity is accuracy and meaning: a valid diagnosis explains the person's difficulties and predicts course and treatment response.

  • Reliability is necessary but not sufficient for validity: clinicians who share a bias can agree and both be wrong.

  • Rosenhan (1973) suggested diagnosis lacked reliability and validity, but its ethics, its method and its reporting have been strongly criticised.

Finished the lesson?

Try the 3 quick checks above and the lesson completes itself, or mark it complete now.

Lesson 7 of 80% read

Save your progress with a free account