Lesson 7: Reliability and validity of diagnosis
Calculate per cent agreement between two clinicians
SkillReliability and validity of diagnosis
Worked example
Two clinicians independently assess the same 40 fictional people for an anxiety disorder. Their judgements are shown in the table.
| Clinician B: disorder | Clinician B: no disorder | |
|---|---|---|
| Clinician A: disorder | 14 | 5 |
| Clinician A: no disorder | 3 | 18 |
(a) Calculate the per cent agreement. (b) State what this measures. (c) Suggest one way to improve it.
Step 1: Find the agreed cases
The clinicians agree when both say "disorder" (14) or both say "no disorder" (18): . The other are disagreements; check: .
Agreeing that someone does not have a disorder is just as much agreement as agreeing that they do.
Step 2: Calculate the percentage
Step 3: Interpret
This measures inter-rater reliability: how consistently two clinicians diagnose the same people. It says nothing about whether their diagnoses were correct.
Step 4: Improve it
Use explicit criteria with a structured interview, so both clinicians ask the same questions and score answers the same way, and train them together.
Answer
(a) 80%. (b) Inter-rater reliability. (c) For example, a structured interview based on explicit criteria.
Try a similar one
A fresh question of the same kind, checked as you answer, with hints and a full solution.
Method
Agreed cases = both "disorder" + both "no disorder".
Agreement = agreed ÷ total × 100%.
Name it as reliability, not validity.
Watch out for
Using only the 14 "both disorder" cases: .
Calling 80% agreement evidence that the diagnoses are valid.