Lesson 7: Reliability and validity of diagnosis
Agreement is not accuracy
SkillReliability and validity of diagnosis
Worked example
In a fictional study, two psychiatrists who trained together at the same clinic independently assess 20 people referred for low mood. They agree on 18 of the 20 diagnoses. A year later, an independent team carries out a full follow-up assessment and finds that 5 of those 18 agreed diagnoses were wrong.
(a) Calculate the per cent agreement. (b) Calculate the percentage of all 20 people whose agreed diagnosis was confirmed by the follow-up. (c) Explain what the results show about the reliability and validity of the diagnoses, and suggest a reason.
Step 1: Per cent agreement
Step 2: Confirmed diagnoses
Of the 18 agreed diagnoses, were confirmed. As a percentage of all 20 people: .
Step 3: Reliability
90% agreement is high: the diagnoses were reliable.
Step 4: Validity
But more than a quarter of the agreed diagnoses (, 28%) were wrong, and only 65% of people had a confirmed agreed diagnosis, so validity was much weaker.
Step 5: Suggest a reason
The psychiatrists trained together, so they may share the same habits or biases (for example, a tendency to diagnose depression whenever mood is low). A shared bias makes clinicians agree with each other without making them right.
Reliability can be high when errors are shared. Validity needs a check against something independent, such as a follow-up.
Answer
(a) 90%. (b) 65%. (c) The diagnoses were reliable but not very valid: the psychiatrists agreed, but many agreed diagnoses were wrong, possibly because of a bias they shared from training together.
Try a similar one
A fresh question of the same kind, checked as you answer, with hints and a full solution.
Method
Agreement measures reliability; confirmation by an independent follow-up measures validity.
Look for a reason errors could be shared.
Watch out for
Concluding the diagnoses were valid because the psychiatrists agreed.
Dividing 13 by 18 when the question asks for a percentage of all 20 people.