Lesson 7: Reliability and validity of diagnosis
Analyse an experiment on structured interviews
SkillsResearch on diagnosisReliability and validity of diagnosis
Worked example
A researcher randomly allocated 20 pairs of trainee clinicians to two groups. In each pair, the two trainees independently diagnosed the same 20 video-recorded, fictional case interviews. Ten pairs used a structured interview checklist; ten pairs used their usual unstructured approach. The researcher recorded each pair's per cent agreement.
| Group | Per cent agreement for each pair |
|---|---|
| Structured | 90, 85, 95, 80, 90, 85, 90, 95, 85, 80 |
| Unstructured | 70, 65, 75, 60, 80, 70, 65, 75, 70, 60 |
An unpaired t-test gave , .
(a) Identify the independent and dependent variables and the experimental design. (b) Calculate the mean and standard deviation of each group, to 2 decimal places. (c) Draw a conclusion about the hypothesis that structured interviews increase agreement. (d) Evaluate one limitation of the study.
Step 1: Variables and design
IV: the interview method (structured checklist or unstructured). DV: each pair's per cent agreement on the 20 cases. Different pairs did each condition, randomly allocated: an independent groups design.
Step 2: Means
Structured: . Unstructured: .
Step 3: Standard deviations
Using the sample standard deviation (): structured , unstructured percentage points.
On a calculator, use the sample standard deviation (), which divides by .
Step 4: Conclusion
The structured group's mean agreement was 18.5 percentage points higher, the error bars do not overlap, and (indeed ). So we reject the null hypothesis: the structured checklist was associated with significantly higher agreement, supporting the hypothesis.
Step 5: Evaluate a limitation
Agreement measures reliability only: pairs could agree on wrong diagnoses, so the study does not show that structured interviews make diagnosis more valid. Also, trainees diagnosing recorded fictional cases may not behave like experienced clinicians with real clients, which limits external validity. A follow-up study could compare diagnoses with an expert panel's, with real clinicians.
Answer
(a) IV: interview method; DV: per cent agreement; independent groups. (b) Structured 87.50% (s = 5.40); unstructured 69.00% (s = 6.58). (c) Significant difference (): reject the null hypothesis; structured interviews increased agreement. (d) For example, agreement is reliability, not validity, and trainees with recorded cases limit generalisation.
Try a similar one
A fresh question of the same kind, checked as you answer, with hints and a full solution.
Mean per cent agreement between pairs of trainee clinicians (error bars: ±1 standard deviation). Fictional data.
Method
Name IV, DV and design from how participants were allocated.
Calculate means and sample standard deviations; keep units (percentage points).
Use to decide about the null hypothesis, then state the conclusion in words.
Limit the conclusion to what the DV measured, and to whom it generalises.
Watch out for
Calling it repeated measures because every pair saw the same 20 cases. Each pair did only one condition.
Saying the study proves structured interviews make diagnoses correct.