Lesson 7: Reliability and validity of diagnosis

Analyse an experiment on structured interviews

SkillsResearch on diagnosisReliability and validity of diagnosis

Worked example

  • Complex unfamiliar
  • Technology active

A researcher randomly allocated 20 pairs of trainee clinicians to two groups. In each pair, the two trainees independently diagnosed the same 20 video-recorded, fictional case interviews. Ten pairs used a structured interview checklist; ten pairs used their usual unstructured approach. The researcher recorded each pair's per cent agreement.

GroupPer cent agreement for each pair
Structured90, 85, 95, 80, 90, 85, 90, 95, 85, 80
Unstructured70, 65, 75, 60, 80, 70, 65, 75, 70, 60

An unpaired t-test gave t(18)=6.87t(18) = 6.87, p<0.001p < 0.001.

(a) Identify the independent and dependent variables and the experimental design. (b) Calculate the mean and standard deviation of each group, to 2 decimal places. (c) Draw a conclusion about the hypothesis that structured interviews increase agreement. (d) Evaluate one limitation of the study.

1 of 5 steps
  1. Step 1: Variables and design

    IV: the interview method (structured checklist or unstructured). DV: each pair's per cent agreement on the 20 cases. Different pairs did each condition, randomly allocated: an independent groups design.

Try a similar one

A fresh question of the same kind, checked as you answer, with hints and a full solution.

Mean per cent agreement between pairs of trainee clinicians (error bars: ±1 standard deviation). Fictional data.

Method

  1. Name IV, DV and design from how participants were allocated.

  2. Calculate means and sample standard deviations; keep units (percentage points).

  3. Use p<0.05p < 0.05 to decide about the null hypothesis, then state the conclusion in words.

  4. Limit the conclusion to what the DV measured, and to whom it generalises.

Watch out for

  • Calling it repeated measures because every pair saw the same 20 cases. Each pair did only one condition.

  • Saying the study proves structured interviews make diagnoses correct.

Save your progress with a free account