Observer Variation in the Assessment of Outcome in Traumatic Brain Injury

OBJECTIVE: Accurate and consistent outcome assessment is essential to randomized clinical trials. We aimed to explore observer variation in the assessment of outcome in a recently completed trial of dexanabinol in head injury and to consider steps to reduce such variation. METHODS: Eight hundred sixty-one patients with severe traumatic brain injury who were admitted to 86 centers were included in a multicenter, placebo-controlled, Phase III trial. Outcome was assessed at 3 and 6 months postinjury using the extended Glasgow Outcome Scale; standardized assessment was facilitated by the use of a structured interview. Before initiation of trial centers, outcome ratings were obtained for sample cases to establish initial levels of agreement. Training sessions in outcome assessment were held, and problems in assigning outcome were investigated. During the trial, a process of central review was established to monitor performance. Interobserver variation was analyzed using the kappa statistic. RESULTS: Substantial observer variation was found in the rating of sample cases (weighted kappa, 0.72; confidence interval, 0.68-0.75) and in assigning outcome based on completed structured interviews (weighted kappa, 0.61; confidence interval, 0.57-0.64). In the early stages of the trial, a relatively large number of discrepancies (29-37%) were identified on central review. This number declined as the trial progressed and coincided with investigator training and feedback from central review. Centers with higher enrollment rates showed better performance. CONCLUSION: Observer variation in outcome assessment is a significant problem for head injury trials. Consistency can be improved by standardizing procedures, training assessors, and monitoring the quality of assessments and providing feedback to interviewers.