An investigation into the intra and inter rater scoring reliability of the Addenbrooke’s Cognitive Examination-III

BACKGROUND: Cognitive screening tests are essential to the process of early detection and diagnosis of dementia. The Addenbrooke's Cognitive Examination-III (ACE-III) is one such tool. Rater reliability in scoring is an important psychometric property of all tests. AIMS: To investigate rater accuracy in scoring the ACE-III across different raters and by the same raters at two different time points. A secondary exploratory analysis examined whether scoring accuracy is associated with participants' training and experience with the ACE-III. METHODS: A filmed vignette of the ACE-III being administered to an older adult actor (mock patient) was used to assess scoring accuracy across different raters. The vignette had pre-determined "true" scores. Participants were UK National Health Service staff who routinely administer and score the ACE-III as part of their clinical practice. They were asked to view the filmed vignette and complete an ACE-III scoring sheet. After two months, participants scored the same vignette again. RESULTS AND CONCLUSIONS: At Time 1, 20% of participants' scores matched the true score, with 32% deviating by 3-5 points, and an overall range of 10 points. At Time 2, 24% of scores matched the true score, with 11% deviating by three points, and an overall range of six points. Errors were mainly accounted for by the domains requiring subjective judgements, namely the visuospatial and language subtests. Intra-rater consistency was low to moderate. Previous experience of using the ACE-III, nor previous ACE-III training, led to statisically significant differences in scoring performance. Health professionals should consider these findings when scoring the ACE-III and utilize the ACE-III administration and scoring guide to improve accuracy.