Inter-rater agreement and variability in visual reading and of [18F]Flutemetamol PET

Abstract Objective: The purpose of this study was to validate the concordance of visual ratings of [18F]flutemetamol amyloid positron emission tomography (PET) images and to investigate the correlation between the agreement of each rater and the Centiloid (CL) scale. Methods: A total of 192 participants, clinically classified as cognitively normal (CN) (n=59), mild cognitive impairment (MCI) (n=65), Alzheimer's disease (AD) (n=55), or non-AD dementia (n=13), participated in this study. Three experts conducted visual ratings of the amyloid PET images for all 192 patients, assigning a confidence level to each rating on a three-point scale (certain, probable, or neither). The positive or negative determination of amyloid PET results was made by majority vote. The CL value was calculated using the CapAIBL pipeline. Results: Overall, 101 images were determined to be positive, while 91 images were negative. Of the 101 positive results, the three raters were in complete agreement for 92 results and in disagreement for 9 results. Of the 91 negative results, the three raters were in complete agreement for 75 results and in disagreement for 16 results. Interrater reliability among the three experts was particularly high, with both Fleiss’ kappa and Conger’s kappa measuring 0.83 (0.76-0.89). The CL value was significantly greater for positive cases with complete agreement and significantly lower for negative cases with complete agreement. Cases with rater disagreement had intermediate CLs. In cases with a high confidence level, the visual ratings judged as positive or negative were in almost complete agreement. However, as confidence levels decreased, experts’ visual ratings became more variable. The lower the confidence level was, the greater the number of cases with disagreement in visual ratings. Conclusions: Three experts independently rated 192 amyloid PET images, achieving a high level of interrater agreement. However, in patients with intermediate amyloid accumulation, visual ratings varied. Therefore, caution should be taken in determining positive and negative decisions in these patients.