Inter‐ and intra‐rater reliability of two grading systems for equine gastric glandular disease

Background The recommended combined descriptive terminology scoring system for equine gastric glandular disease has poor reliability whereas a published numerical scoring system has substantial reliability. Descriptive terminology appears more affected by clinical experience than numerical scoring. Individual descriptive terminologies have previously correlated with descriptions of severity. Objectives To compare the reliability of two scoring systems across varied clinical experience and between numerical scores and descriptive terms. The hypothesis was that the numerical scoring system would have better reliability regardless of clinical experience and numerical scores would correlate with individual descriptive terms. Study Design Prospective, blinded observer study. Methods Duplicate gastroscopy images of 30 glandular lesions were assessed by 12 raters using both scoring systems; raters were split into four groups: internal medicine specialists, residents, general practitioners and new graduates. Intra‐ and inter‐rater reliability of both systems and individual criteria therein were determined using Krippendorff's alpha. Results Intra and inter‐rater reliability were higher for the numerical scoring system when assessing all raters combined ( α = 0.89, 95% CI 0.86–0.90 and α = 0.67, 95% CI 0.65–0.69 respectively) compared to the combined descriptive terminology system ( α = 0.61, 95% CI 0.55–0.66 and α = 0.09, 95% CI 0.08–0.10 respectively). Intra‐rater reliability was high for all groups with the numerical system, but only high for specialists with the descriptive system. Intra‐rater reliability was more affected in the descriptive scoring system by clinical experience than the numerical scoring system. Inter‐rater reliability was moderate for specialists, residents and recent graduates with the numerical system, but poor for all groups with the descriptive system. There was a large and statistically significant association between numerical score and the score for ‘severity’ ( η 2 _ H = 0.66, 95% CI 0.52–0.75; p < 0.001). Main Limitations Use of still images visualised on individual screens. Conclusions The numerical scoring system was more reliable than the descriptive system regardless of clinical experience. Numerical score and ‘severity’ are strongly associated.