Interobserver agreement when using a four‐point mobility scoring system to score dairy cattle from video footage

Background Our observational study aimed to investigate the interobserver agreement between mobility observers of different research institutes. Methods Interobserver agreement when using a four‐point mobility score was assessed among 10 experienced observers. Video footage comprised of individual clips of 207 unique cows recorded while walking on nine different farms. Gwet's agreement coefficient (AC1) and Fleiss’ kappa were calculated and reported to measure the agreement between observers. Results Lameness (mobility score ≥ 2) prevalence estimates for the video footage sample varied from 36.2% to 57.0% between observers. The pairwise interobserver agreement (Gwet's AC1) ranged from 35.6% to 74.5%. The average observer agreement was 47.8%‐66.9%, with three observers below 60%. Measuring across all observers, overall agreement was poor (Fleiss’ κ = 0.28) despite their high level of experience. Limitations Our study used a small pool of observers with considerable experience in mobility scoring. We also used videos to ensure consistency in the cows presented to the observers, which may not be reflective of real‐world scoring conditions. Conclusions Standardisation procedures for lameness outcome assessments (among researchers and by logical extension, assessors working in the field) urgently require further development.