Implications of inter-observer variability for image-based monitoring of salmon lice (Lepeophtheirus salmonis) in commercial aquaculture

Image-based counting is increasingly used for automated monitoring of parasitic salmon lice ( Lepeophtheirus salmonis ) in salmon aquaculture. Machine-learning models for detection and classification of lice are trained using human annotations and thus ultimately depend on human judgement. Yet the extent to which observers agree in their assessment of image material, particularly under changing environmental conditions, has not been investigated. To address this, image sequences collected under a range of farming conditions were independently assessed for quality by two analysts. Sequences with concordant Good or Fair classifications were subsequently evaluated by three experienced analysts to assess inter-observer agreement in lice counting and its association with sequence quality. Sequence quality assessment showed moderate inter-observer agreement based on Cohen's kappa indicating some variability in categorical assessment. Lice count showed high inter-rater reliability based on intraclass correlation coefficient (ICC), whereas exact agreement showed more variation at the individual sequence level. Analysts disagreed on lice presence or absence in a subset of the sequences, and the exact agreement was higher for adult female than mobile lice. Exact agreement was also higher, and analyst uncertainty was lower in Good compared with Fair sequences, showing that sequence quality should be considered when interpreting agreement in image-based lice counts. Results further suggest that consistency remains high even though analyst uncertainty increased for sequences of reduced quality. The study demonstrates high reproducibility among experienced analysts working under standardized counting guidelines, while also highlighting individual-sequence discrepancies that should be considered when interpreting human-generated reference annotations for automated lice detection.