Multimodal abnormal and dead laying hen detection using transformer-based model

In commercial caged laying hen farming, daily monitoring for abnormal and dead hens is critical. The crowded and low-light conditions often render unimodal images insufficient for accurate detection. To address this, we propose a Transformer-based multimodal abnormal and dead laying hen detection method tailored to actual caged environments. First, we propose a multimodal chicken image fusion network based on the Mamba, names MCIFusion-Mamba. A Mamba-based image fusion network and a fusion loss function that simultaneously considers image similarity, texture, intensity, and saliency objectives are proposed to achieve better spatial perception and retain more complete target semantic information. The fused images provide more distinct features, addressing the challenges of low-light and crowded conditions and improving subsequent detection accuracy. Secondly, based on RT-DETR model, we propose a Transformer-based multimodal abnormal and dead laying hen detection method. We improve the backbone and encoder networks of the RT-DETR based on ResNet-18 (RT-DETR-R18). Specifically, we propose a Deep Feature Pyramid Module to redesign the Cross-scale Feature Fusion structure, which can address the impact of varying laying hen sizes on detection performance. To reduce the computational cost from the quadratic complexity of the Attention-based Intra-scale Feature Interaction (AIFI), we incorporate Efficient Additive Attention. Finally, we introduced the Visual State Space (VSS) block into the backbone network to enhance the CNN backbone’s ability for deep feature information extraction and global context perception. Experimental results demonstrate that MCIFusion-Mamba outperforms other state-of-the-art image fusion networks in both qualitative and quantitative outcomes. Meanwhile, the proposed detection model for abnormal and dead laying hens also outperforms other state-of-the-art object detection models. The dead category can achieve precision, recall, mAP@0.5 and mAP@0.95 of 0.976, 0.958, 0.968 and 0.683, and the abnormal category can achieve precision, recall, mAP@0.5 and mAP@0.95 of 0.981, 0.961, 0.951 and 0.788. Additionally, an ablation experiment has been conducted to verify the effectiveness of the proposed improvements. In summary, this study not only proposes an innovative Mamba-based image fusion network but also introduces a new multimodal detection method that significantly advances Precision Livestock Farming by enabling accurate detection of abnormal and dead hens. This work can serve as a valuable reference for other livestock researchers.