A Multimodal Vision-Based Fish Environment and Growth Monitoring in an Aquaculture Cage

Fish condition detection, including the identification of feeding desire, biological attachments, fence breaches, and dead fishes, has become an important research frontier in fishery aquaculture. However, perception in underwater conditions is less satisfactory and remains a tricky problem. Firstly, we have developed a multimodal dataset based on Neuromorphic vision (NeuroVI) and RGB images, encompassing challenging fishery aquaculture scenarios. Within the fishery aquaculture dataset, a spike neural network (SNN) method is designed to filter NeuroVI images, and the sift feature points are leveraged to select the optimal image. Next, we propose a dual-image cross-attention learning network that achieves scene segmentation in a fishery aquaculture cage. This network comprises double-channels feature extraction and guided attention learning modules. In detail, the feature matrix of NeuroVI images serves as the query matrix for RGB images, generating attention for calculating key and value matrices. Then, to alleviate the computational burden of the dual-channel network, we replace dot-product multiplication with element-wise multiplication, thereby reducing the computational load among different matrices. Finally, our experimental results from the fishery cage demonstrate that the proposed method achieves the state-of-the-art segmentation performance in the management process of fishery aquaculture.