ISMSFuse: Multi-modal fusing recognition algorithm for rice bacterial blight disease adaptable in edge computing scenarios

Frequent occurrences of pests and diseases in farmland pose a severe threat to crop yield and quality. Various techniques utilized for the automatic detection of diseases and pests are mostly based on a single modality with limited diagnostic information. In this study, a multi-modal classification algorithm, ISMSFuse, was proposed to fuse the image and spectral information. This algorithm combines a MobilenNetV2 lightweight neural network with a traditional low-computational linear SVM method to adapt to edge computing terminals. The spectral information was extracted with non-imaging spectral signals to reduce the acquisition costs and hardware requirements. A proprietary multi-mode rice bacterial blight dataset, denoted as ImgS-RBB2022, was curated specifically for evaluating the algorithm. The effectiveness of the image features extracted by the algorithm model was evaluated by the t-SNE method and the Fisher scores. Finally, comparing the results with only using imaging data, the multi-mode fusion algorithm had a superior recognition accuracy of 98.14%. Through the SHAP method, it was shown that the added spectral features became the most critical feature affecting the output of the algorithm model. To further explore the application potential of the proposed algorithm, we deployed it on a board-based microcomputer Raspberry Pi to test its feasibility in an edge computing circumstance. The proposed method can be applied to various agricultural diseases and pest detection tasks, particularly in the case of foliar-type diseases and pests.