Multimodal AI for plant stress classification across multiple plant species

Plant stress monitoring is invaluable in realizing sustainable agriculture because it enables the people practicing it to take early measures to counteract losses in yield caused by environmental stressors like drought and nutrient deficiencies, as well as caused by pathogen infections. The proposed study presents a new Multi-Modal Vision Transformer (MMViT) architecture that is designed to combine both thermal and RGB imagery to take detection to the next level. In order to support this methodology and further studies, we are now publicly releasing a new collection of synchronized thermo-RGB image pairs of stressed and healthy plants, collected both in controlled settings and in the field. The data is labeled to differentiate various stress phenotype and contains over 4286 of images, and hence forms a substantial platform to evaluate multimodal plant phenotyping methods. Empirical evaluations indicate that MMViT model achieves a general classification of 94.3% when using the two modalities, which is better than the single-modality ViT used on the thermal images (85.5%) and the RGB images (93.3%). These experimental results emphasize the performance of multimodal fusion whereby the other spectral cues are used to complement a stress classification. The described framework, together with the useful dataset, will contribute to the advancement of precision agriculture as it is an open and data-driven instrument to monitor plant health automatically.