Intrinsic explainability of multimodal learning for crop yield simulation

Multimodal learning enables various machine learning tasks to benefit from diverse data sources, effectively mimicking the interplay of different factors in real-world applications, particularly in agriculture. While the heterogeneous nature of involved data modalities may necessitate the design of complex architectures, the model interpretability is often overlooked. In this study, we leverage the intrinsic explainability of Transformer-based models to explain multimodal learning networks, focusing on the task of crop yield simulation at the subfield level. The large datasets used cover various crops, regions, and years, and include four different input modalities: multispectral satellite and weather time series, terrain elevation maps and soil properties. Based on the self-attention mechanism, we estimate feature attributions using two methods, namely the Attention Rollout (AR) and Generic Attention (GA), and evaluate their performance against Shapley-based model-agnostic estimations, Shapley Value Sampling (SVS). Additionally, we propose the Weighted Modality Activation (WMA) method to assess modality attributions and compare it with SVS attributions. Our findings indicate that Transformer-based models outperform other architectures, specifically convolutional and recurrent networks, achieving R 2 scores that are higher by 0. 10 and 0. 04 at the subfield and field levels, respectively. AR is shown to provide more robust and reliable temporal attributions, as confirmed through qualitative and quantitative evaluation, compared to GA and SVS values. Information about crop phenology stages was leveraged to interpret the explanation results in the light of established agronomic knowledge. Furthermore, modality attributions revealed varying patterns across the two methods compared. For instance, SVS estimated the contribution of satellite data at 89. 5% on average, whereas the WMA method provided a significantly lower estimate of 29. 4%. These results call for further analysis and quantitative evaluations. Overall, this work contributes to the growing body of research aiming at enhancing the interpretability of multimodal networks in challenging data-rich contexts in agriculture and remote sensing. The implementation details of the model and attribution methods is available at https: //github. com/hibanajjar998/intrinsicₓaiₘml. • Subfield yield simulation across multiple regions and crop types. • Multimodal learning and Transformers-based processing of multi-source data. • Extensive experiments to explain the model intrinsically. • Proposal of a new method to estimate modality importance scores.