Data Fusion in Multi-View Learning for Earth Observation Applications With Missing Views

Nowadays, more than 9000 satellites orbit and collect data about our planet. These remote observations, regarded as views, have been vital to analyzing and studying complex phenomena on Earth. Thus, the advances in sensor technology and the accessibility of their collected data have increased the research of deep learning models for Earth Observation (EO), such as Multi-View Learning (MVL). However, the integration of this multi-view data, called data fusion, is a challenging process in the EO domain. This is because EO data has a heterogeneous nature with different acquisition modes (e.g. optical and radar), spectral characteristics (e.g. RGB and multi-spectral), and resolutions (e.g. spatio-temporal). This non-standard data goes beyond natural images and benchmark datasets in computer vision, limiting the applicability of modeling approaches and architectures from other domains. Moreover, the lack of systematically available sensor data (i.e. views) covering the same region and period is an inherent problem in the EO domain. This is because data collection occurs under operational constraints in real-world environments, affected by factors like limited spatial coverage, weather conditions, and unexpected errors. These situations beyond our control, as well as human decisions, affect the availability of views at inference time, i.e. when trained models are used to predict new, unseen data. This thesis tackles the challenging problem of missing views at inference time in MVL. In this thesis, two main subjects are addressed. i) MVL modeling, where two research questions are explored: How has data fusion been explored in the EO literature? and What are the best approaches and architectures of MVL?; ii) MVL robustness, which is dragged by the questions: How are MVL models affected by missing data in the EO domain? and How to increase the robustness of MVL models to missing views in the EO domain? In summary, the contributions of this thesis in MVL with EO data are as follows. Provide a conceptual basis of MVL to discuss and compare fusion strategies in the same model framework. Under this framework, an adaptive fusion model is proposed and validated in agricultural applications. This model has the advantage of avoiding saturation of different views used for training. Moreover, the effect of missing views during inference is studied by introducing a configuration to assess robustness, i.e. the decline in performance caused by missing data. Here, the mitigation factors are detached into robustness components and data processing. Finally, four MVL methods are introduced based on the Missing data as Augmentation (MAug) component while employing different fusion strategies and data processing techniques. These models show robustness to different missing views scenarios, including two primary cases: moderate (single-view missing) and extreme (single-view available). In this way, this thesis advances the design of adaptive MVL models capable of handling arbitrary combinations of EO views at inference time.