A multimodal approach to heritage preservation in the context of climate change

This is the authors' abstract. We don't add key points for this paper.

Cultural heritage sites face accelerating degradations due to climate change, yet traditional monitoring relies on unimodal analysis (visual inspection or environmental sensors alone) that fails to capture the complex interplay between environmental stressors and material deterioration. We propose a lightweight multimodal architecture that fuses sensor data (temperature, humidity) with visual imagery to predict degradation severity at heritage sites. Our approach adapts PerceiverIO with two key innovations: (1) simplified encoders (64D latent space) that prevent overfitting on small datasets (37 samples for training, 555 with data augmentation; 13 for validation, and 13 for testing), and (2) Adaptive Barlow Twins loss that encourages modality complementarity rather than redundancy. On data from Strasbourg Cathedral, our model achieves 76.9% accuracy and 77.0% weighted-F1 score on the test set, a 43% improvement over standard multimodal architectures (VisualBERT, Transformer) and 25% over vanilla PerceiverIO. Ablation studies reveal that sensor-only achieves 61.5% while image-only reaches 46.2%, confirming successful multimodal synergy. A systematic hyperparameter study identifies an optimal moderate correlation target (τ = 0.3) that balances alignment and complementarity, achieving 69.2% accuracy compared to other τ values (τ = 0.1/0.5/0.7: 53.8%, τ = 0.9: 61.5%). This work demonstrates that architectural simplicity combined with contrastive regularization enables effective multimodal learning in data-scarce heritage monitoring contexts, providing a foundation for AI-driven conservation decision support systems.