What question did this study set out to answer?

This research aims to develop a novel framework for captioning remote sensing images using self-supervised and hybrid decoding techniques.

March 12, 2026Open Access

Remote Sensing Image Captioning via Self-Supervised DINOv3 and Transformer Fusion

Key Points

This research aims to develop a novel framework for captioning remote sensing images using self-supervised and hybrid decoding techniques.
Developed a DINOv3-based encoder for feature extraction without labeled pretraining.
Utilized a hybrid decoder combining Transformer layers for global context with LSTM layers for sequential caption generation.
Evaluated the model on traditional datasets like RSICD and UCM-Captions, along with advanced datasets such as RSITMD and DisasterM3.
Measured performance using BLEU, CIDEr, ROUGE-L, and METEOR evaluation metrics.
Achieved a performance improvement of 9-12% over conventional CNN and Vision Transformer models.
Demonstrated robust multi-modal understanding in remote sensing image analysis and captioning.
Validation through attention heatmaps confirmed effective identification and description of key spatial elements.

Abstract

Effective interpretation of coherent and usable information from aerial images (e.g., satellite imagery or high-altitude drone photography) can greatly reduce human effort in many situations, both natural (e.g., earthquakes, forest fires, tsunamis) and man-made (e.g., highway pile-ups, traffic congestion), particularly in disaster management. This research proposes a novel encoder–decoder framework for captioning of remote sensing images that integrates self-supervised DINOv3 visual features with a hybrid Transformer–LSTM decoder. Unlike existing approaches that rely on supervised CNN-based encoders (e.g., ResNet, VGG), the proposed method leverages DINOv3’s self-supervised learning capabilities to extract dense, semantically rich features from aerial images without requiring domain-specific labeled pretraining. The proposed hybrid decoder combines Transformer layers for global context modeling with LSTM layers for sequential caption generation, producing coherent and context-aware descriptions. Feature extraction is performed using the DINOv3 model, which employs the gram-anchoring technique to stabilize dense feature maps. Captions are generated through a hybrid of Transformer with Long Short-Term Memory (LSTM) layers, which adds contextual meaning to captions through sequential hidden layer modeling with gated memory. The model is first evaluated on two traditional remote sensing image captioning datasets: RSICD and UCM-Captions. Multiple evaluation metrics like Bilingual Evaluation Understudy (BLEU), Consensus-based Image Description Evaluation (CIDEr), Recall-Oriented Understudy for Gisting Evaluation (ROUGE-L), and Metric for Evaluation of Translation with Explicit Ordering (METEOR), are used to quantify the performance and robustness of the proposed DINOv3 hybrid model. The proposed model outperforms conventional Convolutional Neural Network (CNN) and Vision Transformers (ViT)-based models by approximately 9–12% across most evaluation metrics. Attention heatmaps are also employed to qualitatively validate the proposed model when identifying and describing key spatial elements. In addition, the proposed model is evaluated on advanced remote sensing datasets, including RSITMD, DisasterM3, and GeoChat. The results demonstrate that self-supervised vision transformers are robust encoders for multi-modal understanding in remote sensing image analysis and captioning.

Read Full Paperexternally

Ask AI

Mark Helpful

Bookmark

Relay

View Full Paper