Urban informal settlements classification via a transformer-based spatial-temporal fusion network using multimodal remote sensing and time-series human activity data

Urban informal settlements (UIS) are high-density population areas with low urban infrastructure standards. UIS classification, which automates identifying UIS, is of great significance for various urban computing tasks. Fast and accurate extraction of UIS has the following difficulties. First, from a high-resolution perspective, the buildings in informal settlement areas are low-floor and dense, with complex spatial relationships. Second, informal settlements’ remote sensing observation characteristics are highly inconspicuous, caused by the shooting angle and imaging environment. Therefore, it is inadequate to classify UIS using only a single remote sensing image modality. Multimodality data with multiple temporal and spatial characteristics provide a prospective opportunity for the more accurate mapping of UIS. Still, there is a lack of relevant works on UIS classification at present. In this paper, we proposed a hybrid Transformer-based spatio-temporal fusion network, namely, STNet, which integrates a proposed PDNet, ResMixer, and Transformer-based spatio-temporal fusing layer to classify UIS using very-high-resolution (VHR) remote sensing images and time-series Tencent population density (TPD) data. Experiments were conducted in Shenzhen City, confirming the superior performance of the proposed STNet and the fusing of spatio-temporal multimodal remote sensing and time-series TPD data. The proposed STNet reached an overall accuracy (OA) of 88.58% and Kappa of 0.7716, with increases of around 1% to 12% and around 0.03 to 0.25 in OA and Kappa, respectively, compared to other models.