CrowdVoxel-Net: Voxelized geometry and transformer-based multimodal fusion for activity recognition and anomaly detection in UAV-based crowded scenes

Recognizing human activities and detecting anomalous behaviors in crowded scenes from unmanned aerial vehicle (UAV) imagery remains a challenging problem due to severe occlusions, scale compression, viewpoint distortion, and weak appearance cues, which limit the effectiveness of conventional ground-based and 2D appearance-driven methods. To address these challenges, this paper proposes CrowdVoxel-Net, a geometry-aware multimodal framework for UAV-based activity recognition and anomaly detection, in which a novel aerial silhouette extraction network, SA-Net, is introduced to enable reliable human foreground isolation under complex drone viewpoints. Building upon the extracted silhouettes, pseudo-3D human structure is reconstructed via monocular depth estimation and encoded using voxelized orthogonal depth projections to capture spatial and volumetric cues in a view-consistent manner. Long-range and occlusion-resilient spatiotemporal motion dynamics are modeled using CoTracker, while robust and domain-invariant semantic representations are extracted using a Vision Transformer–enabled DINO self-supervised backbone. In addition, local structural cues are strengthened using a compact keypoint–driven descriptor and a fast cross-frame matching module, improving geometric alignment under challenging aerial viewpoints. These complementary geometric, semantic, spatial, and temporal representations are fused and modeled using a Transformer-based classifier for joint activity recognition and anomaly detection. The proposed framework is evaluated on four challenging benchmarks, including UAV-Human, Drone Action, ShanghaiTech, and CADG, achieving classification accuracies of 53.3% on the full 155-class UAV-Human dataset, 85.2% on its 15-class subset, 91.2% on Drone Action, 95.2% on ShanghaiTech, and 85.7% on CADG, consistently outperforming existing state-of-the-art methods and demonstrating strong robustness and generalization under diverse UAV surveillance scenarios.