Deep Neighborhood-Similarity Preservation Hashing for Cross-Modal Retrieval

Due to low storage cost and fast query efficiency, cross-modal hashing has attracted considerable interest in multi-modal data retrieval. However, existing hashing methods face several challenges: one major challenge arises from the neglect of both local and non-local neighborhood structural relationships within multi-modal information, which makes it difficult to establish fine-grained semantic consistency associations between heterogeneous modalities. Additionally, the imbalance in the number of training samples limits the improvement of retrieval performance. To address these challenges, a Deep Neighborhood-similarity Preservation Hashing (DNsPH) method is proposed for cross-modal retrieval. To obtain the high-order statistical features of images, we first design a Context-aware Cross-layer Bilinear Fusion Network (C2BF-Net), which uses Long Short-Term Memory (LSTM) to model the context-dependent features of different convolutional layers. Furthermore, the image, text, and semantic labels information are fused through an adaptive weighting strategy to reconstruct the joint semantic similarity matrix to explore the fine-grained neighborhood structure between different modalities. Finally, we introduce a multi-similarity loss based on an adaptive margin to mining and weighting informative sample pairs, to alleviate the impact of sample imbalance on model training, and thereby generate more discriminative hash codes. Extensive experiments performed on the MIRFLICKR-25K and NUS-WIDE datasets demonstrate that DNsPH outperforms state-of-the-art cross-modal retrieval applications.