Joint-Semantics Multi-Similarity Hashing for Cross-Modal Retrieval

Recently, cross-modal hashing has attracted much attention in large-scale image retrieval scenarios. However, most existing methods ignore the potential higher-order relationships and label semantic information between heterogeneous modality data. Besides, the imbalanced training samples could bias the learning process in most classes and affect the retrieval performance. To solve the above problems, we proposed a Joint-semantics Multi-Similarity Hashing method for cross-modal retrieval (JMSH). We first construct a joint semantic similarity matrix, which supervises hash learning by integrating multi-modal features and semantic labels. This method generates higher-order semantic features that maintain semantic correlation effectively. Then, we propose a multi-similarity loss based on adaptive margin, which can collect and weight informative pairs efficiently and accurately, thus producing more discriminative hashing code and improving retrieval performance. Extensive experiments on two benchmark datasets show the superiority of JMSH in cross-modal retrieval tasks.