Similarity Transitivity Broken-Aware Multi-Modal Hashing

Due to the low storage cost and fast retrieval speed, multi-modal hashing, which maps the instances with different modal data-views into hash codes, has earned increasing research attention. Most existing supervised multi-modal hashing methods exploit the label information to define the similarities between instance pairs to supervise their training process. However, such methods ignore that the transitivity of their defined similarity has been broken in the multi-label scenarios, i.e., the instance <tex-math notation="LaTeX">x</tex-math> is similar to the instance <tex-math notation="LaTeX">y</tex-math>, and the instance <tex-math notation="LaTeX">z</tex-math> is also similar to the instance <tex-math notation="LaTeX">y</tex-math>, but <tex-math notation="LaTeX">x</tex-math> may be not similar to <tex-math notation="LaTeX">z</tex-math>, which will lead to fluctuations in the model optimization process and damage their retrieval performance. For example, in the first batch with instances <tex-math notation="LaTeX">x</tex-math> and <tex-math notation="LaTeX">y</tex-math> but without <tex-math notation="LaTeX">z</tex-math>, the model will be optimized to make the hash codes of <tex-math notation="LaTeX">x</tex-math> and <tex-math notation="LaTeX">y</tex-math> similar to each other; In the second batch with instances <tex-math notation="LaTeX">z</tex-math> and <tex-math notation="LaTeX">y</tex-math> but without <tex-math notation="LaTeX">x</tex-math>, the model will be optimized to make the hash codes of <tex-math notation="LaTeX">z</tex-math> and <tex-math notation="LaTeX">y</tex-math> similar to each other; In the third batch with the instances <tex-math notation="LaTeX">x</tex-math> and <tex-math notation="LaTeX">z</tex-math> but without <tex-math notation="LaTeX">y</tex-math>, the model will be optimized to make the hash codes of <tex-math notation="LaTeX">z</tex-math> and <tex-math notation="LaTeX">x</tex-math> dissimilar to each other, meanwhile in this process, the hash codes of <tex-math notation="LaTeX">z</tex-math> and <tex-math notation="LaTeX">x</tex-math> may be dissimilar to that of <tex-math notation="LaTeX">y</tex-math> which damage the optimizing results of the first two batches. Therefore, we propose a novel Similarity Transitivity Broken-aware Multi-modal Hashing, called STBMH, to solve this problem by adding a novel regularization loss into the original pair-wise loss. For each instance <tex-math notation="LaTeX">x</tex-math> in a training batch, the regularization loss will take all instances in the training set into account. Extensive experiments on four widely used datasets show that the proposed method achieves better performance than the state-of-the-art baselines on multi-modal retrieval task.