Difficulty Guided Image Retrieval Using Linear Multiple Feature Embedding

Existing image retrieval systems suffer from a performance variance for different queries. Severe performance variance may greatly degrade the effectiveness of the subsequent query-dependent ranking optimization algorithms, especially those that utilize the information mined from the initial search results. In this paper, we tackle this problem by proposing a query difficulty guided image retrieval system, which can predict the queries' ranking performance in terms of their difficulties and adaptively apply ranking optimization approaches. We estimate the query difficulty by comprehensively exploring the information residing in the query image, the retrieval results, and the target database. To handle the high-dimensional and multi-model image features in the large-scale image retrieval setting, we propose a linear multiple feature embedding algorithm which learns a linear transformation from a small set of data by integrating a joint subspace in which the neighborhood information is preserved. The transformation can be effectively and efficiently used to infer the subspace features of the newly observed data in the online setting. We prove the significance of query difficulty to image retrieval by applying it to guide the conduction of three retrieval refinement applications, i.e., reranking, federated search, and query suggestion. Thorough empirical studies on three datasets suggest the effectiveness and scalability of the proposed image query difficulty estimation algorithm, as well as the promising of the image difficulty guided retrieval system.