Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning

The focus in machine learning has branched beyond training classifiers on a task to investigating how previously acquired knowledge in a source can be leveraged to facilitate learning in a related target domain, as inductive transfer learning. Three active lines of research have explored transfer learning using neural networks. In weight, a model trained on the source domain is used as an initialization for a network to be trained on the target domain. In deep metric, the source domain is used to construct an embedding that captures structure in both the source and target domains. In few-shot learning, focus is on generalizing well in the target domain based on a limited of labeled examples. We compare state-of-the-art methods from these paradigms and also explore hybrid adapted-embedding methods that use target-domain data to fine tune embeddings constructed from-domain data. We conduct a systematic comparison of methods in a variety domains, varying the number of labeled instances available in the target (k), as well as the number of target-domain classes. We reach three conclusions: (1) Deep embeddings are far superior, compared to weight, as a starting point for inter-domain transfer or model re-use (2) Our methods robustly outperform every few-shot learning and every deep learning method previously proposed, with a mean error reduction of 34% state-of-the-art. (3) Among loss functions for discovering embeddings, the loss (Ustinova & Lempitsky, 2016) is most robust. We hope our results motivate a unification of research in weight transfer, deep metric, and few-shot learning.