Word Embedding Calculus in Meaningful Ultradense Subspaces

We decompose a standard embedding space into interpretable orthogonal subspaces and a "remainder" subspace. We consider four interpretable subspaces in this paper: polarity, concreteness, frequency and part-of-speech (POS) subspaces. We introduce a new calculus for subspaces that supports operations like "-1 hate = love" and "give me a neutral word for greasy" (i.e., oleaginous). This calculus extends analogy computations like "king-man+woman = queen". For the tasks of Antonym Classification and POS Tagging our method outperforms the state of the art. We create test sets for Morphological Analogies and for the new task of Polarity Spectrum Creation.