Revisiting Embedding Features for Simple Semi-supervised Learning

Recent work has shown success in us-ing continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems, which is re-garded as a simple semi-supervised learn-ing mechanism. However, fundamen-tal problems on effectively incorporating the word embedding features within the framework of linear models remain. In this study, we investigate and analyze three different approaches, including a new pro-posed distributional prototype approach, for utilizing the embedding features. The presented approaches can be integrated into most of the classical linear models in NLP. Experiments on the task of named entity recognition show that each of the proposed approaches can better utilize the word embedding features, among which the distributional prototype approach per-forms the best. Moreover, the combination of the approaches provides additive im-provements, outperforming the dense and continuous embedding features by nearly 2 points of F1 score. 1