Determining the difficulty of Word Sense Disambiguation

Automatic processing of biomedical documents is made difficult by the fact that many of the terms they are ambiguous. Word Sense Disambiguation (WSD) systems attempt to resolve these ambiguities identify the correct meaning. However, the published literature on WSD systems for biomedical documents considerable differences in performance for different terms. The development of WSD systems often expensive with respect to acquiring the necessary training data. It would therefore be useful be able to predict in advance which terms WSD systems are likely to perform well or badly on. paper explores various methods for estimating the performance of WSD systems on a wide range ambiguous biomedical terms (including ambiguous words/phrases and abbreviations). The methods both supervised and unsupervised approaches. The supervised approaches make use of information labeled training data while the unsupervised ones rely on the UMLS Metathesaurus. The are evaluated by comparing their predictions about how difficult disambiguation will be ambiguous terms against the output of two WSD systems. We find the supervised methods are the predictors of WSD difficulty, but are limited by their dependence on labeled training data. The unsupervised all perform well in some situations and can be applied more widely.