Third message understanding evaluation and conference (MUC-3)

The Naval Ocean Systems Center is conducting the third in a series of evaluations of English text analysis systems. The premise on which the evaluations are based is that task-oriented tests enable straightforward comparisons among systems and provide useful quantitative data on the state of the art in text understanding. Furthermore, the data can be interpreted in light of information known about each system's text analysis techniques in order to yield qualitative insights into the relative validity of those techniques as applied to the general problem of information extraction. A dry-run phase of the third evaluation was completed in February, 1991, and the official testing will be done in May, 1991, concluding with the Third Message Understanding Conference (MUC-3). Twelve sites reported results for the dry-run test at a meeting held in February, 1991. All systems are being evaluated on the basis of performance on the information extraction task in a blind test at the end of each phase of the evaluation.