Key points are not available for this paper at this time.
The MUC-3 evaluation metrics are measures of performance for the MUC-3 template fill task. Obtaining summary measures of performance necessitates the loss of information about many details of performance. The utility of summary measures for comparison of performance over time and across systems should outweigh this loss of detail. The template fill task is complex because of the varying nature of the fills for each slot and the interdependencies of the slots. The evaluation metrics used in MUC-3 were adapted from traditional measures in information retrieval and signal procesing and were still evolving to fit the more complex data extraction task of MUC-3 when the evaluation was performed. The scoring of the template fill task and the calculation of the metrics used in MUC-3 will be described here. This description is meant to assist in the analysis of the MUC-3 results and in the further evolution of the evaluation metrics.
Nancy Chinchor (Tue,) studied this question.