The measurement of term importance in automatic indexing

Abstract The frequency characteristics of terms in the documents of a collection have been used as indicators of term importance for content analysis and indexing purposes. In particular, very rare or very frequent terms are normally believed to be less effective than medium‐frequency terms. Recently automatic indexing theories have been devised that use not only the term frequency characteristics but also the relevance properties of the terms. The major term‐weighting theories are first briefly reviewed. The term precision and term utility weights that are based on the occurrence characteristics of the terms in the relevant, as opposed to the nonrelevant, documents of a collection are then introduced. Methods are suggested for estimating the relevance properties of the terms based on their overall occurrence characteristics in the collection. Finally, experimental evaluation results are shown comparing the weighting systems using the term relevance properties with the more conventional frequency‐based methodologies.