A Comparison of Two Smoothing Methods for Word Bigram Models

A COMPARISON OF TWO SMOOTHING METHODS FOR WORD BIGRAM MODELS Linda Bauman Peto Department of Computer Science University of Toronto Abstract Word bigram models estimated from text corpora require smoothing methods to estimate the probabilities of unseen bigrams. The deleted estimation method uses the formula: Pr (i|j) = lambda fᵢ + (1-lambda) fᵢ|j, where fᵢ and fᵢ|j are the relative frequency of i and the conditional relative frequency of i given j, respectively, and lambda is an optimized parameter. MacKay (1994) proposes a Bayesian approach using Dirichlet priors, which yields a different formula: Pr (i|j) = (alpha/Fⱼ + alpha) mᵢ + (1 - alpha/Fⱼ + alpha) fᵢ|j where Fⱼ is the count of j and alpha and mᵢ are optimized parameters. This thesis describes an experiment in which the two methods were trained on a two-million-word corpus taken from the Canadian Hansard_ and compared on the basis of the experimental perplexity that they assigned to a shared test corpus. The methods proved to be about equally accurate, with MacKay's method using fewer resources.