Optimizing sequence generation in sparse contexts using Entropy Adaptive Dirichlet Smoothing

Hidden Markov Models (HMMs) predict sequences of events based on transition and emission probabilities. They have been widely applied across domains such as medicine, finance, and computing. In text generation, traditional HMMs often produce outputs lacking coherence and accuracy. This study explores the enhancement of text generation using HMMs by introducing Entropy Adaptive Dirichlet Smoothing that adjusts transition probabilities through a novel smoothing technique. We developed multiple text generation models based on Markov chains, each trained on the same dataset. We established an experimental framework where text generation was performed using our Dirichlet-based smoothing technique. Perplexity and Average Conditional Entropy measurements revealed that this innovative model significantly outperformed the original models in shaping probabilities and predicting text sequences. This research demonstrates that, in a small, fractional training context, Entropy Adaptive Dirichlet Smoothing is able to notably improve sequence generation, providing a practical approach that balances relatively simple complexity and performance, without recurring to the use of Neural Networks. This work has been limited by the use of few training iterations in the face of restricted computing resources and the absence of extrinsic benchmarking.