An Overview of Diffusion Models for Text Generation

Given the great success that diffusion models have achieved in generating various types of continuous data, including image, video and audio, there has been a growing interest in the application of these models to text generation. However, the discrete nature of text presents a challenge for diffusion models initially designed for application in a continuous feature space. The two main lines of work that aim to bring together diffusion models and natural language processing are focused on either defining the diffusion process in continuous space by converting discrete tokens to embeddings or defining the diffusion process in discrete space. These recent works attempt to combine diffusion models with leading sequence-to-sequence generation Transformer architecture as well as with existing pre-trained language models. In this work, we give a detailed overview of the approaches developed to date. We present and analyze the benefits and limitations that each model introduces, along with how they compare to the autoregressive models that dominate this field.