Diffusion Models: Tutorial and Survey

Diffusion models are a family of generative models which work based on a Markovian process. In their forward process, they gradually add noise to data until it becomes a complete noise. In the backward process, the data are gradually generated out of noise. In this tutorial paper, the Denoising Diffusion Probabilistic Model (DDPM) is fully explained. Detailed simplification of the variational lower bound of its likelihood, parameters of the distributions, and the loss function of the diffusion model are discussed. Some modifications to the original DDPM, including non-fixed covariance matrix, reducing the gradient noise, improving the noise schedule, and non-standard Gaussian noise distribution, and conditional diffusion model are introduced. Finally, continuous noise schedule by Stochastic Differential Equation (SDE), where the noise schedule is in a continuous domain, is explained.