Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes

We study the covariance structure of a Markov chain generated by the Gibbs sampler, with emphasis on data augmentation. When applied to a Bayesian missing data problem, the Gibbs sampler produces two natural approximations for the posterior distribution of the parameter vector: the empirical distribution based on the sampled values of the parameter vector, and a mixture of complete data posteriors. We prove that Rao-Blackwellization causes a one-lag delay for the autocovariances among dependent samples obtained from data augmentation, and consequently, the mixture approximation produces estimates with smaller variances than the empirical approximation. The covariance structure results are used to compare different augmentation schemes. It is shown that collapsing and grouping random components in a Gibbs sampler with two or three components usually result in more efficient sampling schemes.