Variational Bayesian Inference for Modelswith Nuisance Parameters and an Intractable Likelihood

Abstract A primary challenge in Bayesian analysis lies in computing the posterior distributionof model parameters, a task that becomes more challenging for models with a largenumber of parameters or when the likelihood is intractable; as seen in Bayesian Lassoand state-space models. Often, the focus of the analysis is on a subset of the parameters,with the remainder regarded as nuisance parameters, which are either not of inferen-tial interest or are introduced to simplify computational processes. This complexitynecessitates more refined computational methods. Variational Bayesian inference (VB) has emerged as a powerful solution, enhancing computational efficiency by recastinginference as an optimization problem within a family of tractable distributions. How-ever, common VB techniques sometimes fall short, especially for models with nuisanceparameters or intractable likelihoods. To overcome these limitations, we introduce a uni-fied VB framework, termed Hybrid Variational Bayes (HVB), designed to achieve moreprecise Bayesian inference in such scenarios. This framework innovatively leverages ahybrid variational structure between the parameters of interest and nuisance parame-ters. A significant contribution of this work is the development of two robust gradientapproaches that effectively reduce variance, enhancing reliability in gradient-based op-timization algorithms for approximating complex posterior distributions. Furthermore,we link our strategy to Fisher’s identity, providing deeper theoretical insights. We alsoestablish a method to treat the stochastic element of likelihood estimation as a nuisanceparameter in models with intractable likelihoods due to complexity or computationaldemands. Through theoretical exploration and a series of illustrative examples, ourapproach demonstrates notable improvements over traditional VB methods.