Regression density estimation with variational methods and stochastic approximation

Regression density estimation is the problem of flexibly estimating a response distribution as a function of covariates. An important approach to regression density estimation uses finite mixture models and our article considers flexible mixtures of heteroscedastic regression (MHR) models where the response distribution is a normal mixture, with the component means, variances and mixture weights all varying as a function of covariates. Our article develops fast variational approximation methods for inference. Our motivation is that alternative computationally intensive MCMC methods for fitting mixture models are difficult to apply when it is desired to fit models repeatedly in exploratory analysis and model choice. Our article makes three contributions. First, a variational approximation for MHR models is described where the variational lower bound is in closed form. Second, the basic approximation can be improved by using stochastic approximation methods to perturb the initial solution to attain higher accuracy. Third, the advantages of our approach for model choice and evaluation compared to MCMC based approaches are illustrated. These advantages are particularly compelling for time series data where repeated refitting for one step ahead prediction in model choice and diagnostics and in rolling window computations is very common. Supplemental materials for the article are available online.
funding agencies|Singapore Ministry of Education (MOE)|R-155-000-068-133|Singapore Delft Water Alliances (SDWA) tropical reservoir research program||Australian Research Council (ARC)|DP0988579