A Note on Least-Squares Approximation in the Bayesian Analysis of Regression Models

Regression models are introduced in the framework of Bayesian cuts (see, for example, Florens and Mouchart, 1977). It is well known that once the prior distribution is not natural-conjugate or the sampling process is non-normal, the computation of the posterior distribution may rapidly become intractable. This involves the temptation of specifying simplistic models in order to keep control on the tractability. Examples of these complications are well known when introducing fat-tails distributions in order to treat outliers or when facing asymmetrically distributed residuals. When the main interest lies in the computation of the posterior expectations of the parameters, we show that Least-Squares (L.S.) approximations allows one to render tractable those nonstandard models. This suggests that approximate solutions to reasonable models may be attractive alternatives to the exact solution to simplified models. The Least-Squares approximations may be interpreted in the framework of a normal approximation to the joint distribution of parameters and observations. This suggests that their practical relevance will crucially depend on the choice of co-ordinates. This note handles that choice in the field of regression models. In this paper the main emphasis is on parameter estimation (i.e. approximation of the posterior expectations of the parameter) while Goldstein (1976) uses similar tools to concentrate on predictions and de Vylder (1982) pays particular attention to restricted covariance matrices in the context of credibility theory. In Section 2 we introduce the basic model. Then in Section 3 we present our L.S. approximations trying to take advantage of the use of unbiased estimators, while in Section 4 we consider the particular case of singular covariance matrices.