Search

Although existing literature on high-dimensional regression models is rich, the vast majority of studies have focused on independent and homogeneous error terms. In this article, we consider the problem of selecting high-dimensional regression models with heteroscedastic and time series errors, which have broad applications in economics, quantitative finance, environmental science, and many other fields. The error term in our model is the product of two components: one time series component, allowing for a short-memory, long-memory, or conditional heteroscedasticity effect, and a high-dimensional dispersion function accounting for exogenous heteroscedasticity. By making use of the orthogonal greedy algorithm and the high-dimensional information criterion, we propose a new model selection procedure that consistently chooses the relevant variables in both the regression and the dispersion functions. The finite sample performance of the proposed procedure is also illustrated via simulations and real data analysis.

Principal component analysis (PCA) is the most commonly used statistical procedure for dimension reduction. An important issue for applying PCA is to determine the rank, which is the number of dominant eigenvalues of the covariance matrix. The Akaike information criterion (AIC) and Bayesian information criterion (BIC) are among the most widely used rank selection methods. Both use the number of free parameters for assessing model complexity. In this work, we adopt the generalized information criterion (GIC) to propose a new method for PCA rank selection under the high-dimensional framework. The GIC model complexity takes into account the sizes of covariance eigenvalues and can be better adaptive to practical applications. Asymptotic properties of GIC are derived and the selection consistency is established under the generalized spiked covariance model., 9 figures

We are interested in constructing confidence intervals for the autoregressive (AR) coefficient of a first†order AR model with i.i.d. positive errors via an extreme value estimate (EVE). We assume that the error distribution has a density function fIµ(x) behaving like b1,0xI±0−1 as x→0, where b1,0 and I±0 are unknown positive constants. These specifications imply that the EVE has a limiting distribution depending on b1,0 and I±0 from which only an infeasible interval estimate can be obtained. To alleviate this difficulty, we introduce a novel procedure to estimate these two constants and establish the desired consistency. This consistency result enables us not only to gain a better understanding of the underlying error distribution, but also to construct a feasible, asymptotically valid confidence interval of the AR coefficient, without resorting to a bootstrap procedure described in Datta and McCormick (1995). The performance of the proposed interval estimate is further illustrated through simulation studies and real data analysis.

Motivated by applications to root-cause identification of faults in multistage manufacturing processes that involve a large number of tools or equipment at each stage, we consider multiple testing in regression models whose outputs represent the quality characteristics of a multistage manufacturing process. Because of the large number of input variables that correspond to the tools or equipments used, this falls in the framework of regression modeling in the modern era of big data. On the other hand, with quick fault detection and diagnosis followed by tool rectification, sparsity can be assumed in the regression model. We introduce a new approach to address the multiple testing problem and demonstrate its advantages over existing methods. We also illustrate its performance in an application to semiconductor wafer fabrication that motivated this development. Supplementary materials for this article are available online.

Consider finite parametric time series models. “I have $n$ observations and $k$ models, which model should I choose on the basis of the data alone” is a frequently asked question in many practical situations. This poses the key problem of selecting a model from a collection of candidate models, none of which is necessarily the true data generating process (DGP). Although existing literature on model selection is vast, there is a serious lacuna in that the above problem does not seem to have received much attention. In fact, existing model selection criteria have avoided addressing the above problem directly, either by assuming that the true DGP is included among the candidate models and aiming at choosing this DGP, or by assuming that the true DGP can be asymptotically approximated by an increasing sequence of candidate models and aiming at choosing the candidate having the best predictive capability in some asymptotic sense. In this article, we propose a misspecification-resistant information criterion (MRIC) to address the key problem directly. We first prove the asymptotic efficiency of MRIC whether the true DGP is among the candidates or not, within the fixed-dimensional framework. We then extend this result to the high-dimensional case in which the number of candidate variables is much larger than the sample size. In particular, we show that MRIC can be used in conjunction with a high-dimensional model selection method to select the (asymptotically) best predictive model across several high-dimensional misspecified time series models.

There is an extensive literature on fixed-size confidence regions for the regression parameters in a linear model with p regressors, attaining a prescribed coverage probability when p is fixed and the size d approaches 0. Motivated by recent developments in regression modeling in response to applications for which p is considerably larger than the sample size, we develop herein a more versatile sequential methodology for fixed-size confidence regions that can handle the case p = p(d) → ∞ as d → 0.

In this paper, we assume that observations are generated by a linear regression model with short- or long-memory dependent errors. We establish inverse moment bounds for k n -dimensional sample autocovariance matrices based on the least squares residuals (also known as the detrended time series), where k n ≪ n , k n → ∞ and n is the sample size. These results are then used to derive the mean-square error bounds for the finite predictor coefficients of the underlying error process. Based on the detrended time series, we further estimate the inverse of the n-dimensional autocovariance matrix, R n − 1 , of the error process using the banded Cholesky factorization. By making use of the aforementioned inverse moment bounds, we obtain the convergence of moments of the difference between the proposed estimator and R n − 1 under spectral norm.

Modeling multistage manufacturing processes for fault detection and diagnosis in modern production systems has emerged as a cutting-edge research area at the interface of the engineering and statistical sciences. We give an overview of the developments in this area and describe some recent innovations in statistical modeling and inference associated with these developments.

We begin by reviewing recent results of Ing and Lai (Stat Sin 21:1473–1513, 2011) on the statistical properties of the orthogonal greedy algorithm (OGA) in high-dimensional sparse regression models with independent observations. In particular, when the regression coefficients are absolutely summable, the conditional mean squared prediction error and the empirical norm of OGA derived by Ing and Lai (Stat Sin 21:1473–1513, 2011) are introduced. We then explore the performance of OGA under more general sparsity conditions. Finally, we obtain the convergence rate of OGA in high-dimensional time series models, and illustrate the advantage of our results compared to those established for Lasso by Basu and Michailidis (Ann Stat 43:1535–1567, 2015) and Wu and Wu (Electron J Stat 10:352–379, 2016).

We consider a stochastic process model with time trend and measurement error. We establish consistency and derive the limiting distributions of the maximum likelihood (ML) estimators of the covariance function parameters under a general asymptotic framework, including both the fixed domain and the increasing domain frameworks, even when the time trend model is misspecified or its complexity increases with the sample size. In particular, the convergence rates of the ML estimators are thoroughly characterized in terms of the growing rate of the domain and the degree of model misspecification/complexity., Published at http://dx.doi.org/10.3150/15-BEJ740 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

Let observations y1, …, yn be generated from a first-order autoregressive (AR) model with positive errors. In both the stationary and unit root cases, we derive moment bounds and limiting distributions of an extreme value estimator, , of the AR coefficient. These results enable us to provide asymptotic expressions for the mean squared error (MSE) of and the mean squared prediction error (MSPE) of the corresponding predictor, , of yn + 1. Based on these expressions, we compare the relative performance of () and the least-squares predictor (estimator) from the MSPE (MSE) point of view. Our comparison reveals that the better predictor (estimator) is determined not only by whether a unit root exists, but also by the behavior of the underlying error distribution near the origin, and hence is difficult to identify in practice. To circumvent this difficulty, we suggest choosing the predictor (estimator) with the smaller accumulated prediction error and show that the predictor (estimator) chosen in this way is as...

We show that Akaike’s Information Criterion (AIC) and its variants are asymptotically efficient in integrated autoregressive processes of infinite order (AR(∞)). This result, together with its stationary counterpart established previously in the literature, ensures that AIC can ultimately achieve prediction efficiency in an AR(∞) process, without knowing the integration order.

The article by Aoshima and Yata considers two-stage procedures to construct fixed-size confidence regions and to select variables in high-dimensional models with a bounded number of nonzero parameters. In this discussion, we describe some recent work on alternative variable selection methods and fixed-width confidence intervals in high-dimensional sparse regression models.

Assume that observations are generated from nonstationary autoregressive (AR) processes of infinite order. We adopt a finite-order approximation model to predict future observations and obtain an asymptotic expression for the mean-squared prediction error (MSPE) of the least squares predictor. This expression provides the first exact assessment of the impacts of nonstationarity, model complexity, and model misspecification on the corresponding MSPE. It not only provides a deeper understanding of the least squares predictors in nonstationary time series, but also forms the theoretical foundation for a companion paper by the same authors, which obtains asymptotically efficient order selection in nonstationary AR processes of possibly infinite order.

This work aims at estimating inverse autocovariance matrices of long memory processes admitting a linear representation. A modified Cholesky decomposition is used in conjunction with an increasing order autoregressive model to achieve this goal. The spectral norm consistency of the proposed estimate is established. We then extend this result to linear regression models with long-memory time series errors. In particular, we show that when the objective is to consistently estimate the inverse autocovariance matrix of the error process, the same approach still works well if the estimated (by least squares) errors are used in place of the unobservable ones. Applications of this result to estimating unknown parameters in the aforementioned regression model are also given. Finally, a simulation study is performed to illustrate our theoretical findings., Comment: Published at http://dx.doi.org/10.3150/14-BEJ692 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

Consider a regression model with infinitely many parameters and time series errors. We are interested in choosing weights for averaging across generalized least squares (GLS) estimators obtained from a set of approximating models. However, GLS estimators, depending on the unknown inverse covariance matrix of the errors, are usually infeasible. We therefore construct feasible generalized least squares (FGLS) estimators using a consistent estimator of the unknown inverse matrix. Based on this inverse covariance matrix estimator and FGLS estimators, we develop a feasible autocovariance-corrected Mallows model averaging criterion to select weights, thereby providing an FGLS model averaging estimator of the true regression function. We show that the generalized squared error loss of our averaging estimator is asymptotically equivalent to the minimum one among those of GLS model averaging estimators with the weight vectors belonging to a continuous set, which includes the discrete weight set used in Hansen (2007) as its proper subset.

Zhang and Shaman considered the problem of estimating the conditional mean-squared prediciton error (CMSPE) for a Gaussian autoregressive (AR) process. They used the final prediction error (FPE) of Akaike to estimate CMSPE and proposed that FPE's effectiveness be judged by its asymptotic correlation with CMSPE. However, as pointed out by Kabaila and He, the derivation of this correlation by Zhang and Shaman is incomplete, and the performance of FPE in estimating CMSPE is also poor in Kabaila and He's simulation study. Kabaila and He further proposed an alternative estimator of CMSPE, V, in the stationary AR(1) model. They reported that V has a larger normalized correlation with CMSPE through Monte Carlo simulation results. In this paper, we propose a generalization of V, V˜, in the higher-order AR model, and obtain the asymptotic correlation of FPE and V˜ with CMSPE. We show that the limit of the normalized correlation of V˜ with CMSPE is larger than that of FPE with CMSPE, and hence Kabaila and He's finding is justified theoretically. In addition, the performances of the above estimators of CMSPE are re-examined in terms of mean-squared errors (MSE). Our main conclusion is that from the MSE point of view, V˜ is the best choice among a family of asymptotically unbiased estimators of CMSPE including FPE and V˜ as its special cases.

Information criteria, such as Akaike's information criterion and Bayesian information criterion are often applied in model selection. However, their asymptotic behaviors for selecting geostatistical regression models have not been well studied, particularly under the fixed domain asymptotic framework with more and more data observed in a bounded fixed region. In this article, we study the generalized information criterion (GIC) for selecting geostatistical regression models under a more general mixed domain asymptotic framework. Via uniform convergence developments of some statistics, we establish the selection consistency and the asymptotic loss efficiency of GIC under some regularity conditions, regardless of whether the covariance model is correctly or wrongly specified. We further provide specific examples with different types of explanatory variables that satisfy the conditions. For example, in some situations, GIC is selection consistent, even when some spatial covariance parameters cannot be estimated consistently. On the other hand, GIC fails to select the true polynomial order consistently under the fixed domain asymptotic framework. Moreover, the growth rate of the domain and the degree of smoothness of candidate regressors in space are shown to play key roles for model selection., Published in at http://dx.doi.org/10.1214/14-AOS1258 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

An asymptotic expression for the mean-squared prediction error (MSPE) of the least squares predictor is obtained in the random walk model. It is shown that the term of order 1/n in this error, where n is the sample size, is twice as large as the one obtained from the first-order autoregressive (AR(1)) model satisfying the stationary assumption. Moreover, while the correlation between the squares of the (normalized) regressor variable and normalized least squares estimator is asymptotically negligible in the stationary AR(1) model, we have found that the correlation has significantly negative value in the random walk model. To obtain these results, a new methodology, which is found to be useful in dealing with the moment properties of a strongly dependent process, is introduced.

A moment bound for the normalized conditional-sum-of-squares (CSS) estimate of a general autoregressive fractionally integrated moving average (ARFIMA) model with an arbitrary unknown memory parameter is derived in this paper. To achieve this goal, a uniform moment bound for the inverse of the normalized objective function is established. An important application of these results is to establish asymptotic expressions for the one-step and multi-step mean squared prediction errors (MSPE) of the CSS predictor. These asymptotic expressions not only explicitly demonstrate how the multi-step MSPE of the CSS predictor manifests with the model complexity and the dependent structure, but also offer means to compare the performance of the CSS predictor with the least squares (LS) predictor for integrated autoregressive models. It turns out that the CSS predictor can gain substantial advantage over the LS predictor when the integration order is high. Numerical findings are also conducted to illustrate the theoretical results., Published in at http://dx.doi.org/10.1214/13-AOS1110 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

In this paper, a uniform (over some parameter space) moment bound for the inverse of Fisher's information matrix is established. This result is then applied to develop moment bounds for the normalized least squares estimate in (nonlinear) stochastic regression models. The usefulness of these results is illustrated using time series models. In particular, an asymptotic expression for the mean squared prediction error of the least squares predictor in autoregressive moving average models is obtained. This asymptotic expression provides a solid theoretical foundation for some model selection criteria., Published in at http://dx.doi.org/10.1214/10-AOS861 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

We introduce a fast stepwise regression method, called the orthogonal greedy algorithm (OGA), that selects input variables to enter a p-dimensional linear regression model (with p ? n, the sample size) sequentially so that the selected variable at each step minimizes the residual sum squares. We derive the convergence rate of OGA and develop a consistent model selection procedure along the OGA path that can adjust for potential spuriousness of the greedily chosen regressors among a large number of candidate variables. The resultant regression estimate is shown to have the oracle property of being equivalent to least squares regression on an asymptotically minimal set of relevant regressors under a strong sparsity condition.

This paper investigates multistep prediction errors for non-stationary autoregressive processes with both model order and true parameters unknown. We give asymptotic expressions for the multistep mean squared prediction errors and accumulated prediction errors of two important methods, plug-in and direct prediction. These expressions not only characterize how the prediction errors are influenced by the model orders, prediction methods, values of parameters and unit roots, but also inspire us to construct some new predictor selection criteria that can ultimately choose the best combination of the model order and prediction method with probability 1. Finally, simulation analysis confirms the satisfactory finite sample performance of the newly proposed criteria., Comment: Published in at http://dx.doi.org/10.3150/08-BEJ165 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

A major research area of Ching-Zong Wei (1949--2004) was time series models and their applications in econometrics and engineering, to which he made many important contributions. A conference on time series and related topics in memory of him was held on December 12--14, 2005, at Academia Sinica in Taipei, where he was Director of the Institute of Statistical Science from 1993 to 1999. Of the forty-two speakers at the conference, twenty contributed to this volume. These papers are listed under the following three headings.

In this article asymptotic expressions for the final prediction error (FPE) and the accumulated prediction error (APE) of the least squares predictor are obtained in regression models with nonstationary regressors. It is shown that the term of order $1/n$ in FPE and the term of order $\log n$ in APE share the same constant, where $n$ is the sample size. Since the model includes the random walk model as a special case, these asymptotic expressions extend some of the results in Wei (1987) and Ing (2001). In addition, we also show that while the FPE of the least squares predictor is not affected by the contemporary correlation between the innovations in input and output variables, the mean squared error of the least squares estimate does vary with this correlation., Published at http://dx.doi.org/10.1214/074921706000000950 in the IMS Lecture Notes Monograph Series (http://www.imstat.org/publications/lecnotes.htm) by the Institute of Mathematical Statistics (http://www.imstat.org)

We establish a maximal moment inequality for the weighted sum of a long- range dependent process. An extension to H$\acute{a}$jek-R$\acute{e}$ny and Chow's type inequality is then obtained. It enables us to deduce a strong law for the weighted sum of a stationary long-range dependent time series. To illustrate its usefulness, applications of the inequality to estimation and model selection in multiple regression models with long-range dependent errors are given.

The predictive capability of a modification of Rissanen's accumulated prediction error (APE) criterion, APE$_{\delta_n}$, is investigated in infinite-order autoregressive (AR($\infty$)) models. Instead of accumulating squares of sequential prediction errors from the beginning, APE$_{\delta_n}$ is obtained by summing these squared errors from stage $n\delta_n$, where $n$ is the sample size and $1/n\leq \delta_n\leq 1-(1/n)$ may depend on $n$. Under certain regularity conditions, an asymptotic expression is derived for the mean-squared prediction error (MSPE) of an AR predictor with order determined by APE$_{\delta_n}$. This expression shows that the prediction performance of APE$_{\delta_n}$ can vary dramatically depending on the choice of $\delta_n$. Another interesting finding is that when $\delta_n$ approaches 1 at a certain rate, APE$_{\delta_n}$ can achieve asymptotic efficiency in most practical situations. An asymptotic equivalence between APE$_{\delta_n}$ and an information criterion with a suitable penalty term is also established from the MSPE point of view. This offers new perspectives for understanding the information and prediction-based model selection criteria. Finally, we provide the first asymptotic efficiency result for the case when the underlying AR($\infty$) model is allowed to degenerate to a finite autoregression., Comment: Published at http://dx.doi.org/10.1214/009053606000001550 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

We consider the problem of choosing the optimal (in the sense of mean-squared prediction error) multistep predictor for an autoregressive (AR) process of finite but unknown order. If a working AR model (which is possibly misspecified) is adopted for multistep predictions, then two competing types of multistep predictors (i.e., plug-in and direct predictors) can be obtained from this model. We provide some interesting examples to show that when both plug-in and direct predictors are considered, the optimal multistep prediction results cannot be guaranteed by correctly identifying the underlying model”s order. This finding challenges the traditional model (order) selection criteria, which usually aim to choose the order of the true model. A new prediction selection criterion, which attempts to seek the best combination of the prediction order and the prediction method, is proposed to rectify this difficulty. When the underlying model is stationary, the validity of the proposed criterion is justified theoretically. To obtain this result, asymptotic properties of accumulated squares of multistep prediction errors are investigated. In addition to overcoming the above difficulty, some other advantages of the proposed criterion are also mentioned.

In this paper, two competing types of multistep predictors, i.e., plug-in and direct predictors, are considered in autoregressive (AR) processes. When a working model AR( k ) is used for the h -step prediction with h > 1, the plug-in predictor is obtained from repeatedly using the fitted (by least squares) AR( k ) model with an unknown future value replaced by their own forecasts, and the direct predictor is obtained by estimating the h -step prediction model's coefficients directly by linear least squares. Under rather mild conditions, asymptotic expressions for the mean-squared prediction errors (MSPEs) of these two predictors are obtained in stationary cases. In addition, we also extend these results to models with deterministic time trends. Based on these expressions, performances of the plug-in and direct predictors are compared. Finally, two examples are given to illustrate that some stationary case results on these MSPEs can not be generalized to the nonstationary case. The author is deeply grateful to the co-editor Pentti Saikkonen and two referees for their helpful suggestions and comments on a previous version of this paper.