Search

AbstractDistance covariance (Székely, Rizzo, and Bakirov) is a fascinating recent notion, which is popular as a test for dependence of any type between random variables X and Y. This approach deserves to be touched upon in modern courses on mathematical statistics. It makes use of distances of the type |X−X′| and |Y−Y′|, where (X′,Y′) is an independent copy of (X, Y). This raises natural questions about independence of variables like X−X′ and Y−Y′, about the connection between cov(|X−X′|,|Y−Y′|) and the covariance between doubly centered distances, and about necessary and sufficient conditions for independence. We show some basic results and present a new and nontechnical counterexample to a common fallacy, which provides more insight. We also show some motivating examples involving bivariate distributions and contingency tables, which can be used as didactic material for introducing distance correlation. [ABSTRACT FROM AUTHOR]

Abstract The usual Minimum Covariance Determinant (MCD) estimator of a covariance matrix is robust against casewise outliers. These are cases (that is, rows of the data matrix) that behave differently from the majority of cases, raising suspicion that they might belong to a different population. On the other hand, cellwise outliers are individual cells in the data matrix. When a row contains one or more outlying cells, the other cells in the same row still contain useful information that we wish to preserve. We propose a cellwise robust version of the MCD method, called cellMCD. Its main building blocks are observed likelihood and a penalty term on the number of flagged cellwise outliers. It possesses good breakdown properties. We construct a fast algorithm for cellMCD based on concentration steps (C-steps) that always lower the objective. The method performs well in simulations with cellwise outliers, and has high finite-sample efficiency on clean data. It is illustrated on real data with visualizations of the results. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]

Classification by neural nets and by tree-based methods are powerful tools of machine learning. There exist interesting visualizations of the inner workings of these and other classifiers. Here we pursue a different goal, which is to visualize the cases being classified, either in training data or in test data. An important aspect is whether a case has been classified to its given class (label) or whether the classifier wants to assign it to a different class. This is reflected in the (conditional and posterior) probability of the alternative class (PAC). A high PAC indicates label bias, that is, the possibility that the case was mislabeled. The PAC is used to construct a silhouette plot which is similar in spirit to the silhouette plot for cluster analysis. The average silhouette width can be used to compare different classifications of the same dataset. We will also draw quasi residual plots of the PAC versus a data feature, which may lead to more insight in the data. One of these data features is how far each case lies from its given class. The graphical displays are illustrated and interpreted on datasets containing images, mixed features, and tweets. for this article are available online. [ABSTRACT FROM AUTHOR]

Classification is a major tool of statistics and machine learning. A classification method first processes a training set of objects with given classes (labels), with the goal of afterward assigning new objects to one of these classes. When running the resulting prediction method on the training data or on test data, it can happen that an object is predicted to lie in a class that differs from its given label. This is sometimes called label bias, and raises the question whether the object was mislabeled. The proposed class map reflects the probability that an object belongs to an alternative class, how far it is from the other objects in its given class, and whether some objects lie far from all classes. The goal is to visualize aspects of the classification results to obtain insight in the data. The display is constructed for discriminant analysis, the k-nearest neighbor classifier, support vector machines, logistic regression, and coupling pairwise classifications. It is illustrated on several benchmark datasets, including some about images and texts. [ABSTRACT FROM AUTHOR]

The product moment covariance matrix is a cornerstone of multivariate data analysis, from which one can derive correlations, principal components, Mahalanobis distances and many other results. Unfortunately, the product moment covariance and the corresponding Pearson correlation are very susceptible to outliers (anomalies) in the data. Several robust estimators of covariance matrices have been developed, but few are suitable for the ultrahigh-dimensional data that are becoming more prevalent nowadays. For that one needs methods whose computation scales well with the dimension, are guaranteed to yield a positive semidefinite matrix, and are sufficiently robust to outliers as well as sufficiently accurate in the statistical sense of low variability. We construct such methods using data transformations. The resulting approach is simple, fast, and widely applicable. We study its robustness by deriving influence functions and breakdown values, and computing the mean squared error on contaminated data. Using these results we select a method that performs well overall. This also allows us to construct a faster version of the DetectDeviatingCells method (Rousseeuw and Van den Bossche 2018) to detect cellwise outliers, which can deal with much higher dimensions. The approach is illustrated on genomic data with 12,600 variables and color video data with 920,000 dimensions. for this article are available online. [ABSTRACT FROM AUTHOR]

Multivariate data are typically represented by a rectangular matrix (table) in which the rows are the objects (cases) and the columns are the variables (measurements). When there are many variables one often reduces the dimension by principal component analysis (PCA), which in its basic form is not robust to outliers. Much research has focused on handling rowwise outliers, that is, rows that deviate from the majority of the rows in the data (e.g., they might belong to a different population). In recent years also cellwise outliers are receiving attention. These are suspicious cells (entries) that can occur anywhere in the table. Even a relatively small proportion of outlying cells can contaminate over half the rows, which causes rowwise robust methods to break down. In this article, a new PCA method is constructed which combines the strengths of two existing robust methods to be robust against both cellwise and rowwise outliers. At the same time, the algorithm can cope with missing values. As of yet it is the only PCA method that can deal with all three problems simultaneously. Its name MacroPCA stands for PCA allowing for Missingness And Cellwise & Rowwise Outliers. Several simulations and real datasets illustrate its robustness. New residual maps are introduced, which help to determine which variables are responsible for the outlying behavior. The method is well-suited for online process control. [ABSTRACT FROM AUTHOR]

A multivariate dataset consists of n cases in d dimensions, and is often stored in an n by d data matrix. It is well-known that real data may contain outliers. Depending on the situation, outliers may be (a) undesirable errors, which can adversely affect the data analysis, or (b) valuable nuggets of unexpected information. In statistics and data analysis, the word outlier usually refers to a row of the data matrix, and the methods to detect such outliers only work when at least half the rows are clean. But often many rows have a few contaminated cell values, which may not be visible by looking at each variable (column) separately. We propose the first method to detect deviating data cells in a multivariate sample which takes the correlations between the variables into account. It has no restriction on the number of clean rows, and can deal with high dimensions. Other advantages are that it provides predicted values of the outlying cells, while imputing missing values at the same time. We illustrate the method on several real datasets, where it uncovers more structure than found by purely columnwise methods or purely rowwise methods. The proposed method can help to diagnose why a certain row is outlying, for example, in process control. It also serves as an initial step for estimating multivariate location and scatter matrices. [ABSTRACT FROM AUTHOR]

Functional data analysis covers a wide range of data types. They all have in common that the observed objects are functions of a univariate argument (e.g., time or wavelength) or a multivariate argument (say, a spatial position). These functions take on values which can in turn be univariate (such as the absorbance level) or multivariate (such as the red/green/blue color levels of an image). In practice it is important to be able to detect outliers in such data. For this purpose we introduce a new measure of outlyingness that we compute at each gridpoint of the functions’ domain. The proposed directional outlyingness (DO) measure accounts for skewness in the data and only requires computation time per direction. We derive the influence function of the DO and compute a cutoff for outlier detection. The resulting heatmap and functional outlier map reflect local and global outlyingness of a function. To illustrate the performance of the method on real data it is applied to spectra, MRI images, and video surveillance data. [ABSTRACT FROM AUTHOR]

We introduce a new method for robust principal component analysis (PCA). Classical PCA is based on the empirical covariance matrix of the data and hence is highly sensitive to outlying observations. Two robust approaches have been developed to date. The first approach is based on the eigenvectors of a robust scatter matrix such as the minimum covariance determinant or an S-estimator and is limited to relatively low-dimensional data, The second approach is based on projection pursuit and can handle high- dimensional data. Here we propose the ROBPCA approach, which combines projection pursuit ideas with robust scatter matrix estimation. ROBPCA yields more accurate estimates at noncontaminated datasets and more robust estimates at contaminated data. ROBPCA can be computed rapidly, and is able to detect exact-fit situations. As a by-product, ROBPCA produces a diagnostic plot that displays and classifies the outliers. We apply the algorithm to several datasets from chemometrics and engineering. [ABSTRACT FROM AUTHOR]

This article examines Ivan Mizera and Christine H. Müller's theory of a halfspace depth in the univariate location-scale model, and the concept of the Student depth and the Student median. Mizera and Müller raised the question of the maximal Student depth of an absolutely continuous distribution equal to 1/2, and they noted that among all symmetric and asymmetric distributions, the Cauchy distribution has maximal depth 1/2. The influence function of the Student depth itself can be obtained by applying techniques similar to M. Romanazzi. Theorem 3 of the authors states that the Student depth of (μ, σ) is the infimum of P of the sets that lie in any Poincaré halfspace (PHS) with the point (μ, σ) on its boundary. The influence function of the Student median is somewhat harder to compute. The influence functions of the Student median at the Cauchy distribution are shown in Figure 2. Both curves are smooth and bounded, and they coincide with the influence functions of the location and scale Cauchy maximum likelihood estimators. Mizera and Müller showed that the breakdown value of the Student median is at least 1/3. To illustrate the breakdown behavior of the Student median, the author considered the Salary dataset available at Data and Story Library.

We introduce a robust method for multivariate regression based on robust estimation of the joint location and scatter matrix of the explanatory and response variables. As a robust estimator of location and scatter, we use the minimum covariance determinant (MCD) estimator of Rousseeuw. Based on simulations, we investigate the finite-sample performance and robustness of the estimator. To increase the efficiency. we propose a reweighted estimator selected from several possible reweighting schemes. The resulting multivariate regression does not need much computation time and is applied to real datasets. We show that the multivariate regression estimator has the appropriate equivariance properties, has a bounded influence function, and inherits the breakdown value of the MCD estimator. These theoretical robustness properties confirm the good finite-sample results obtained from the simulations. [ABSTRACT FROM AUTHOR]

Presents information on a study that proposed the distance-distance plot (D-D plot) which displays minimum covariance determinant (MCD)-based robust distances versus Mahalanobis distances. Overview on the problem used in the study; Details of the construction of the algorithm; Pseudocode of FAST-MCD; Advantages of FAST-MCD; Details on some applications to compare the FAST-MCD results with the classical mean and covariance matrix.

The article presents a response by Peter J. Rousseeuw and colleagues to commentaries regarding their article on deepest regression method. They explained the computational formulas they developed for regression analysis. They clarified the opinions of critics on their regression depth. They also discussed the assumptions underlying each formula for regression depth.

Focuses on a study that explored the analogies between depth in regression and in location. Background of regression depth; Uses of deepest regression line; Details on the computation of multiple regression; Application of relation with location indepth.

We propose the bagplot, a bivariate generalization of the univariate boxplot. The key notion is the halfspace location depth of a point relative to a bivariate dataset, which extends the univariate concept of rank. The "depth median" is the deepest location, and it is surrounded by a "bag" containing the n/2 observations with largest depth. Magnifying the bag by a factor 3 yields the "fence" (which is not plotted). Observations between the bag and the fence are marked by a light gray loop, whereas observations outside the fence are flagged as outliers. The bagplot visualizes the location, spread, correlation, skewness, and tails of the data. It is equivariant for linear transformations, and not limited to elliptical distributions. Software for drawing the bagplot is made available for the S-Plus and MATLAB environments. The bagplot is illustrated on several datasets--for example, in a scatterplot matrix of multivariate data. [ABSTRACT FROM AUTHOR]

Focuses on the intersection of the fuzzy set theory and statistics called fuzzy clustering. Comparison between the crisp clustering method and fuzzy clustering; Existence of specific clustering methods for special cases; Use of fuzziness in algorithms.

In this article we introduce a new type of positive-breakdown regression method, called a generalized S-estimator (or GS-estimator), based on the minimization of a generalized M-estimator of residual scale. We compare the class of GS-estimators with the usual S-estimators, including least median of squares. It turns out that GS-estimators attain a much higher efficiency than S-estimators, at the cost of a slightly increased worst-case bias. We investigate the breakdown point, the maxbias curve, and the influence function of GS-estimators. We also give an algorithm for computing GS-estimators and apply it to real and simulated data. [ABSTRACT FROM AUTHOR]

Looks for alternatives to the median absolute deviation (MAD) that can be used as initial or ancillary scale estimates in the same way but that are more efficient and not slanted towards symmetric distributions. Simplicity of the sample median and the MAD; Advantages and disadvantages of MAD.

Presents a reply to the comments on their article "Unmasking Multivariate Outliers and Leverage Points." Description of the projection algorithm; Data set consisting of two copies of the wood gravity data; Criticism that there were too many outliers and violation of dimensionality requirement.

Detecting outliers in a multivariate point cloud is not trivial, especially when there are several outliers. The classical identification method does not always find them, because it is based on the sample mean and covariance matrix, which are themselves affected by the outliers. That is how the outliers get masked. To avoid the masking effect, we propose to compute distances based on very robust estimates of location and covariance. These robust distances are better suited to expose the outliers. In the case of regression data, the classical least squares approach masks outliers in a similar way. Also here, the outliers may be unmasked by using a highly robust regression method. Finally, a new display is proposed in which the robust regression residuals are plotted versus the robust distances. This plot classifies the data into regular observations, vertical outliers, good leverage points, and bad leverage points. Several examples are discussed. [ABSTRACT FROM AUTHOR]

Proposes a robust estimator that can be computed by means of a single pass updating mechanism. Applications to averaging; Storage and computation time; Finite-sample breakdown point of an estimator; Consistency of the estimator.

Classical least squares regression consists of minimizing the sum of the squared residuals. Many authors have produced more robust versions of this estimator by replacing the square by something else, such as the absolute value. In this article a different approach is introduced in which the sum is replaced by the median of the squared residuals. The resulting estimator can resist the effect of nearly 50% of contamination in the data. In the special case of simple regression, it corresponds to finding the narrowest strip covering half of the observations. Generalizations are possible to multivariate location, orthogonal regression, and hypothesis testing in linear models. [ABSTRACT FROM AUTHOR]

In this article, the authors define the change-of-variance curve of location M-estimators in order to investigate the infinitesimal stability of the asymptotic variance. This relatively new tool for robustness theory, bearing some resemblance to the influence curve, was discovered in 1972. The curve as well as the estimators were sometimes referred to briefly, but they not yet appeared in print explicitly. The purpose of this article is to present an exact development of these notions, including a proof of the existence of tanh-estimators, certain numerical computations of their defining constants and a theorem showing they are optimally robust. The investigation takes place in the classical framework of M-estimation of a location parameter when scale is known. Redescending M-estimators have become quite popular in the field of robust statistics, partly because of their successful performance in the Princeton Monte Carlo study. In that study, stability of the asymptotic variance. The authors also construct the so-called hyperbolic tangent estimators, proving their existence and performing certain numerical computations of their defining constants.

Examines the shape of correlation matrices. Correlations between three variables; Correlation between four variables; Banded correlation matrices.

The article presents comments on an article published in the March 01, 2002 issue of the periodical "Journal of the American Statistical Association." Data acquisition is much less expensive today than, say 20 years ago. Therefore, modern statistical methods should be able to cope with large datasets. This is particularly true for linear regression methods, because they are widely used in many applications. Positive-breakdown estimators have been developed to protect against outlying data points, but they are often hard to compute exactly. Although, authors proposing a new algorithm usually focus on its computation time and performance, Hawkins and Olive in the first part of their article take a closer look at the breakdown value and the (in)consistency of algorithm based on elemental samples. Contrary to Hawkins and Olive, the authors find the extreme computational load of the X-cluster algorithm problematic. In their experience, practitioners can rarely he convinced to use a method that cannot be computed within reasonable time.

AbstractMultivariate Singular Spectrum Analysis (MSSA) is a powerful and widely used nonparametric method for multivariate time series, which allows the analysis of complex temporal data from diverse fields such as finance, healthcare, ecology, and engineering. However, MSSA lacks robustness against outliers because it relies on the singular value decomposition, which is very sensitive to the presence of anomalous values. MSSA can then give biased results and lead to erroneous conclusions. In this article a new MSSA method is proposed, named RObust Diagonalwise Estimation of SSA (RODESSA), which is robust against the presence of cellwise and casewise outliers. In particular, the decomposition step of MSSA is replaced by a new robust low-rank approximation of the trajectory matrix that takes its special structure into account. A fast algorithm is constructed, and it is proved that each iteration step decreases the objective function. In order to visualize different types of outliers, a new graphical display is introduced, called an enhanced time series plot. An extensive Monte Carlo simulation study is performed to compare RODESSA with competing approaches in the literature. A real data example about temperature analysis in passenger railway vehicles demonstrates the practical utility of the proposed approach. [ABSTRACT FROM AUTHOR]