Search

AbstractDistance covariance (Székely et al. 2007) is a fascinating recent notion, which is popular as a test for dependence of any type between random variables X and Y. This approach deserves to be touched upon in modern courses on mathematical statistics. It makes use of distances of the type |X−X′| and |Y−Y′| , where (X′,Y′) is an independent copy of (X,Y) . This raises natural questions about independence of variables like X−X′ and Y−Y′ , about the connection between Cov(|X−X′|,|Y−Y′|) and the covariance between doubly centered distances, and about necessary and sufficient conditions for independence. We show some basic results and present a new and nontechnical counterexample to a common fallacy, which provides more insight. We also show some motivating examples involving bivariate distributions and contingency tables, which can be used as didactic material for introducing distance correlation. [ABSTRACT FROM AUTHOR]

Abstract The usual Minimum Covariance Determinant (MCD) estimator of a covariance matrix is robust against casewise outliers. These are cases (that is, rows of the data matrix) that behave differently from the majority of cases, raising suspicion that they might belong to a different population. On the other hand, cellwise outliers are individual cells in the data matrix. When a row contains one or more outlying cells, the other cells in the same row still contain useful information that we wish to preserve. We propose a cellwise robust version of the MCD method, called cellMCD. Its main building blocks are observed likelihood and a penalty term on the number of flagged cellwise outliers. It possesses good breakdown properties. We construct a fast algorithm for cellMCD based on concentration steps (C-steps) that always lower the objective. The method performs well in simulations with cellwise outliers, and has high finite-sample efficiency on clean data. It is illustrated on real data with visualizations of the results. Supplementary materials for this article are available online. [ABSTRACT FROM AUTHOR]

Classification by neural nets and by tree-based methods are powerful tools of machine learning. There exist interesting visualizations of the inner workings of these and other classifiers. Here we pursue a different goal, which is to visualize the cases being classified, either in training data or in test data. An important aspect is whether a case has been classified to its given class (label) or whether the classifier wants to assign it to a different class. This is reflected in the (conditional and posterior) probability of the alternative class (PAC). A high PAC indicates label bias, that is, the possibility that the case was mislabeled. The PAC is used to construct a silhouette plot which is similar in spirit to the silhouette plot for cluster analysis. The average silhouette width can be used to compare different classifications of the same dataset. We will also draw quasi residual plots of the PAC versus a data feature, which may lead to more insight in the data. One of these data features is how far each case lies from its given class. The graphical displays are illustrated and interpreted on datasets containing images, mixed features, and tweets. for this article are available online. [ABSTRACT FROM AUTHOR]

Classification is a major tool of statistics and machine learning. A classification method first processes a training set of objects with given classes (labels), with the goal of afterward assigning new objects to one of these classes. When running the resulting prediction method on the training data or on test data, it can happen that an object is predicted to lie in a class that differs from its given label. This is sometimes called label bias, and raises the question whether the object was mislabeled. The proposed class map reflects the probability that an object belongs to an alternative class, how far it is from the other objects in its given class, and whether some objects lie far from all classes. The goal is to visualize aspects of the classification results to obtain insight in the data. The display is constructed for discriminant analysis, the k-nearest neighbor classifier, support vector machines, logistic regression, and coupling pairwise classifications. It is illustrated on several benchmark datasets, including some about images and texts. [ABSTRACT FROM AUTHOR]

The product moment covariance matrix is a cornerstone of multivariate data analysis, from which one can derive correlations, principal components, Mahalanobis distances and many other results. Unfortunately, the product moment covariance and the corresponding Pearson correlation are very susceptible to outliers (anomalies) in the data. Several robust estimators of covariance matrices have been developed, but few are suitable for the ultrahigh-dimensional data that are becoming more prevalent nowadays. For that one needs methods whose computation scales well with the dimension, are guaranteed to yield a positive semidefinite matrix, and are sufficiently robust to outliers as well as sufficiently accurate in the statistical sense of low variability. We construct such methods using data transformations. The resulting approach is simple, fast, and widely applicable. We study its robustness by deriving influence functions and breakdown values, and computing the mean squared error on contaminated data. Using these results we select a method that performs well overall. This also allows us to construct a faster version of the DetectDeviatingCells method (Rousseeuw and Van den Bossche 2018) to detect cellwise outliers, which can deal with much higher dimensions. The approach is illustrated on genomic data with 12,600 variables and color video data with 920,000 dimensions. for this article are available online. [ABSTRACT FROM AUTHOR]

Multivariate data are typically represented by a rectangular matrix (table) in which the rows are the objects (cases) and the columns are the variables (measurements). When there are many variables one often reduces the dimension by principal component analysis (PCA), which in its basic form is not robust to outliers. Much research has focused on handling rowwise outliers, that is, rows that deviate from the majority of the rows in the data (e.g., they might belong to a different population). In recent years also cellwise outliers are receiving attention. These are suspicious cells (entries) that can occur anywhere in the table. Even a relatively small proportion of outlying cells can contaminate over half the rows, which causes rowwise robust methods to break down. In this article, a new PCA method is constructed which combines the strengths of two existing robust methods to be robust against both cellwise and rowwise outliers. At the same time, the algorithm can cope with missing values. As of yet it is the only PCA method that can deal with all three problems simultaneously. Its name MacroPCA stands for PCA allowing for Missingness And Cellwise & Rowwise Outliers. Several simulations and real datasets illustrate its robustness. New residual maps are introduced, which help to determine which variables are responsible for the outlying behavior. The method is well-suited for online process control. [ABSTRACT FROM AUTHOR]

A multivariate dataset consists of n cases in d dimensions, and is often stored in an n by d data matrix. It is well-known that real data may contain outliers. Depending on the situation, outliers may be (a) undesirable errors, which can adversely affect the data analysis, or (b) valuable nuggets of unexpected information. In statistics and data analysis, the word outlier usually refers to a row of the data matrix, and the methods to detect such outliers only work when at least half the rows are clean. But often many rows have a few contaminated cell values, which may not be visible by looking at each variable (column) separately. We propose the first method to detect deviating data cells in a multivariate sample which takes the correlations between the variables into account. It has no restriction on the number of clean rows, and can deal with high dimensions. Other advantages are that it provides predicted values of the outlying cells, while imputing missing values at the same time. We illustrate the method on several real datasets, where it uncovers more structure than found by purely columnwise methods or purely rowwise methods. The proposed method can help to diagnose why a certain row is outlying, for example, in process control. It also serves as an initial step for estimating multivariate location and scatter matrices. [ABSTRACT FROM AUTHOR]

Functional data analysis covers a wide range of data types. They all have in common that the observed objects are functions of a univariate argument (e.g., time or wavelength) or a multivariate argument (say, a spatial position). These functions take on values which can in turn be univariate (such as the absorbance level) or multivariate (such as the red/green/blue color levels of an image). In practice it is important to be able to detect outliers in such data. For this purpose we introduce a new measure of outlyingness that we compute at each gridpoint of the functions’ domain. The proposed directional outlyingness (DO) measure accounts for skewness in the data and only requires computation time per direction. We derive the influence function of the DO and compute a cutoff for outlier detection. The resulting heatmap and functional outlier map reflect local and global outlyingness of a function. To illustrate the performance of the method on real data it is applied to spectra, MRI images, and video surveillance data. [ABSTRACT FROM AUTHOR]

Real data often contain anomalous cases, also known as outliers. These may spoil the resulting analysis but they may also contain valuable information. In either case, the ability to detect such anomalies is essential. A useful tool for this purpose is robust statistics, which aims to detect the outliers by first fitting the majority of the data and then flagging data points that deviate from it. We present an overview of several robust methods and the resulting graphical outlier detection tools. We discuss robust procedures for univariate, low‐dimensional, and high‐dimensional data, such as estimating location and scatter, linear regression, principal component analysis, classification, clustering, and functional data analysis. Also the challenging new topic of cellwise outliers is introduced. WIREs Data Mining Knowl Discov 2018, 8:e1236. doi: 10.1002/widm.1236 This article is categorized under: Algorithmic Development > Spatial and Temporal Data Mining Technologies > Classification Technologies > Structure Discovery and Clustering Technologies > Visualization [ABSTRACT FROM AUTHOR]

Clustering non-Euclidean data is difficult, and one of the most used algorithms besides hierarchical clustering is the popular algorithm Partitioning Around Medoids (PAM), also simply referred to as k -medoids clustering. In Euclidean geometry the mean – as used in k -means – is a good estimator for the cluster center, but this does not exist for arbitrary dissimilarities. PAM uses the medoid instead, the object with the smallest dissimilarity to all others in the cluster. This notion of centrality can be used with any (dis-)similarity, and thus is of high relevance to many domains and applications. A key issue with PAM is its high run time cost. We propose modifications to the PAM algorithm that achieve an O (k) -fold speedup in the second ("SWAP") phase of the algorithm, but will still find the same results as the original PAM algorithm. If we relax the choice of swaps performed (while retaining comparable quality), we can further accelerate the algorithm by eagerly performing additional swaps in each iteration. With the substantially faster SWAP, we can now explore faster initialization strategies, because (i) the classic ("BUILD") initialization now becomes the bottleneck, and (ii) our swap is fast enough to compensate for worse starting conditions. We also show how the CLARA and CLARANS algorithms benefit from the proposed modifications. While we do not study the parallelization of our approach in this work, it can easily be combined with earlier approaches to use PAM and CLARA on big data (some of which use PAM as a subroutine, hence can immediately benefit from these improvements), where the performance with high k becomes increasingly important. In experiments on real data with k = 100 , 200 , we observed a 458 × respectively 1191 × speedup compared to the original PAM SWAP algorithm, making PAM applicable to larger data sets, and in particular to higher k. • Faster k-Medoids (PAM) clustering algorithm. • Scalable for large number of clusters (large k). • Sampling-based approximations for large data sets (large n). • Same quality as previous state-of-the-art techniques (PAM). • Included in popular clustering tools such as ELKI and R. [ABSTRACT FROM AUTHOR]

We introduce a new method for robust principal component analysis (PCA). Classical PCA is based on the empirical covariance matrix of the data and hence is highly sensitive to outlying observations. Two robust approaches have been developed to date. The first approach is based on the eigenvectors of a robust scatter matrix such as the minimum covariance determinant or an S-estimator and is limited to relatively low-dimensional data, The second approach is based on projection pursuit and can handle high- dimensional data. Here we propose the ROBPCA approach, which combines projection pursuit ideas with robust scatter matrix estimation. ROBPCA yields more accurate estimates at noncontaminated datasets and more robust estimates at contaminated data. ROBPCA can be computed rapidly, and is able to detect exact-fit situations. As a by-product, ROBPCA produces a diagnostic plot that displays and classifies the outliers. We apply the algorithm to several datasets from chemometrics and engineering. [ABSTRACT FROM AUTHOR]

This article examines Ivan Mizera and Christine H. Müller's theory of a halfspace depth in the univariate location-scale model, and the concept of the Student depth and the Student median. Mizera and Müller raised the question of the maximal Student depth of an absolutely continuous distribution equal to 1/2, and they noted that among all symmetric and asymmetric distributions, the Cauchy distribution has maximal depth 1/2. The influence function of the Student depth itself can be obtained by applying techniques similar to M. Romanazzi. Theorem 3 of the authors states that the Student depth of (μ, σ) is the infimum of P of the sets that lie in any Poincaré halfspace (PHS) with the point (μ, σ) on its boundary. The influence function of the Student median is somewhat harder to compute. The influence functions of the Student median at the Cauchy distribution are shown in Figure 2. Both curves are smooth and bounded, and they coincide with the influence functions of the location and scale Cauchy maximum likelihood estimators. Mizera and Müller showed that the breakdown value of the Student median is at least 1/3. To illustrate the breakdown behavior of the Student median, the author considered the Salary dataset available at Data and Story Library.

We introduce a robust method for multivariate regression based on robust estimation of the joint location and scatter matrix of the explanatory and response variables. As a robust estimator of location and scatter, we use the minimum covariance determinant (MCD) estimator of Rousseeuw. Based on simulations, we investigate the finite-sample performance and robustness of the estimator. To increase the efficiency. we propose a reweighted estimator selected from several possible reweighting schemes. The resulting multivariate regression does not need much computation time and is applied to real datasets. We show that the multivariate regression estimator has the appropriate equivariance properties, has a bounded influence function, and inherits the breakdown value of the MCD estimator. These theoretical robustness properties confirm the good finite-sample results obtained from the simulations. [ABSTRACT FROM AUTHOR]

Let P be a general probability distribution on Rp, which need not have a density or moments. We investigate the relation between angular symmetry of P (a.k.a. directional symmetry) and the halfspace (Tukey) depth. When P is angularly symmetric about some θ0 we derive the expression of the maximal Tukey depth. Surprisingly, the converse also holds, hence angular symmetry is completely characterized by Tukey depth. This fact puts some existing tests for centrosymmetry and for uniformity of a directional distribution in a new perspective. In the multiple regression framework, we assume that X is a (p−1)-variate r.v. and Y is a univariate r.v. such that the joint distribution of (X,Y) is again a totally general probability distribution on Rp. The concept of regression symmetry (RS) about a potential fit θ0 means that in each x the conditional probability of a positive error equals that of a negative error. If a distribution is regression symmetric about some θ0 then the maximal regression depth has a certain expression. It turns out that the converse holds as well. Therefore, regression depth characterizes the linearity of the conditional median of Y on X, which we use to construct a statistical test for linearity. [Copyright &y& Elsevier]

Our aim is to construct a factor analysis method that can resist the effect of outliers. For this we start with a highly robust initial covariance estimator, after which the factors can be obtained from maximum likelihood or from principal factor analysis (PFA). We find that PFA based on the minimum covariance determinant scatter matrix works well. We also derive the influence function of the PFA method based on either the classical scatter matrix or a robust matrix. These results are applied to the construction of a new type of empirical influence function (EIF), which is very effective for detecting influential data. To facilitate the interpretation, we compute a cutoff value for this EIF. Our findings are illustrated with several real data examples. [Copyright &y& Elsevier]

Deepest regression (DR) is a method for linear regression introduced by P. J. Rousseeuw and M. Hubert (1999, J. Amer. Statis. Assoc.94, 388–402). The DR method is defined as the fit with largest regression depth relative to the data. In this paper we show that DR is a robust method, with breakdown value that converges almost surely to 1/3 in any dimension. We construct an approximate algorithm for fast computation of DR in more than two dimensions. From the distribution of the regression depth we derive tests for the true unknown parameters in the linear regression model. Moreover, we construct simultaneous confidence regions based on bootstrapped estimates. We also use the maximal regression depth to construct a test for linearity versus convexity/concavity. We extend regression depth and deepest regression to more general models. We apply DR to polynomial regression and show that the deepest polynomial regression has breakdown value 1/3. Finally, DR is applied to the Michaelis–Menten model of enzyme kinetics, where it resolves a long-standing ambiguity. [Copyright &y& Elsevier]

When faced with high-dimensional data, one often uses principal component analysis (PCA) for dimension reduction. Classical PCA constructs a set of uncorrelated variables, which correspond to eigenvectors of the sample covariance matrix. However, it is well-known that this covariance matrix is strongly affected by anomalous observations. It is therefore necessary to apply robust methods that are resistant to possible outliers.Li and Chen [J. Am. Stat. Assoc. 80 (1985) 759] proposed a solution based on projection pursuit (PP). The idea is to search for the direction in which the projected observations have the largest robust scale. In subsequent steps, each new direction is constrained to be orthogonal to all previous directions. This method is very well suited for high-dimensional data, even when the number of variables p is higher than the number of observations n. However, the algorithm of Li and Chen has a high computational cost. In the references [C. Croux, A. Ruiz-Gazen, in COMPSTAT: Proceedings in Computational Statistics 1996, Physica-Verlag, Heidelberg, 1996, pp. 211–217; C. Croux and A. Ruiz-Gazen, High Breakdown Estimators for Principal Components: the Projection-Pursuit Approach Revisited, 2000, submitted for publication.], a computationally much more attractive method is presented, but in high dimensions (large p) it has a numerical accuracy problem and still consumes much computation time.In this paper, we construct a faster two-step algorithm that is more stable numerically. The new algorithm is illustrated on a data set with four dimensions and on two chemometrical data sets with 1200 and 600 dimensions. [Copyright &y& Elsevier]

AbstractMultivariate Singular Spectrum Analysis (MSSA) is a powerful and widely used nonparametric method for multivariate time series, which allows the analysis of complex temporal data from diverse fields such as finance, healthcare, ecology, and engineering. However, MSSA lacks robustness against outliers because it relies on the singular value decomposition, which is very sensitive to the presence of anomalous values. MSSA can then give biased results and lead to erroneous conclusions. In this paper a new MSSA method is proposed, named RObust Diagonalwise Estimation of SSA (RODESSA), which is robust against the presence of cellwise and casewise outliers. In particular, the decomposition step of MSSA is replaced by a new robust low-rank approximation of the trajectory matrix that takes its special structure into account. A fast algorithm is constructed, and it is proved that each iteration step decreases the objective function. In order to visualize different types of outliers, a new graphical display is introduced, called an enhanced time series plot. An extensive Monte Carlo simulation study is performed to compare RODESSA with competing approaches in the literature. A real data example about temperature analysis in passenger railway vehicles demonstrates the practical utility of the proposed approach. [ABSTRACT FROM AUTHOR]

Presents information on a study that proposed the distance-distance plot (D-D plot) which displays minimum covariance determinant (MCD)-based robust distances versus Mahalanobis distances. Overview on the problem used in the study; Details of the construction of the algorithm; Pseudocode of FAST-MCD; Advantages of FAST-MCD; Details on some applications to compare the FAST-MCD results with the classical mean and covariance matrix.

We propose the bagplot, a bivariate generalization of the univariate boxplot. The key notion is the halfspace location depth of a point relative to a bivariate dataset, which extends the univariate concept of rank. The "depth median" is the deepest location, and it is surrounded by a "bag" containing the n/2 observations with largest depth. Magnifying the bag by a factor 3 yields the "fence" (which is not plotted). Observations between the bag and the fence are marked by a light gray loop, whereas observations outside the fence are flagged as outliers. The bagplot visualizes the location, spread, correlation, skewness, and tails of the data. It is equivariant for linear transformations, and not limited to elliptical distributions. Software for drawing the bagplot is made available for the S-Plus and MATLAB environments. The bagplot is illustrated on several datasets--for example, in a scatterplot matrix of multivariate data. [ABSTRACT FROM AUTHOR]

The article presents a response by Peter J. Rousseeuw and colleagues to commentaries regarding their article on deepest regression method. They explained the computational formulas they developed for regression analysis. They clarified the opinions of critics on their regression depth. They also discussed the assumptions underlying each formula for regression depth.

Focuses on a study that explored the analogies between depth in regression and in location. Background of regression depth; Uses of deepest regression line; Details on the computation of multiple regression; Application of relation with location indepth.

Focuses on the intersection of the fuzzy set theory and statistics called fuzzy clustering. Comparison between the crisp clustering method and fuzzy clustering; Existence of specific clustering methods for special cases; Use of fuzziness in algorithms.

Examines the shape of correlation matrices. Correlations between three variables; Correlation between four variables; Banded correlation matrices.

In this article we introduce a new type of positive-breakdown regression method, called a generalized S-estimator (or GS-estimator), based on the minimization of a generalized M-estimator of residual scale. We compare the class of GS-estimators with the usual S-estimators, including least median of squares. It turns out that GS-estimators attain a much higher efficiency than S-estimators, at the cost of a slightly increased worst-case bias. We investigate the breakdown point, the maxbias curve, and the influence function of GS-estimators. We also give an algorithm for computing GS-estimators and apply it to real and simulated data. [ABSTRACT FROM AUTHOR]

Looks for alternatives to the median absolute deviation (MAD) that can be used as initial or ancillary scale estimates in the same way but that are more efficient and not slanted towards symmetric distributions. Simplicity of the sample median and the MAD; Advantages and disadvantages of MAD.

Detecting outliers in a multivariate point cloud is not trivial, especially when there are several outliers. The classical identification method does not always find them, because it is based on the sample mean and covariance matrix, which are themselves affected by the outliers. That is how the outliers get masked. To avoid the masking effect, we propose to compute distances based on very robust estimates of location and covariance. These robust distances are better suited to expose the outliers. In the case of regression data, the classical least squares approach masks outliers in a similar way. Also here, the outliers may be unmasked by using a highly robust regression method. Finally, a new display is proposed in which the robust regression residuals are plotted versus the robust distances. This plot classifies the data into regular observations, vertical outliers, good leverage points, and bad leverage points. Several examples are discussed. [ABSTRACT FROM AUTHOR]

Presents a reply to the comments on their article "Unmasking Multivariate Outliers and Leverage Points." Description of the projection algorithm; Data set consisting of two copies of the wood gravity data; Criticism that there were too many outliers and violation of dimensionality requirement.

Proposes a robust estimator that can be computed by means of a single pass updating mechanism. Applications to averaging; Storage and computation time; Finite-sample breakdown point of an estimator; Consistency of the estimator.

Classical least squares regression consists of minimizing the sum of the squared residuals. Many authors have produced more robust versions of this estimator by replacing the square by something else, such as the absolute value. In this article a different approach is introduced in which the sum is replaced by the median of the squared residuals. The resulting estimator can resist the effect of nearly 50% of contamination in the data. In the special case of simple regression, it corresponds to finding the narrowest strip covering half of the observations. Generalizations are possible to multivariate location, orthogonal regression, and hypothesis testing in linear models. [ABSTRACT FROM AUTHOR]

In this article, the authors define the change-of-variance curve of location M-estimators in order to investigate the infinitesimal stability of the asymptotic variance. This relatively new tool for robustness theory, bearing some resemblance to the influence curve, was discovered in 1972. The curve as well as the estimators were sometimes referred to briefly, but they not yet appeared in print explicitly. The purpose of this article is to present an exact development of these notions, including a proof of the existence of tanh-estimators, certain numerical computations of their defining constants and a theorem showing they are optimally robust. The investigation takes place in the classical framework of M-estimation of a location parameter when scale is known. Redescending M-estimators have become quite popular in the field of robust statistics, partly because of their successful performance in the Princeton Monte Carlo study. In that study, stability of the asymptotic variance. The authors also construct the so-called hyperbolic tangent estimators, proving their existence and performing certain numerical computations of their defining constants.

The article presents comments on an article published in the March 01, 2002 issue of the periodical "Journal of the American Statistical Association." Data acquisition is much less expensive today than, say 20 years ago. Therefore, modern statistical methods should be able to cope with large datasets. This is particularly true for linear regression methods, because they are widely used in many applications. Positive-breakdown estimators have been developed to protect against outlying data points, but they are often hard to compute exactly. Although, authors proposing a new algorithm usually focus on its computation time and performance, Hawkins and Olive in the first part of their article take a closer look at the breakdown value and the (in)consistency of algorithm based on elemental samples. Contrary to Hawkins and Olive, the authors find the extreme computational load of the X-cluster algorithm problematic. In their experience, practitioners can rarely he convinced to use a method that cannot be computed within reasonable time.

The minimum covariance determinant (MCD) method is a highly robust estimator of multivariate location and scatter, for which a fast algorithm is available. Since estimating the covariance matrix is the cornerstone of many multivariate statistical methods, the MCD is an important building block when developing robust multivariate techniques. It also serves as a convenient and efficient tool for outlier detection. The MCD estimator is reviewed, along with its main properties such as affine equivariance, breakdown value, and influence function. We discuss its computation, and list applications and extensions of the MCD in applied and methodological multivariate statistics. Two recent extensions of the MCD are described. The first one is a fast deterministic algorithm which inherits the robustness of the MCD while being almost affine equivariant. The second is tailored to high‐dimensional data, possibly with more dimensions than cases, and incorporates regularization to prevent singular matrices. This article is categorized under: Statistical and Graphical Methods of Data Analysis > Multivariate Analysis Statistical and Graphical Methods of Data Analysis > Robust Methods Statistical Learning and Exploratory Methods of the Data Sciences > Knowledge Discovery [ABSTRACT FROM AUTHOR]

Correct classification of breast cancer subtypes is of high importance as it directly affects the therapeutic options. We focus on triple-negative breast cancer which has the worst prognosis among breast cancer types. Using cutting edge methods from the field of robust statistics, we analyze Breast Invasive Carcinoma transcriptomic data publicly available from The Cancer Genome Atlas data portal. Our analysis identifies statistical outliers that may correspond to misdiagnosed patients. Furthermore, it is illustrated that classical statistical methods may fail to identify outliers due to their heavy influence, prompting the need for robust statistics. Using robust sparse logistic regression we obtain 36 relevant genes, of which ca. 60% have been previously reported as biologically relevant to triple-negative breast cancer, reinforcing the validity of the method. The remaining 14 genes identified are new potential biomarkers for triple-negative breast cancer. Out of these, JAM3, SFT2D2, and PAPSS1 were previously associated to breast tumors or other types of cancer. The relevance of these genes is confirmed by the new DetectDeviatingCells outlier detection technique. A comparison of gene networks on the selected genes showed significant differences between triple-negative breast cancer and non-triple-negative breast cancer data. The individual role of FOXA1 in triple-negative breast cancer and non-triple-negative breast cancer, and the strong FOXA1-AGR2 connection in triple-negative breast cancer stand out. The goal of our paper is to contribute to the breast cancer/triple-negative breast cancer understanding and management. At the same time it demonstrates that robust regression and outlier detection constitute key strategies to cope with high-dimensional clinical data such as omics data. [ABSTRACT FROM AUTHOR]

Quadratic discriminant analysis (QDA) is a widely used classification technique. Based on a training dataset, each class in the data is characterized by an estimate of its center and shape, which can then be used to assign unseen observations to one of the classes. The traditional QDA rule relies on the empirical mean and covariance matrix. Unfortunately, these estimators are sensitive to label and measurement noise which often impairs the model's predictive ability. Robust estimators of location and scatter are resistant to this type of contamination. However, they have a prohibitive computational cost for large scale industrial experiments. We present a novel QDA method based on a recent real-time robust algorithm. We additionally integrate an anomaly detection step to classify the most atypical observations into a separate class of outliers. Finally, we introduce the label bias plot, a graphical display to identify label and measurement noise in the training data. The performance of the proposed approach is illustrated in a simulation study with huge datasets, and on real datasets about diabetes and fruit. • Classical discriminant analysis is sensitive to mislabeled cases and outliers. • The new robust classification method runs much faster than existing ones. • The proposed label bias plot shows label noise and outliers in the training data. • In a classification of spectra, varying illumination setups created subgroups within classes. [ABSTRACT FROM AUTHOR]

Modern industrial machines can generate gigabytes of data in seconds, frequently pushing the boundaries of available computing power. Together with the time criticality of industrial processing this presents a challenging problem for any data analytics procedure. We focus on the deterministic minimum covariance determinant method (DetMCD), which detects outliers by fitting a robust covariance matrix. We construct a much faster version of DetMCD by replacing its initial estimators by two new methods and incorporating update-based concentration steps. The computation time is reduced further by parallel computing, with a novel robust aggregation method to combine the results from the threads. The speed and accuracy of the proposed real-time DetMCD method (RT-DetMCD) are illustrated by simulation and a real industrial application to food sorting. • Multivariate outliers can be detected relative to a robust center and covariance matrix. • Novel algorithmic techniques drastically speed up the computation of robust estimates. • New developments enable the processing of industrial data volumes. • A new aggregation method combines the results of parallel computations. • Real-time automatic food sorting is possible in milliseconds. [ABSTRACT FROM AUTHOR]