Search

Recent advances in engineering technologies have enabled the collection of a large number of longitudinal features. This wealth of information presents unique opportunities for researchers to investigate the complex nature of diseases and uncover underlying disease mechanisms. However, analyzing such kind of data can be difficult due to its high dimensionality, heterogeneity and computational challenges. In this article, we propose a Bayesian nonparametric mixture model for clustering high‐dimensional mixed‐type (eg, continuous, discrete and categorical) longitudinal features. We employ a sparse factor model on the joint distribution of random effects and the key idea is to induce clustering at the latent factor level instead of the original data to escape the curse of dimensionality. The number of clusters is estimated through a Dirichlet process prior. An efficient Gibbs sampler is developed to estimate the posterior distribution of the model parameters. Analysis of real and simulated data is presented and discussed. Our study demonstrates that the proposed model serves as a useful analytical tool for clustering high‐dimensional longitudinal data. [ABSTRACT FROM AUTHOR]

AbstractIn this work, we propose a Semi-supervised Triply Robust Inductive transFer LEarning (STRIFLE) approach, which integrates heterogeneous data from a label-rich source population and a label-scarce target population and utilizes a large amount of unlabeled data simultaneously to improve the learning accuracy in the target population. Specifically, we consider a high dimensional covariate shift setting and employ two nuisance models, a density ratio model and an imputation model, to combine transfer learning and surrogate-assisted semi-supervised learning strategies organically and achieve triple robustness. While the STRIFLE approach assumes the target and source populations to share the same conditional distribution of outcome Y given both the surrogate features S and predictors X , it allows the true underlying model of Y�X to differ between the two populations due to the potential covariate shift in S and X . Different from double robustness, even if both nuisance models are misspecified or the distribution of Y�S,X is not the same between the two populations when the shifted source population and the target population share enough similarities, the triply robust STRIFLE estimator can still partially utilize the source population, and it is guaranteed to be no worse than the target-only surrogate-assisted semi-supervised estimator with an additional error term from transferability detection. These desirable properties of our estimator are established theoretically and verified in finite samples via extensive simulation studies. We utilize the STRIFLE estimator to train a Type II diabetes polygenic risk prediction model for the African American target population by transferring knowledge from electronic health records linked genomic data observed in a larger European source population. [ABSTRACT FROM AUTHOR]

The k-Nearest Neighbors (kNN) method, established in 1951, has since evolved into a pivotal tool in data mining, recommendation systems, and Internet of Things (IoT), among other areas. This paper presents a comprehensive review and performance analysis of modifications made to enhance the exact kNN techniques, particularly focusing on kNN Search and kNN Join for high-dimensional data. We delve deep into 31 kNN search methods and 12 kNN join methods, providing a methodological overview and analytical insight into each, emphasizing their strengths, limitations, and applicability. An important feature of our study is the provision of the source code for each of the kNN methods discussed, fostering ease of experimentation and comparative analysis for readers. Motivated by the rising significance of kNN in high-dimensional spaces and a recognized gap in comprehensive surveys on exact kNN techniques, our work seeks to bridge this gap. Additionally, we outline existing challenges and present potential directions for future research in the domain of kNN techniques, offering a holistic guide that amalgamates, compares, and dissects existing methodologies in a coherent manner. [ABSTRACT FROM AUTHOR]

Abstract The k-Nearest Neighbors (kNN) method, established in 1951, has since evolved into a pivotal tool in data mining, recommendation systems, and Internet of Things (IoT), among other areas. This paper presents a comprehensive review and performance analysis of modifications made to enhance the exact kNN techniques, particularly focusing on kNN Search and kNN Join for high-dimensional data. We delve deep into 31 kNN search methods and 12 kNN join methods, providing a methodological overview and analytical insight into each, emphasizing their strengths, limitations, and applicability. An important feature of our study is the provision of the source code for each of the kNN methods discussed, fostering ease of experimentation and comparative analysis for readers. Motivated by the rising significance of kNN in high-dimensional spaces and a recognized gap in comprehensive surveys on exact kNN techniques, our work seeks to bridge this gap. Additionally, we outline existing challenges and present potential directions for future research in the domain of kNN techniques, offering a holistic guide that amalgamates, compares, and dissects existing methodologies in a coherent manner. Graphical Abstract

Geotechnical site characterization using multivariate, limited (sparse), and missing (incomplete) data is an important but challenging task, particularly in high dimensions. Toward this problem, this study proposes a Bayesian vine algorithm. In the proposed algorithm, the task of Bayesian update in higher dimensions is translated into a series of lower-dimensional (usually ≤2) update tasks using conditional correlation vine. This feature of the proposed algorithm makes it scalable and computationally efficient in higher dimensions. Multiple examples using two-dimensional (2D), five-dimensional (5D), 10-dimensional (10D), 20-dimensional (20D), 50-dimensional (50D), and 100-dimensional (100D) data are shown to demonstrate the capability of the proposed algorithm. The results suggest that the proposed algorithm can be used successfully for geotechnical site characterization. Even an ultrahigh 50D joint distribution with >1,000 parameters (1,325) can be estimated in around 20 min. The proposed algorithm can handle multivariate data sets with limited and missing values and can also handle non-Gaussian multivariate joint distributions. The proposed algorithm only considers cross-correlation in the site data and doesn't take into account spatial correlation. [ABSTRACT FROM AUTHOR]

The neural activities of the brain are detected by Electroencephalography (EEG) that allows for analysis and classification of epileptic disease. The existing methods fail to capture high-dimensional data between adjacent sequences which make it difficult for the classifier to process and maximize the classification errors. This research proposes Hierarchical Long Short-Term Memory (H-LSTM) with a skip connection-based epileptic seizure classification method. The H- LSTM captures both short and long-term dependencies between adjacent sequences in the high-dimensional data. The skip connection is introduced between H- LSTM layers, facilitating the flow of data across adjacent sequences to improve the classification performance of epileptic seizures. The datasets used to evaluate the proposed H- LSTM with skip connection-based classification method are BONN-EEG and CHB-MIT EEG. The proposed H-LSTM with skip connection method attains 99.81% accuracy on BONN - EEG while attaining 99.34% accuracy on the CHB-MIT EEG dataset which is more effective than the existing methods namely, Bidirectional Gated Recurrent Unit (Bi-GRU) and Graph Convolutional Network (GCN). [ABSTRACT FROM AUTHOR]

AbstractWe propose a distribution-free distance-based method for high dimensional change points that can address challenging situations when the sample size is very small compared to the dimension as in the so-called HDLSS data or when non-sparse changes may occur due to change in many variables but with small significant magnitudes. Our method can detect changes in mean or variance of high dimensional observations as well as other distributional changes. We present efficient algorithms that can detect single and multiple high dimensional change points. We use nonparametric metrics, including a new dissimilarity measure and some new distance and difference distance matrices, to develop a procedure to estimate change point locations. We also introduce a nonparametric test to determine the significance of estimated change points. We provide theoretical guaranties for our method and demonstrate its empirical performance in comparison with some of the recent methods for high dimensional change points. An R package called HDDchangepoint is developed to implement the proposed method. [ABSTRACT FROM AUTHOR]

Big data is growing fast following Big data has the characteristics of high volume, speed, and variety (3v) and continues to grow exponentially following the development of the world's use of information and communication technology. The main problem with big data is the data deluge. The need for technology and large data storage and processing methods to keep pace with the exponential data growth rate is potentially limitless, giving rise to the problem of exponentially increasing technology requirements as well. The weakness of previous big data analysis approaches (batch and online real time processing) is that it requires high technology (large storage, memory and processing). This paper proposes a new approach in big-data analysis by separating the basic knowledge (BK) construction process from original data with a much smaller volume (volume reduction), then analyzing it into final knowledge, thus requiring smaller/simpler analysis technology. The proposals include formulating the definition and representation of BK, developing methods for constructing BK from source data, and analyzing BK into final knowledge. We propose a BK construction method based on a knowledge extraction technique using BIRCH clustering algorithm for instance reduction, and handling high-dimensional problems by parallelizing the dimension calculation process to calculate the distance between instances. We use the Adjusted Rand Index (ARI) to measure the similarity of the final knowledge of the baseline and proposed methods. First, the BIRCH baseline was modified by parallelizing the calculations has succeeded in increasing speed from 17% to 25%. Next, the parallel BIRCH (PBIRCH) baseline was broken into BK construction and BK analysis, has succeeded in reducing volume by 96% or more and increasing speed by 43.50%, with similar final knowledge results (ARI=1). Based on these results, we conclude that the BK construction method and analysis from BK into final knowledge for highdimensional big data have significantly reduced volume and speed up the analytical process without reducing the quality of the final knowledge. [ABSTRACT FROM AUTHOR]

Interpretability and stability are two important characteristics required for the application of high dimensional data in statistics. Although the former has been favored by many existing forecasting methods to some extent, the latter in the sense of controlling the fraction of wrongly discovered features is still largely underdeveloped. Under the accelerated failure time model, this paper introduces a controlled variable selection method with the general framework of Model-X knockoffs to tackle high dimensional data. We provide theoretical justifications on the asymptotic false discovery rate (FDR) control. The proposed method has attracted significant interest due to its strong control of the FDR while preserving predictive power. Several simulation examples are conducted to assess the finite sample performance with desired interpretability and stability. A real data example from Acute Myeloid Leukemia study is analyzed to demonstrate the utility of the proposed method in practice. [ABSTRACT FROM AUTHOR]

The clustering of high-dimensional predictors draws increasing attention in various scientific areas, such as text mining and biological data analysis. In standard clustering procedures, when predictors are clustered, they only showcase the inherent patterns within the predictor set, lacking the capacity to predict the response variable. To this end, a new supervised weeding algorithm is advocated to address the dual requirement of detecting sparse clusters and capturing the prediction effects. The proposed algorithm is based on an iterative feature screening and coherence evaluation procedure. It iteratively weeds off the unimportant predictors in a backward fashion, forming sequences of nested sets to determine data-driven optimal cut-offs. This study uses Monte Carlo simulation to assess the finite-sample performance of the proposed method. The findings demonstrate that both the clustering and prediction performance of the proposed method are comparable to existing methods that concentrate solely on one aspect of the dual targets. An analysis of a job description dataset is conducted to explore significant groups of keywords that affect employees' salaries. [ABSTRACT FROM AUTHOR]

The aging process is crucial for Chinese Baijiu production, significantly enhancing the spirit's flavor, aroma and quality. However, aging involves a complex interplay of numerous compounds, and the extensive duration required for aging leads to a scarcity of samples available for scientific research. These limitations pose a challenge in analyzing high-dimensional data with collinearity, complicating the understanding of the intricate chemical processes at play. In this article, a two-step framework was proposed that integrated Relaxed Lasso regression models with Lasso-selected predictors to address this issue. Baijiu samples subjected to various aging conditions were analyzed using direct GC-MS and HS-GC-MS, and the obtained data was processed by this approach. The results demonstrate significantly superior performance compared to other methods, including PLSR and Gradient Boosting. Analyses were also performed on a previously documented dataset, yielding enhanced results and underscoring the method's advantage in processing high dimensional data with multicollinearity. Moreover, this method proved effective in screening of potential indicative compounds, highlighting its utility in Baijiu aging research.

With the development of technology and the widespread collection of data, high-dimensional data analysis has become a research hotspot in many fields. Traditional parameter methods often face problems such as dimensional disasters in high-dimensional data analysis. Non parametric methods have broad application prospects in high-dimensional data because they do not rely on specific parameter distribution assumptions. The Bayesian rule is more suitable for dealing with noise and outliers in high-dimensional data because it takes uncertainty into account. Therefore, it is of great significance to combine non parametric methods with Bayesian methods for application research in high-dimensional data analysis. In this paper, the nonparametric Bayesian method was applied to the analysis of high-dimensional data, and the Dirichlet process Mixture model was used to cluster high-dimensional data. The regression analysis of high-dimensional data was carried out through the prediction model of nonparametric Bayesian regression. In this paper, the nonparametric Bayesian method based on Bayesian sparse linear model was used for feature selection of high-dimensional data. In order to determine the superiority of nonparametric Bayesian methods in high-dimensional data analysis, this paper conducted experiments on nonparametric Bayesian methods and traditional parametric methods in high-dimensional data analysis from five aspects of cluster analysis, classification analysis, regression analysis, feature selection and anomaly detection, and evaluated them through multiple indicators. This article explored the application of non parametric Bayesian methods in high-dimensional data analysis from these aspects through simulation experiments. The experimental results show that the clustering accuracy of the non parametric Bayesian clustering algorithm was 0.93, and the accuracy of the non parametric Bayesian classification algorithm was between 0.93 and 0.99; the coefficient of determination of nonparametric Bayesian regression algorithm was 0.98; the F1 values of non parametric Bayesian methods in anomaly detection ranged from 0.86 to 0.91, which was superior to traditional methods. Non parametric Bayesian methods have broad application prospects in high-dimensional data analysis, and can be applied in multiple fields such as clustering, classification, regression, etc. [ABSTRACT FROM AUTHOR]

In prospective genomic studies (e.g. DNA methylation, metagenomics, and transcriptomics), it is crucial to estimate the overall fraction of phenotypic variance (OFPV) attributed to the high-dimensional genomic variables, a concept similar to heritability analyses in genome-wide association studies (GWAS). Unlike genetic variants in GWAS, these genomic variables are typically measured with error due to technical limitation and temporal instability. While the existing methods developed for GWAS can be used, ignoring measurement error may severely underestimate OFPV and mislead the design of future studies. Assuming that measurement error variances are distributed similarly between causal and noncausal variables, we show that the asymptotic attenuation factor equals to the average intraclass correlation coefficients of all genomic variables, which can be estimated based on a pilot study with repeated measurements. We illustrate the method by estimating the contribution of microbiome taxa to body mass index and multiple allergy traits in the American Gut Project. Finally, we show that measurement error does not cause meaningful bias when estimating the correlation of effect sizes for two traits. [ABSTRACT FROM AUTHOR]

This paper describes a correlation-based batch process for addressing high dimensional imputation problems. There are relatively few algorithms designed to efficiently handle imputation of missing data in high dimensional contexts. Fewer still are flexible enough to natively handle mixed-type data, often requiring lengthy pre-processing to get the data into proper shape, and then post-processing to return the data to usable form. Such decisions as well as assumptions made by many methods (e.g., data generating process) limit their performance, flexibility, and usability. Built on a set of complementary algorithms for nonparametric imputation via chained random forests, I introduce a batching process to ease computational costs associated with high dimensional imputation by subsetting data based on ranked cross-feature absolute correlations. The algorithm then imputes each batch separately, and joins imputed subsets in the final step. The process, hdImpute, is fast and accurate. As a result, high dimensional imputation is more accessible, and researchers are not forced to decide between speed or accuracy. Complementary software is available in the form of an R package, and is openly developed on Github under the MIT public license. In the spirit of open science, collaboration and engagement with the actively developing software are encouraged. [ABSTRACT FROM AUTHOR]

The most common preprocessing techniques used to deal with datasets having high dimensionality and a low number of instances—or wide data—are feature reduction (FR), feature selection (FS), and resampling. This study explores the use of FR and resampling techniques, expanding the limited comparisons between FR and filter FS methods in the existing literature, especially in the context of wide data. We compare the optimal outcomes from a previous comprehensive study of FS against new experiments conducted using FR methods. Two specific challenges associated with the use of FR are outlined in detail: finding FR methods that are compatible with wide data and the need for a reduction estimator of nonlinear approaches to process out-of-sample data. The experimental study compares 17 techniques, including supervised, unsupervised, linear, and nonlinear approaches, using 7 resampling strategies and 5 classifiers. The results demonstrate which configurations are optimal, according to their performance and computation time. Moreover, the best configuration—namely, k Nearest Neighbor (KNN) + the Maximal Margin Criterion (MMC) feature reducer with no resampling—is shown to outperform state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

The penalized regression is an important technique for high-dimensional data analysis, but penalized estimation method hinge on finding a suitable choice of tuning parameter. In this paper, a simple modified L curve method is proposed to select the tuning parameter for penalized estimation including Lasso, SCAD and MCP in linear regression models. Through data simulation and actual data analysis, we find that the modified L curve method can be simpler and more accurate than traditional tuning parameter selection schemes such as CV and BIC. Furthermore, the method is able to identify the true model consistently and has the less model error, especially for the cases where there is a high correlation between predictors. [ABSTRACT FROM AUTHOR]

Perhaps the first nonparametric, asymptotically optimal prediction intervals are provided for univariate random walks, with applications to renewal processes. Perhaps the first nonparametric prediction regions are introduced for vector-valued random walks. This paper further derives nonparametric data-splitting prediction regions, which are underpinned by very simple theory. Some of the prediction regions can be used when the data distribution does not have first moments, and some can be used for high-dimensional data, where the number of predictors is larger than the sample size. The prediction regions can make use of many estimators of multivariate location and dispersion. [ABSTRACT FROM AUTHOR]

The likelihood ratio test used to test complete independence cannot be used in high-dimensional data. In literature, many test statistics were proposed to cope with this problem. However, these statistics are sensitive to outliers. In this study, we suggest a test statistic that is not sensitive to outliers and can be used in high-dimensional data to test complete independence. We investigate the performance of our suggested statistic in terms of the empirical size and the empirical power by using three simulation studies and a real example dataset. And, we construct an R package to implement our proposed test statistic on real high-dimensional data in applications. [ABSTRACT FROM AUTHOR]

Learning from high dimensional data has been utilized in various applications such as computational biology, image classification, and finance. Most classical machine learning algorithms fail to give accurate predictions in high dimensional settings due to the enormous feature space. In this article, we present a novel ensemble of classification trees based on weighted random subspaces that aims to adjust the distribution of selection probabilities. In the proposed algorithm base classifiers are built on random feature subspaces in which the probability that influential features will be selected for the next subspace, is updated by incorporating grouping information based on previous classifiers through a weighting function. As an interpretation tool, we show that variable importance measures computed by the new method can identify influential features efficiently. We provide theoretical reasoning for the different elements of the proposed method, and we evaluate the usefulness of the new method based on simulation studies and real data analysis. [ABSTRACT FROM AUTHOR]

Perhaps the first nonparametric, asymptotically optimal prediction intervals are provided for univariate random walks, with applications to renewal processes. Perhaps the first nonparametric prediction regions are introduced for vector-valued random walks. This paper further derives nonparametric data-splitting prediction regions, which are underpinned by very simple theory. Some of the prediction regions can be used when the data distribution does not have first moments, and some can be used for high-dimensional data, where the number of predictors is larger than the sample size. The prediction regions can make use of many estimators of multivariate location and dispersion.

We develop a method for constructing prediction intervals for a nonstationary variable, such as GDP. The method uses a Factor Augmented Regression (FAR) model. The predictors in the model include a small number of factors generated to extract most of the information in a set of panel data on a large number of macroeconomic variables that are considered to be potential predictors. The novelty of this article is that it provides a method and justification for a mixture of stationary and nonstationary factors as predictors in the FAR model; we refer to this as mixture-FAR method. This method is important because typically such a large set of panel data, for example the FRED-QD, is likely to contain a mixture of stationary and nonstationary variables. In our simulation study, we observed that the proposed mixture-FAR method performed better than its competitor that requires all the predictors to be nonstationary; the MSE of prediction was at least 33% lower for mixture-FAR. Using the data in FRED-QD for the United States, we evaluated the aforementioned methods for forecasting the nonstationary variables, GDP and Industrial Production. We observed that the mixture-FAR method performed better than its competitors. [ABSTRACT FROM AUTHOR]

A wealth of gene expression data generated by high‐throughput techniques provides exciting opportunities for studying gene‐gene interactions systematically. Gene‐gene interactions in a biological system are tightly regulated and are often highly dynamic. The interactions can change flexibly under various internal cellular signals or external stimuli. Previous studies have developed statistical methods to examine these dynamic changes in gene‐gene interactions. However, due to the massive number of possible gene combinations that need to be considered in a typical genomic dataset, intensive computation is a common challenge for exploring gene‐gene interactions. On the other hand, oftentimes only a small proportion of gene combinations exhibit dynamic co‐expression changes. To solve this problem, we propose Bayesian variable selection approaches based on spike‐and‐slab priors. The proposed algorithms reduce the computational intensity by focusing on identifying subsets of promising gene combinations in the search space. We also adopt a Bayesian multiple hypothesis testing procedure to identify strong dynamic gene co‐expression changes. Simulation studies are performed to compare the proposed approaches with existing exhaustive search heuristics. We demonstrate the implementation of our proposed approach to study the association between gene co‐expression patterns and overall survival using the RNA‐sequencing dataset from The Cancer Genome Atlas breast cancer BRCA‐US project. [ABSTRACT FROM AUTHOR]

Identifying the existence and locations of change points has been a broadly encountered task in many statistical application areas. The existing change point detection methods may produce unsatisfactory results for high‐dimensional data since certain distributional assumptions are made on data, which are hard to verify in practice. Moreover, some parameters (such as the number of change points) need to be estimated beforehand for some methods, making their powers sensitive to these values. Here, we propose a kernel‐based U$$ U $$‐statistic to identify change points (KUCP) for high dimensional data, which is free of distributional assumptions and sup‐parameter estimations. Specifically, we employ a kernel function to describe similarities among the subjects and construct a U$$ U $$‐statistic to test the existence of change point for a given location. The asymptotic properties of the U$$ U $$‐statistic are deduced. We also develop a procedure to locate the change points sequentially via a dichotomy algorithm. Extensive simulations demonstrate that KUCP has higher sensitivity in identifying existence of change points and higher accuracy in locating these change points than its counterparts. We further illustrate its practical utility by analyzing a gene expression data of human brain to detect the time point when gene expression profiles begin to change, which has been reported to be closely related with aging brain. [ABSTRACT FROM AUTHOR]

The maximum inner product search (MIPS), which finds the item with the highest inner product with a given query user, is an essential problem in the recommendation field. Usually e-commerce companies face situations where they want to promote and sell new or discounted items. In these situations, we have to consider the following questions: Who is interested in the items, and how do we find them? This article answers this question by addressing a new problem called reverse maximum inner product search (reverse MIPS). Given a query vector and two sets of vectors (user vectors and item vectors), the problem of reverse MIPS finds a set of user vectors whose inner product with the query vector is the maximum among the query and item vectors. Although the importance of this problem is clear, its straightforward implementation incurs a computationally expensive cost. We therefore propose Simpfer, a simple, fast, and exact algorithm for reverse MIPS. In an offline phase, Simpfer builds a simple index that maintains a lower bound of the maximum inner product. By exploiting this index, Simpfer judges whether the query vector can have the maximum inner product or not, for a given user vector, in a constant time. Our index enables filtering user vectors, which cannot have the maximum inner product with the query vector, in a batch. We theoretically demonstrate that Simpfer outperforms baselines employing state-of-the-art MIPS techniques. In addition, we answer two new research questions. Can approximation algorithms further improve reverse MIPS processing? Is there an exact algorithm that is faster than Simpfer? For the former, we show that approximation with quality guarantee provides a little speed-up. For the latter, we propose Simpfer++, a theoretically and practically faster algorithm than Simpfer. Our extensive experiments on real datasets show that Simpfer is at least two orders of magnitude faster than the baselines, and Simpfer++ further improves the online processing time. [ABSTRACT FROM AUTHOR]

Anomaly detection has an active research contribution to large-scale industrial production equipment; time-based performance indicators; and frequent healthcare monitoring queries. The network behavior identified in time series data monitoring for different analyses and results is irrelevant due to high dimensional data. The existing neural network frameworks do not provide continuous network behaviors and the computation process does not fulfill the time series analysis [1] [2]. The proposed system is integrated with long-term memory access and a time series data behavior analysis system. Long-term memory access processes time series context and accurately predicts behavior. The linear neural network model trains the highdimensional data, extracts the content, and circulates it to get high accuracy [1] [2]. The linear neural network model supports a distribution system to train the network behavior identification. We propose a linear neural network model with a long-term memory access experiment in time series data to show its frequent efficiency and accuracy. The experimental results show that the network behavior prediction accuracy is as high as 99.95%, which is a highly improved one than any other conventional model that did not utilize time series data. [ABSTRACT FROM AUTHOR]

Suspicious events or objects that differ from the norm in data can be discovered using anomaly identification. Identifying anomalies is critical for many applicable domains of life, e.g., preventing credit card theft and spotting intrusions into networks. It is possible to detect anomalies on a global scale as well as at the local level. A global outlier is a data point beyond the norm for the entire dataset, while a local outlier may be inside the norm for the entire dataset but outside the surrounding data points. Data outlier identification methods that are performed locally are inadequate. Therefore, better algorithms are required to investigate the high velocity of data and identify local outliers. Machine learning and data mining techniques need to be investigated to determine the pros and cons of anomaly identification residing inside data. The density based LOF method can be applied as the best choice to identify local outliers. While many variants of LOF exist for low-dimensional data, none are suitable for high-dimensional data. This research study discusses LOF, COF, and CBLOF methods for spotting local outliers in low and high-dimensional data. Regarding the size of the dimension, the performance of density-based algorithms is examined based on accuracy and time complexity. In this scenario, CBLOF achieves outstanding results due to its distinctive method of employing cluster-based local outlier detection.

With the development of IoT monitoring equipment, an increasing number of monitoring indicators are employed to monitor the operational status of water pumps, thereby resulting in the challenge of data redundancy. This paper proposes an algorithm for predicting the health status of pumps that integrates multiple monitoring variables. Initially, the original dataset is classified using the maximum relevance minimum redundancy method. Next, principal component dimensionality reduction is used to reduce the dimensionality of the classified dataset. Finally, a long and short term memory neural network is employed to construct the association model between monitoring data and equipment health. The proposed algorithm takes into account the correlation between variables and the negative impacts of long-term dependence on the prediction results. It is capable of predicting abnormal working conditions, which has been experimentally verified in the Xiasha Pumping Station located in Hangzhou. The algorithm was compared with LR, SVM, and RNN algorithms, and it was found that the proposed algorithm achieved the highest prediction accuracy. HIGHLIGHTS Sensors measuring vibration, magnetic flux, temperature, and sound loudness have been installed on a water pump to collect data on its operational status.; To address the high dimensionality of the monitoring data, the mRMR-principal component analysis method was employed to reduce the impact of irrelevant variables on the results.; Long short-term memory was used to establish associations between the health status and monitoring data.;

Abstract Background Feature selection is important in high dimensional data analysis. The wrapper approach is one of the ways to perform feature selection, but it is computationally intensive as it builds and evaluates models of multiple subsets of features. The existing wrapper algorithm primarily focuses on shortening the path to find an optimal feature set. However, it underutilizes the capability of feature subset models, which impacts feature selection and its predictive performance. Method and Results This study proposes a novel Artificial Intelligence based Wrapper (AIWrap) algorithm that integrates Artificial Intelligence (AI) with the existing wrapper algorithm. The algorithm develops a Performance Prediction Model using AI which predicts the model performance of any feature set and allows the wrapper algorithm to evaluate the feature subset performance in a model without building the model. The algorithm can make the wrapper algorithm more relevant for high-dimensional data. We evaluate the performance of this algorithm using simulated studies and real research studies. AIWrap shows better or at par feature selection and model prediction performance than standard penalized feature selection algorithms and wrapper algorithms. Conclusion AIWrap approach provides an alternative algorithm to the existing algorithms for feature selection. The current study focuses on AIWrap application in continuous cross-sectional data. However, it could be applied to other datasets like longitudinal, categorical and time-to-event biological data.

Background: Feature selection is important in high dimensional data analysis. The wrapper approach is one of the ways to perform feature selection, but it is computationally intensive as it builds and evaluates models of multiple subsets of features. The existing wrapper algorithm primarily focuses on shortening the path to find an optimal feature set. However, it underutilizes the capability of feature subset models, which impacts feature selection and its predictive performance. Method and Results: This study proposes a novel Artificial Intelligence based Wrapper (AIWrap) algorithm that integrates Artificial Intelligence (AI) with the existing wrapper algorithm. The algorithm develops a Performance Prediction Model using AI which predicts the model performance of any feature set and allows the wrapper algorithm to evaluate the feature subset performance in a model without building the model. The algorithm can make the wrapper algorithm more relevant for high-dimensional data. We evaluate the performance of this algorithm using simulated studies and real research studies. AIWrap shows better or at par feature selection and model prediction performance than standard penalized feature selection algorithms and wrapper algorithms. Conclusion: AIWrap approach provides an alternative algorithm to the existing algorithms for feature selection. The current study focuses on AIWrap application in continuous cross-sectional data. However, it could be applied to other datasets like longitudinal, categorical and time-to-event biological data. [ABSTRACT FROM AUTHOR]

The main aim of this paper is to propose a novel method (RMD-MRCD-PCA) of identification of High Leverage Points (HLPs) in high-dimensional sparse data. It is to address the weakness of the Robust Mahalanobis Distance (RMD) method which is based on the Minimum Regularized Covariance Determinant (RMD-MRCD), which indicates a decrease in its performance as the number of independent variables (p) increases. The RMD-MRCD-PCA is developed by incorporating the Principal Component Analysis (PCA) in the MRCD algorithm whereby this robust approach shrinks the covariance matrix to make it invertible and thus, can be employed to compute the RMD for high dimensional data. A simulation study and two real data sets are used to illustrate the merit of our proposed method compared to the RMD-MRCD and Robust PCA (ROBPCA) methods. Findings show that the performance of the RMD-MRCD is similar to the performance of the RMD-MRCD-PCA for p close to 200. However, its performance tends to decrease when the number of p is more than 200 and worsens at p equals 700 and larger. On the other hand, the ROBPCA is not effective for less than 20% contamination as it suffers from serious swamping problems. [ABSTRACT FROM AUTHOR]

This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc. [ABSTRACT FROM AUTHOR]

The most common preprocessing techniques used to deal with datasets having high dimensionality and a low number of instances—or wide data—are feature reduction (FR), feature selection (FS), and resampling. This study explores the use of FR and resampling techniques, expanding the limited comparisons between FR and filter FS methods in the existing literature, especially in the context of wide data. We compare the optimal outcomes from a previous comprehensive study of FS against new experiments conducted using FR methods. Two specific challenges associated with the use of FR are outlined in detail: finding FR methods that are compatible with wide data and the need for a reduction estimator of nonlinear approaches to process out-of-sample data. The experimental study compares 17 techniques, including supervised, unsupervised, linear, and nonlinear approaches, using 7 resampling strategies and 5 classifiers. The results demonstrate which configurations are optimal, according to their performance and computation time. Moreover, the best configuration—namely, k Nearest Neighbor (KNN) + the Maximal Margin Criterion (MMC) feature reducer with no resampling—is shown to outperform state-of-the-art algorithms.

Summary: This work is motivated by the need to accurately model a vector of responses related to pediatric functional status using administrative health data from inpatient rehabilitation visits. The components of the responses have known and structured interrelationships. To make use of these relationships in modeling, we develop a two‐pronged regularization approach to borrow information across the responses. The first component of our approach encourages joint selection of the effects of each variable across possibly overlapping groups of related responses and the second component encourages shrinkage of effects towards each other for related responses. As the responses in our motivating study are not normally‐distributed, our approach does not rely on an assumption of multivariate normality of the responses. We show that with an adaptive version of our penalty, our approach results in the same asymptotic distribution of estimates as if we had known in advance which variables have non‐zero effects and which variables have the same effects across some outcomes. We demonstrate the performance of our method in extensive numerical studies and in an application in the prediction of functional status of pediatric patients using administrative health data in a population of children with neurological injury or illness at a large children's hospital. [ABSTRACT FROM AUTHOR]

Visualization techniques can aid in the comprehension and appreciation of the inherent connections between ecological and economic data. However, traditional visualization techniques still suffer from the problems of information overload and missing information. To address this problem, the study proposed a technique to improve the clustered data based on force-directed edge binding of segmented skeletons and constructed a visual analysis model for ecological and economic data based on the improved technique. The study used the parallel coordinates technique, the technique of presenting complex data relationships in parallel coordinates, as the control experimental group in the technical performance comparison test. The results showed that the data relevance indexes of the improved technique and all the control experimental groups were 0.145 and 0.138, respectively. The maximum positive value of the accuracy error of the number of clusters in the improved model was 0.012, while the minimum negative value was -0.003. The control experimental groups were 0.022, -0.016 and 0.022, -0.018, respectively. The study proposed a superior visual analysis model for ecological and economic research. The model utilized a force-directed edge-bound clustering technique based on segmented skeleton data, providing accurate and comprehensive insights. The model could provide more accurate and comprehensive data analysis and decision-making basis to promote the development and sustainability of ecological economy. [ABSTRACT FROM AUTHOR]

A robust and sparse estimator for multinomial regression is proposed for high dimensional data. Robustness of the estimator is achieved by trimming the observations, and sparsity of the estimator is obtained by the elastic net penalty. In contrast to multi-group classifiers based on dimension reduction, this model is very appealing in terms of interpretation, since one obtains estimated coefficients individually for every group, and also the sparsity of the coefficients is group specific. Simulation studies are conducted to show the performance in comparison to the non-robust version of the multinomial regression estimator, and some real data examples underline the usefulness of this robust estimator particularly in terms of result interpretation and model diagnostics. [ABSTRACT FROM AUTHOR]

The two-stage feature screening method for linear models applies dimension reduction at first stage to screen out nuisance features and dramatically reduce the dimension to a moderate size; at the second stage, penalized methods such as LASSO and SCAD could be applied for feature selection. A majority of subsequent works on the sure independent screening methods have focused mainly on the linear model. This motivates us to extend the independence screening method to generalized linear models, and particularly with binary response by using the point-biserial correlation. We develop a two-stage feature screening method called point-biserial sure independence screening (PB-SIS) for high-dimensional generalized linear models, aiming for high selection accuracy and low computational cost. We demonstrate that PB-SIS is a feature screening method with high efficiency. The PB-SIS method possesses the sure independence property under certain regularity conditions. A set of simulation studies are conducted and confirm the sure independence property and the accuracy and efficiency of PB-SIS. Finally we apply PB-SIS to one real data example to show its effectiveness. [ABSTRACT FROM AUTHOR]

Analysis of genome-wide association studies (GWAS) is characterized by a large number of univariate regressions where a quantitative trait is regressed on hundreds of thousands to millions of single-nucleotide polymorphism (SNP) allele counts, one at a time. This article proposes an estimator of the SNP heritability of the trait, defined here as the fraction of the variance of the trait explained by the SNPs in the study. The proposed GWAS heritability (GWASH) estimator is easy to compute, highly interpretable, and is consistent as the number of SNPs and the sample size increase. More importantly, it can be computed from summary statistics typically reported in GWAS, not requiring access to the original data. The estimator takes full account of the linkage disequilibrium (LD) or correlation between the SNPs in the study through moments of the LD matrix, estimable from auxiliary datasets. Unlike other proposed estimators in the literature, we establish the theoretical properties of the GWASH estimator and obtain analytical estimates of the precision, allowing for power and sample size calculations for SNP heritability estimates, and forming a firm foundation for future methodological development.