Search

Container-based virtualization has gradually become a main solution in today's cloud computing environments. Detecting and analyzing anomaly in containers present a major challenge for cloud vendors and users. This paper proposes an online container anomaly detection system by monitoring and analyzing multidimensional resource metrics of the containers based on the optimized isolation forest algorithm. To improve the detection accuracy, it assigns each resource metric a weight and changes the random feature selection in the isolation forest algorithm to the weighted feature selection according to the resource bias of the container. In addition, it can identify abnormal resource metrics and automatically adjust the monitoring period to reduce the monitoring delay and system overhead. Moreover, it can locate the cause of the anomalies via analyzing and exploring the container log. The experimental results demonstrate the performance and efficiency of the system on detecting the typical anomalies in containers in both simulated and real cloud environments.

To evaluate quantities in power transmission and transformation projects， an intelligent evaluation method of quantity threshold based on isolated forest and multi-layer perceptron is proposed. Firstly， the Pearson correlation coefficient and permutation feature importance are applied to explore the relationship between various quantities and related technical features and select the features. On this basis， a neural network predictive model is trained by using five-year historical data. Aiming at the problem that the historical data may contain a lot of noises， a data rationality selection method based on isolation forest is proposed and the accuracy of using the proposed to evaluate quantities is verified in test set. The case simulation shows that the proposed feature selection method can improve the evaluation accuracy of neural network. At the same time， the proposed method can provide both rough estimation and intelligent evaluation on the rationality of quantities and can automatically evaluate the quantity rationality for promoting the digital transformation of power grid infrastructure.

Soil total nitrogen is one of the most important basic indicators for fertiliser decision making, but tens of millions of soil total nitrogen sampling data have been accumulated, forming a huge database. In this large database, there is a large amount of anomalous data, which can interfere with data analysis, affect the construction of spatial interpolation and prediction models, and then affect the accuracy of nutrient management decisions. The traditional method of identifying soil total nitrogen anomalies based on boxplots suffers from the problems of not being able to identify local anomalies, which can easily lead to misclassification of soil total nitrogen data anomalies, and the detection efficiency is not high. We propose a method to identify soil total nitrogen outliers by combining the Isolation Forest algorithm and local spatial autocorrelation analysis, which can simultaneously detect global and local outliers from large amounts of data and combine organic matter as an auxiliary indicator in the spatial analysis to help judge local outliers. Finally, the results of global and local anomalies were combined to provide a comprehensive assessment of the soil nitrogen data, avoiding the misjudgement or omission of judgement that can occur when using a single method. Using 25,930 soil test data from Yunnan Province in 2009 as an example, we compared and analysed the typical boxplot method and the unsupervised OneClassSVM method and evaluated the performance of each method in terms of correct detection rate, false positive rate and false negative rate. The results show that the proposed method has a correct detection rate (TR) of 99.97%, a false positive rate (FPR) of 8.06% and a false negative rate (FNR) of 0.01% on the data, which shows high validity and accuracy; it is also comparable to the independent isolated forests (FNR = 4.76%), boxplot (FNR = 3.90%) and OneClassSVM (FNR = 4.77%), and the false negative rate is reduced by 4.75%, 3.89% and 4.76%, respectively. [ABSTRACT FROM AUTHOR]

Short-term load forecasting is a prerequisite for achieving intra-day energy management and optimal scheduling in integrated energy systems. Its prediction accuracy directly affects the stability and economy of the system during operation. To improve the accuracy of short-term load forecasting, this paper proposes a multi-load forecasting method for integrated energy systems based on the Isolation Forest and dynamic orbit algorithm. First, a high-dimensional data matrix is constructed using the sliding window technique and the outliers in the high-dimensional data matrix are identified using Isolation Forest. Next, the hidden abnormal data within the time series are analyzed and repaired using the dynamic orbit algorithm. Then, the correlation analysis of the multivariate load and its weather data is carried out by the AR method and MIC method, and the high-dimensional feature matrix is constructed. Finally, the prediction values of the multi-load are generated based on the TCN-MMoL multi-task training network. Simulation analysis is conducted using the load data from a specific integrated energy system. The results demonstrate the proposed model's ability to significantly improve load forecasting accuracy, thereby validating the correctness and effectiveness of this forecasting approach. [ABSTRACT FROM AUTHOR]

Railway switch and crossing (S&C) systems have complicated moving structures compared with regular rail. They require multiple components that vary in complexity. The complexity of railway S&C, together with the fact that they are discontinuous points of the system, makes them vulnerable to defects such as squats. A squat on the switching rail could potentially cause rail breakage and lead to catastrophic results, such as derailment. In this study, a method based on anomaly scoring was investigated to estimate the status of an S&C system with respect to squat defects. The proposed method was tested in a real environment under controlled measurement sequences. The results show that the methods can differ between an S&C with squats and another one without them. [ABSTRACT FROM AUTHOR]

To compute the frequent metamorphosis of the housing price, the House Price Index (HPI) is one of the effective indicators. Various methodologies are involved in data processing the current house prices, which are affected by factors like house configuration, building class, air conditioning quality, etc. Remarkably, more research papers adopting classical machine learning approaches are introduced to estimate house sale prices accurately. Still, they barely regard the data processing techniques that make the data suitable for modeling more accurate house prices forecasting architectures. This research contributes to a wide variety of adequate data pre-processing. It highlights mechanisms like missingness of data, missing data handling, categorical feature encoding, discretization, outliers, and feature scaling extensively to build efficient predictive models. Comprehensive arguments have been broadly presented to portray the advantages and disadvantages of prevailed data pre-processing techniques at various distribution scenarios of variables in the house price data. The current research conclusions oblige the evolution of modern data-driven research in machine learning. [ABSTRACT FROM AUTHOR]

Abstract Multiple statistical algorithms have been used for species distribution modelling (SDM). Due to shortcomings in species occurrence datasets, presence‐only methods (such as MaxEnt) have become increasingly widely used. However, sampling bias remains a challenging issue, particularly for density‐based approaches. The Isolation Forest (iForest) algorithm is a presence‐only method less sensitive to sampling patterns and over‐fitting because it fits the model by describing the unsuitable instead of suitable conditions. Here, we present the itsdm package for species distribution modelling with iForest, which provides a workflow wrapper for the algorithms in iForest family and convenient tools for model diagnostic and post‐modelling analysis. itsdm allows users to fit and evaluate an iForest SDM using presence‐only occurrence data. It also helps the users to understand relationships between species and the living environment using Shapley values, a suggested technique in explainable artificial intelligence (xAI). Additionally, itsdm can make spatial response maps that indicate how species respond to environmental variables across space and detect areas potentially affected by a changing environment. We demonstrated the usage of the itsdm package and compared iForest with other mainstream SDMs using virtual species. The results enlightened that iForest is an advantageous presence‐only SDM when the actual distribution range is unclear.

Today, Permanent Magnet Synchronous Motors (PMSMs) are a dominant choice in industry applications. During operation, different possible faults in the system can occur, so early and automated fault detection and severity estimation are required to ensure smooth operation and optimal maintenance planning. In this direction, outlier detection methods are employed in this paper. The motor’s current signals are used to extract useful indicators of the fault, along with d-q transform. Statistical indicators in both time and frequency domains are selected to describe fault-related patterns. Based on the extracted features, three outlier detection methods are investigated: the Isolation Forest, the One Class Support Vector Machine, and the Robust Covariance Ellipse. Each method is investigated through different model parameters to evaluate fault detection and severity estimation capabilities. Finally, an ensemble approach is proposed based on decisions and outlier score ensemble. The proposed methodology is verified through different operating conditions in a PMSM test bench.

Stock market manipulation, defined as any attempt to artificially influence stock prices, poses significant challenges by causing financial losses and eroding investor trust. The prevalent reliance on supervised learning models for detecting such manipulations, while showing promise, faces notable hurdles due to the dearth of labeled data and the inability to recognize novel manipulation tactics beyond those explicitly labeled. This study ventures into addressing these gaps by proposing a novel detection framework aimed at identifying suspicious hourly manipulation blocks through an unsupervised learning approach, thereby circumventing the limitations of data labeling and enhancing the adaptability to emerging manipulation strategies. Our methodology involves the innovative creation of features reflecting the behavior of stocks across various time windows followed by the segmentation of the dataset into k subsets. This setup facilitates the identification of potential manipulation instances via a voting ensemble composed of k isolation forest models, which have been chosen for their efficiency in pinpointing anomalies and their linear computational complexity—attributes that are critical for analyzing vast datasets. Evaluated against eight real stocks known to have undergone manipulation, our approach demonstrated a remarkable capability to identify up to 89% of manipulated blocks, thus significantly outperforming previous methods that do not utilize a voting ensemble. This finding not only surpasses the detection rates reported in prior studies but also underscores the enhanced robustness and adaptability of our unsupervised model in uncovering varied manipulation schemes. Through this research, we contribute to the field by offering a scalable and efficient unsupervised learning strategy for stock manipulation detection, thereby marking a substantial advancement over traditional supervised methods and paving the way for more resilient financial markets.

Soft sensors have been extensively utilized to approximate real-time power prediction in wind power generation, which is challenging to measure instantaneously. The short-term forecast of wind power aims at providing a reference for the dispatch of the intraday power grid. This study proposes a soft sensor model based on the Long Short-Term Memory (LSTM) network by combining data preprocessing with Variational Modal Decomposition (VMD) to improve wind power prediction accuracy. It does so by adopting the isolation forest algorithm for anomaly detection of the original wind power series and processing the missing data by multiple imputation. Based on the process data samples, VMD technology is used to achieve power data decomposition and noise reduction. The LSTM network is introduced to predict each modal component separately, and further sum reconstructs the prediction results of each component to complete the wind power prediction. From the experimental results, it can be seen that the LSTM network which uses an Adam optimizing algorithm has better convergence accuracy. The VMD method exhibited superior decomposition outcomes due to its inherent Wiener filter capabilities, which effectively mitigate noise and forestall modal aliasing. The Mean Absolute Percentage Error (MAPE) was reduced by 9.3508%, which indicates that the LSTM network combined with the VMD method has better prediction accuracy.

Proteomic analysis of extracellular vesicles presents several challenges due to the unique nature of these small membrane-bound structures. Alternative analyses could reveal outcomes hidden from standard statistics to explore and develop potential new biological hypotheses that may have been overlooked during the initial evaluation of the data. An analysis sequence focusing on deviating protein expressions from donors’ primary cells was performed, leveraging machine-learning techniques to analyze small datasets, and it has been applied to evaluate extracellular vesicles’ protein content gathered from mesenchymal stem cells cultured on bioactive glass discs doped or not with metal ions. The goal was to provide additional opportunities for detecting details between experimental conditions that are not entirely revealed with classic statistical inference, offering further insights regarding the experimental design and assisting the researchers in interpreting the outcomes. The methodology extracted a set of EV-related proteins whose differences between conditions could be partially explainable with statistics, suggesting the presence of other factors involved in the bioactive glasses’ interactions with tissues. Outlier identification of extracellular vesicles’ protein expression levels related to biomaterial preparation was instrumental in improving the interpretation of the experimental outcomes.

Hyperspectral anomaly detection is used to recognize unusual patterns or anomalies in hyperspectral data. Currently, many spectral–spatial detection methods have been proposed with a cascaded manner; however, they often neglect the complementary characteristics between the spectral and spatial dimensions, which easily leads to yield high false alarm rate. To alleviate this issue, a spectral–spatial information fusion (SSIF) method is designed for hyperspectral anomaly detection. First, an isolation forest is exploited to obtain spectral anomaly map, in which the object-level feature is constructed with an entropy rate segmentation algorithm. Then, a local spatial saliency detection scheme is proposed to produce the spatial anomaly result. Finally, the spectral and spatial anomaly scores are integrated together followed by a domain transform recursive filtering to generate the final detection result. Experiments on five hyperspectral datasets covering ocean and airport scenes prove that the proposed SSIF produces superior detection results over other state-of-the-art detection techniques.

Copyright of Zhejiang Electric Power is the property of Zhejiang Electric Power Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Outliers of ship trajectory from the Automatic Identification System (AIS) onboard a ship will affect the accuracy of maritime situation awareness, especially for a regular ship trajectory mixed with a spoofing ship, which has an unauthorized Maritime Mobile Service Identification code (MMSI) owned by a regular ship. As has been referred to in the literature, the trajectory of these spoofing ships would simply be removed, and more AIS data would be lost. The pre-processing of AIS data should aim to retain more information, which is more helpful in maritime situation awareness for the Maritime Safety Administration (MSA). Through trajectory feature mining, it has been found that there are obvious differences between the trajectory of a regular ship and that of a regular ship mixed with a spoofing ship, such as in terms of speed and distance between adjacent trajectory points. However, there can be a long update time interval in the results of severe missing trajectories of a ship, bringing challenges in terms of the identification of spoofing ships. In order to accurately divide the regular ship trajectory and spoofing ship trajectory, combined with trajectory segmentation by the update time interval threshold, the isolation forest was adopted in this work to train the labeled trajectory point of a regular ship mixed with a spoofing ship. The experimental results show that the average accuracy of the identification of spoofing ships using isolation forest is 88.4%, 91%, 93.1%, and 93.3%, corresponding to different trajectory segmentation by update time intervals (5 h, 10 h, 15 h, and 20 h). The research conducted in this study can almost eliminate the outliers of ship trajectory, and it also provides help for maritime situation awareness for the MSA. [ABSTRACT FROM AUTHOR]

This publication presents a system for integrating machine learning and artificial intelligence solutions with building automation systems. The platform is based on cloud solutions and can integrate with one of the most popular virtual building management solutions, HomeAssistant. The System uses communication based on the Message Queue Telemetry Transport (MQTT) protocol. The example machine learning function described in this publication detects anomalies in the electricity waveforms and raises the alarm. This information determines power quality and detects system faults or unusual power consumption. Recently, increasing electricity prices on global markets have meant that buildings must significantly reduce consumption. Therefore, a fundamental element of energy consumption diagnostics requires detecting unusual forms of energy consumption to optimise the use of individual devices in home and office installations. [ABSTRACT FROM AUTHOR]

We propose a method called Bilateral‐Weighted Online Adaptive Isolation Forest (BWOAIF) for unsupervised anomaly detection based on Isolation Forest (IF), which is applicable to streaming data and able to cope with concept drift. Similar to IF, the proposed method has only few hyperparameters whose effect on the performance are easy to interpret by human intuition and therefore easy to tune. BWOAIF ingests data and classifies it as normal or anomalous, and simultaneously adapts its classifier by removing old trees as well as by creating new ones. We show that BWOAIF adapts gradually to slow concept drifts, and, at the same time, it is able to adapt fast to sudden changes of the data distribution. Numerical results show the efficacy of the proposed algorithm and its ability to learn different classes of concept drifts, such as slow/fast concept shift, concept split, concept appearance, and concept disappearance. [ABSTRACT FROM AUTHOR]

Parking space is usually very limited in major cities, especially Cairo, leading to traffic congestion, air pollution, and driver frustration. Existing car parking systems tend to tackle parking issues in a non-digitized manner. These systems require the drivers to search for an empty parking space with no guarantee of finding any wasting time, resources, and causing unnecessary congestion. To address these issues, this paper proposes a digitized parking system with a proof-of-concept implementation that combines multiple technological concepts into one solution with the advantages of using IoT for real-time tracking of parking availability. User authentication and automated payments are handled using a quick response (QR) code on entry and exit. Some experiments were done on real data collected for six different locations in Cairo via a live popular times library. Several machine learning models were investigated in order to estimate the occupancy rate of certain places. Moreover, a clear analysis of the differences in performance is illustrated with the final model deployed being XGboost. It has achieved the most efficient results with a R2 score of 85.7%. [ABSTRACT FROM AUTHOR]

Insider attacks may inflict far greater damage to an organization than outsider threats since insiders are authorized users who are acquainted with the business’s system, making detection harder. Many techniques to detecting insider threats have been developed, but they are neither flexible nor resilient owing to different obstacles (e.g., lack of real-world dataset and highly skewed class distribution of the available dataset), making insider threat detection an understudied research field. Previous techniques attempted to solve the dataset’s imbalance issue by increasing or lowering the observations of the dataset’s classes, however this might lead to underfitting and overfitting problems. We present an insider threat detection model that addresses the class imbalance problem at the algorithm level using anomaly-based techniques, as an enhancement over previous approaches. To limit the effect of skewed class distribution on insider threat detection, the Isolation Forest (IF) technique is used. The model is verified using the benchmarked CERT’s insider threat dataset, which is significantly unbalanced, with a small number of malicious cases vs a large number of non-malicious instances. Several contamination ratios of IF’s parameters are used to verify the model’s performance throughout a range of anomaly scores. The experimental findings reveal that the suggested model handles the dataset class imbalance problem with an accuracy score of 98%. The findings are compared to the baseline technique to demonstrate how the proposed model enhances detection performance and addresses the problem of data imbalance. The findings indicate the usefulness of the suggested approach for identifying insider threats when compared to previous studies.

Electricity demand is increasing proportionally to the increase in power usage. Without a doubt, energy efficiency has gained significant importance and attention, with one of the primary concerns being the detection and forecasting of abnormal consumption. In this paper, the authors proposed a method to predict the occurrence of abnormal consumption behavior in advance. The proposed method utilizes the Isolation Forest algorithm to label the smart meter electricity consumption readings as normal or abnormal. It generates a sequence of data with varying lengths. Based on the data sequence, two supervised machine learning algorithms, Random Forest, and Decision Tree were developed to forecast the occurrence of power anomaly consumption. Experiment results showed that the proposed methods consistently detect and predict the abnormal status 30 minutes ahead. There is no significant difference between Random Forest and Decision Tree performance on different smart meter readings, dataset sizes, and other data sequence lengths. The proposed methods portray an alternative approach that is capable of auto-label normal and abnormal data and, as a result, dealing with the sequence of label data in the prediction process while avoiding the dynamic behavior of the power consumption data.