Search

Recognizing anomalies is an extremely important process in data analysis, aimed at identifying patterns in data that deviate from known norms or typical standards. These anomalies are often indicative of significant, and sometimes critical, issues such as fraud, network intrusions, and system failures. Traditional anomaly detection algorithms primarily focus on the attributes of individual observations within a dataset, typically establishing a ‘normal’ profile and flagging deviations from this profile as anomalies. This paper introduces an innovative enhancement to the Isolation Forest algorithm, a renowned method for anomaly detection known for its effectiveness and efficiency, especially in large datasets. The Isolation Forest algorithm operates by randomly partitioning the data space and constructing a binary tree, where the oddity score of a data point is ascertained based on its separation from the extremity to the base of the structure, enabling the autonomous detection of outliers in a completely unsupervised manner. The methodology presented in the paper is based on repeatedly building Isolation Forest models on datasets from which individual attributes are excluded. In our research, we used the SHAP (SHapley Additive exPlanations) method which comes from game theory and is used to determine the impact of individual features on the result of the model. When training the Isolation Forest on the full dataset, the SHAP method is used to obtain the coefficients of influence of model attributes on the prediction result. Both negative and positive influences are considered significant when counting the anomaly score. On the foundations of the results from all sub-models, a weighted average is calculated, to which weights are calculated based on the SHAP model. The comparative analysis of evaluation metrics revealed a substantial enhancement attributed to the implemented methodology. The metrics used for evaluation have shown improvement in most cases from 3.5 to 6 percent point. One of the metrics have shown an improvement of 12 percent. Obtained results demonstrate that this integrated approach not only enhances the prediction accuracy of the Isolation Forest algorithm but also offers a more interpretable understanding of the data. This advancement in anomaly detection methodology promises significant implications for various fields where quick and accurate detection of outliers is paramount.

The purpose of the theoretical considerations and research conducted was to indicate the instruments with which the quality of a dataset can be verified for the segmentation of observations occurring in the dataset. The paper proposes a novel way to deal with mixed datasets containing categorical and continuous attributes in a customer segmentation task. The categorical variables were embedded using an innovative unsupervised model based on an autoencoder. The customers were then divided into groups using different clustering algorithms, based on similarity matrices. In addition to the classic k-means method and the more modern DBSCAN, three graph algorithms were used: the Louvain algorithm, the greedy algorithm and the label propagation algorithm. The research was conducted on two datasets: one containing on retail customers and the other containing wholesale customers. The Calinski–Harabasz index, Davies–Bouldins index, NMI index, Fowlkes–Mallows index and silhouette score were used to assess the quality of the clustering. It was noted that the modularity parameter for graph methods was a good indicator of whether a given set could be meaningfully divided into groups.