Search

One of the main steps in the data lifecycle is to publish it for data analysts to discover hidden patterns. But, data publishing may lead to unwanted disclosure of personal information and cause privacy problems. Data anonymization techniques preserve privacy models to prevent the disclosure of individuals’ private information in published data. In this paper, a distributed in-memory method is proposed on the Apache Spark framework to preserve the l-diversity privacy model. This method anonymizes large-scale data in a three-phase process, which includes, seed selection, data clustering for $$\ell$$ -diversity, and finalizing phase. In this method, a hierarchical kmeans-based data clustering algorithm has been designed for data anonymization. One of the major challenges of anonymization methods is to establish a better trade-off between data utility and privacy. Therefore, for calculating the distance between records and forming more cohesive ldiverse-clusters, the authors have designed two Manhattan-based and Euclidean-based distance functions to satisfy the requirements of the l-diversity model. Given the 100-fold speed of the Spark compared to MapReduce, the proposed method is presented using in-memory RDD programming in Apache Spark, to address the runtime, scalability, and performance in large-scale data anonymization as it exists in the previous MapReduce-based algorithms. Our method provides general knowledge to use parallel in-memory computation of Spark in big data anonymization. In experiments, this method has obtained lower information loss and loses about 1% to 2% accuracy and FMeasure criteria; therefore, it establishes a better trade-off than the state-of-the-art MapReduce-based Mondrian methods

For the extraction of useful patterns, the collected data should be distributed to and shared with analyzers. This, however, creates problems and challenges for the individual with respect to their privacy and identity. In this paper, the Mondrian multidimensional anonymization method was developed and improved for satisfaction of the l-diversity privacy model, and it has been presented in a distributed fashion within the Apache Spark framework. Since one of the major challenges in data privacy is the tradeoff between privacy and data utility, the presented method focuses on information loss and classifier evaluation criteria. Therefore, the cut dimension was selected using the coefficient of variation and information gain criteria, and the cut points were chosen dynamically, which led to a decrease in the information loss parameter and an improvement in the classifier performance evaluation criteria such as accuracy and FMeasure compared to the previous algorithms in the literature. The processing speed is 100 times higher in Spark than in the Hadoop framework. Consequently, the proposed method was presented in a distributed fashion based on RDDs programming within Apache Spark framework. This will resolve the problem of speed in large-scale data anonymization as it exists in the previous Hadoop-based algorithms. The results of the experiments performed on the numerical datasets demonstrate the improvements made by the proposed method.