Search

Aiming at the problems of sensitivity of initial center, high communication overhead of nodes and low efficiency of cluster in big data clustering algorithm based on partitioning, this paper proposed a partitioning-based clustering algorithm using grid density and locality sensitive hash function based on MapReduce, named PBGDLSH-MR. Firstly, based on the initial data set, it proposed the GDS(grid density strategy) to get the initial clustering center, which avoided the sensitivity of initial center caused by random selection of initial cluster center. Secondly, it proposed the DP-LSH ( data partitioning based on locality sensitive hash functions) to map more closely related data objects into the same subdataset and get data partitions on the map. Meanwhile, it designed a formula SI( similarity improvement) to evaluate the data partitioning results, reduced the communication overhead between nodes. In addition, this paper designed an AGS (adaptive grouping strategy) to handle data skew in data partitions, which improved the cluster efficiency. Finally, based on MapReduce, it mined the cluster centers in parallel to gene rate the final clustering results. The experimental results show that the PBGDLSH-MR has better clustering results and performs better parallelization in big data. [ABSTRACT FROM AUTHOR]

In traditional and relational database, it is difficult to meet the needs of data storage and quick retrieval due to such huge amount of data. For this problem, this paper studied the mapping model of data file field, the relationship between data file field and HBase column, and the importing execution plan, which had solved the heterogeneity of file object and the universality of storage transformation. It put forward the "RowKey" generation rules and algorithm, and provided the algorithm of data transformation based on mapping model and "RowKey". Then according to the mapping rules of data file fields with HBase table column, the data in data file was transformed into HBase. It realized the fast data access and retrieval according to prefix matching in "RowKey" or keyword matching, which had strong commonality and could be widely used in HBase large data storage applications. [ABSTRACT FROM AUTHOR]