Search

It presents an improved FP-Growth frequent itemset mining algorithm for big data based on Hadoop framework and MapReduce programming model.Firstly, the transaction database is extracted according to each frequent 1-items to generate corresponding projection databases which are distributed to several node machines.Then each node machine divides the projection database into several smaller sub-databases, and uses the improved algorithm to mine partial frequent itemsets in parallel.Finally, all the frequent itemsets are obtained by merging all the partial frequent itemsets.This algorithm does not need to generate a huge FP tree for transaction database as the traditional FP-Growth algorithm and some other improved algorithm, so effectively solves the problem of FP-Growth algorithm and other improved algorithm failure because of insufficient memory for storage of huge FP tree in a single machine.At the same time, because the size of these sub-databases are close to each other, the load distributed to each node is more balanced, which makes the algorithm more efficient. [ABSTRACT FROM AUTHOR]

Aiming at the problems of sensitivity of initial center, high communication overhead of nodes and low efficiency of cluster in big data clustering algorithm based on partitioning, this paper proposed a partitioning-based clustering algorithm using grid density and locality sensitive hash function based on MapReduce, named PBGDLSH-MR. Firstly, based on the initial data set, it proposed the GDS(grid density strategy) to get the initial clustering center, which avoided the sensitivity of initial center caused by random selection of initial cluster center. Secondly, it proposed the DP-LSH ( data partitioning based on locality sensitive hash functions) to map more closely related data objects into the same subdataset and get data partitions on the map. Meanwhile, it designed a formula SI( similarity improvement) to evaluate the data partitioning results, reduced the communication overhead between nodes. In addition, this paper designed an AGS (adaptive grouping strategy) to handle data skew in data partitions, which improved the cluster efficiency. Finally, based on MapReduce, it mined the cluster centers in parallel to gene rate the final clustering results. The experimental results show that the PBGDLSH-MR has better clustering results and performs better parallelization in big data. [ABSTRACT FROM AUTHOR]

Aiming at the problem of low accuracy of traditional prediction methods of anti-electricity stealing, a prediction method of anti-electricity stealing based on big data of electric power was proposed. This method firstly constructed samples of electricity stealing data according to abnormal rules, and introduced the growth rate of line loss rate constraint conditions. Then it used four machine learning classification algorithms to build a prediction model on the voltage, current and power factor data sets respectively, combined the users with abnormal data output and users with abnormal line loss to output a list of users suspected of stealing electricity. Experimental results showed that the prediction accuracy of the method was satisfactory, and it was efficient and feasible in identifying users suspected of stealing electricity. [ABSTRACT FROM AUTHOR]

Most recommendation systems based on the network structure suffer from lack of diversity, so that the paper proposed a recommendation system of big data based on multi-weight projection of bipartite network. Firstly, it abstracted the basic information of datasets, and applied items-users lists as an input to Levenshtein distance algorithm to compute similarity of each property. Then, it computed the number of common neighbors of the nodes in the bipartite network, the degree of common neighbors of the nodes in the bipartite network and degree of each node in bipartite network, and also computed the triple weights of each side of the bipartite network. Lastly, it adopted the enhanced bipartite projection technique to abstract the potential links of the bipartite network to realize the link prediction based on similarity. It realized the experiments based on both of big dataset and small dataset. The results show that the proposed algorithm outperforms different kinds of recommendation systems in terms of accuracy and coverage of recommendation, at the same time, it outperforms the other recommendation system based on network structure. [ABSTRACT FROM AUTHOR]

Aiming at solving the problem of big data's large scale and complex computation, this paper adopted the idea of two-stage progressive clustering, and proposed a parallel computation algorithm for big data clustering based on MapReduce. In the first stage, this method acquired the initialized clustering center through Canopy algorithm, in order to find relatively accurate cluster center points quickly. In the second stage, it presented a novel scheme of parallel computation based on Map Reduce framework, which maked each data node cluster or merge around its adjacent Canopy center node. In this way, the algorithm could make the procedure of data clustering fast and accurately. The results of the experiments deployed on MapReduce show that this algorithm can effectively improve the efficiency of parallel computing, reduce computing time, and improve big data's clustering accuracy. [ABSTRACT FROM AUTHOR]

In traditional and relational database, it is difficult to meet the needs of data storage and quick retrieval due to such huge amount of data. For this problem, this paper studied the mapping model of data file field, the relationship between data file field and HBase column, and the importing execution plan, which had solved the heterogeneity of file object and the universality of storage transformation. It put forward the "RowKey" generation rules and algorithm, and provided the algorithm of data transformation based on mapping model and "RowKey". Then according to the mapping rules of data file fields with HBase table column, the data in data file was transformed into HBase. It realized the fast data access and retrieval according to prefix matching in "RowKey" or keyword matching, which had strong commonality and could be widely used in HBase large data storage applications. [ABSTRACT FROM AUTHOR]

In order to solve the low efficiency and practical application of predicting the potential phone replacement user, this paper designed and built a phone replacement prediction system based on big data platform. This system firstly captured signaling data from multiple network interface and collected external data. Through the parse platform, the data from network interface would be distributed, decoded, synthesized and correlated, and the external data would be processed by ETL tools, and then storing processed data into IIDFS. Further, this paper established a phone replacement prediction model, which based on logistic regression, using spark components in the big data platform and output the potential phone replacement users. Finally, this paper chose part of the western city ’ s user data for system testing. The result shows that the prediction accuracy of the phone replacement prediction system is up to 71%. It can preferably recognize potential phone replacement users, and provide reliable support for the precise marketing of operators and mobile phone manufacturers. [ABSTRACT FROM AUTHOR]

In order to meet the needs of the network under the background of big data, and eliminate inferior data interference data knowledge high precision requirements of large data transmission, this paper proposed variable size adjustment scheme based on the algorithm to expand the network of large data knowledge opportunistic scheduling. Based on the analysis of large data network characteristics, it normalized the adaptive vector encoding, capture the heterogeneous characteristics of large data network, using multi order back-propagation network of heterogeneous data, and then through the real-time transmission of large data network to achieve opportunistic scheduling. At the same time, the knowledge engineering system composed of network data segmentation of fine-grained big data based on the multidimensional feature dimension, the granularity of knowledge transformation was known, then adjusted the size of the dynamic characteristics, making big data set of knowledge engineering with linear characteristics and clear geometric characteristics, improved the accuracy of knowledge acquisition through knowledge expansion. The experimental results are compared with the algorithm based on fine grained knowledge acquisition, which proves the high reliability, real time and high efficiency of network data transmission. [ABSTRACT FROM AUTHOR]

Soochow University has already completed an experimental material procurement management platform. This paper made an overall analysis based on the big data generated by the platform. Using the data to explore user' s behavior, a thinning comparison of different types and the production time of orders on the platform was completed. The purpose of this paper was to emphasize the value of the platform for scientific research support services, to promote the comprehensive optimization and maintenance of the platform, to earnestly implement the state's opinions on implementing the policy of increasing the distribution of knowledge and value. The platform has speeded up the implementation of innovation driven development strategy, stimulated the enthusiasm of scientific research personnel to innovate and start a business. In the whole society, we should create an atmosphere of respecting labour, respecting knowledge, respecting talents and respecting creation. [ABSTRACT FROM AUTHOR]