Search

In order to explore potential value of explosively growing data in thermal power unit, this paper proposes a big data mining method based on Hadoop-based Spark cluster-computing framework and algorithms. Firstly, positive and negative balance methods are used to accurately obtain actual net coal consumption, and maximum information coefficient method is used to select all parameters related to optimization objectives. Then, Spark-based Mini-Batch K-means algorithm and Elbow method are constructed to divide whole operating modes. After that, all data are discretized and mapped to corresponding intervals by using Spark-based Elbow method and Mini-Batch K-means algorithm. Finally, Spark-based parallel FP-growth algorithm is used to deeply mine the potential relationships and laws. To verify the proposed method, a 350-MW thermal power unit is taken as a study case. The important conclusions are as follows: (1) the proposed Spark-based Mini-Batch K-means algorithm reduces the calculation time by 57.11% compared with Mini-Batch K-means algorithm, and 85.61% calculation time compared with K-means algorithm. The proposed Spark-based FP-growth algorithm reduces computational time by 32.8% compared with FP-growth algorithm. (2) Strong association rules of whole operating modes are mined, and operating optimization guidance schemes for important parameters are obtained. Take operating mode 1 as an example: if the optimal result can be reasonably applied, it can save 2.942 g coal per kilowatt hour. (3) Besides, we have found out some other potential relationships among parameters, which have important reference value for on-site operators to analyze economy of the thermal power unit. [ABSTRACT FROM AUTHOR]

The efficient discovery of frequent itemsets from a transaction database is the fundamental step for association rule mining in data analytics. Interesting associations among the items present in a transaction database contribute to knowledge enrichment. Thus, decision-making and pattern generation from the massive amounts of data become effortless. But one of the major problems associated with the algorithms of frequent itemset mining is excessive memory requirements, which cause them to be inappropriate for larger datasets with itemsets having high cardinality. A few novel data structures for mining frequent itemsets have been introduced in recent years. For example, N-List, NodeSet, DiffNodeSet, proximity list, etc. have been proposed that show a coherent mining approach for improving the execution time while still leaving the scope for further improvements in memory requirements. In this paper, we propose a novel algorithm using cellular learning automata (CLA) and multiple FP tree structures for frequent itemset mining that is efficient in both time and memory requirements. Extensive experimentation has been performed by comparing the performance of the proposed method with the leading algorithms and using publicly available real and synthetic datasets designed specifically for pattern mining algorithms. It can be concluded that the proposed method is memory-efficient and shows comparable execution time with varying dataset dimensions and dataset density, assuring its robustness. In addition to the proposal of the new methodology for frequent itemset mining, its potential domain-specific usage in species biodiversity data analysis has also been discussed. The fact that which groups of species are closely related can be derived from huge occurrence records of species datasets. This could help in understanding species co-occurrence in multiple sites, which in turn assists in solving ecology-related issues for afforesting and reforesting. It could be a step forward toward the advantageous use of computer science in the biodiversity domain. [ABSTRACT FROM AUTHOR]

Technology prediction is an important technique to help new energy vehicle (NEV) firms keep market advantage and sustainable development. Under fierce competition in the new energy industry, there is an urgent necessity for innovative technology prediction method to effectively identify core and frontier technologies for NEV firms. Among the various methods of technology prediction, one of the most frequently used methods is to make technology prediction from patent data. This paper synthesizes the frequent pattern growth (FP-growth) algorithm and input-output analysis to construct a new technology prediction method based on the knowledge flow perspective, takes the data of NEV patent family in 1989–2018 the Derwent patent database as a sample, divides the data according to the 5-year standard, and uses the method to identify the core and frontier technologies in the NEV field during different periods. Furthermore, the multiple co-occurrence method applies to analyze the technology layout and evolution patterns in China's NEV field. The results show that the technology prediction method proposed in this paper can effectively identify core and frontier technologies to achieve NEV technology prediction. [ABSTRACT FROM AUTHOR]

Formal concept analysis (FCA) is an unsupervised machine learning technique used for knowledge discovery and representation. A major task in FCA is the enumeration of the implications to construct the implication base. Even though there are many efficient classical and parallel algorithms proposed for constructing the implication base, the existing algorithms are not well suited for large formal contexts because of their architectural complexity. All the existing works use either stem-base or proper-premise approach to find the implication base in exponential time. Hence, we introduce a distributed algorithm to find the implication base quickly in larger datasets in polynomial time. In this paper, we propose a scalable algorithm to find the implication base using machine learning technique FP-growth, big data processing framework Apache Spark and executed on large formal contexts. Extensive experiments on the real-world datasets show that the proposed algorithm has an improved gain in performance metrics such as execution time, CPU and memory usage. The statistical validations on the experimental results prove that the proposed algorithm has the better potential to find the implication base. [ABSTRACT FROM AUTHOR]

Data mining is an important technology to reveal the patterns from crime data. Although there are many researches about this topic, less work models the relations between rates of different kinds of crime. In this paper, an algorithm based on fuzzy association rules (AR) mining is proposed to discover these relations. Two datasets, which are crimes in Chicago from 2012 to 2017 and crimes in NSW from 2008 to 2012, are used for case studies. At first, crime data is preprocessed, where every kind of crime occurring in every district during every month is counted. For a crime in a combination of district and month, the membership function, which is based on hypothesis testing, is designed to evaluate the degree to which its rate is high, normal or low, and the fuzzy transactional dataset is formed. A bridge between fuzzy transactional dataset and binary AR mining algorithm is built, so those mature tools of binary AR mining can be applied to generate fuzzy ARs. In the results of case studies, the strong relations between rates of different crime can be found. There are many interesting and surprise rules, which are worthy to be further studied by domain experts. [ABSTRACT FROM AUTHOR]

In order to recommend an efficient drawing inspecting expert combination, an expert combination is selected by an expert recommendation algorithm based on Pearson's correlation coefficient and FP-growth. By introducing the Pearson correlation coefficient and the FP-growth association rule algorithm, the expert recommendation algorithm can accurately select the participating experts in the historical project similar to the scale of the project to be reviewed, and combine the experts to calculate and obtain the expert group with the highest fit, namely, the expert combination of project to be reviewed. This expert recommendation algorithm based on Pearson correlation coefficient and FP-growth can effectively recommend a kind of expert group with the highest efficiency of collaborative review, which solves the problem of how to recommend efficient expert combination accurately for drawing inspecting system. [ABSTRACT FROM AUTHOR]

According to the complexity of ocean data, this paper adopts a parallel mining algorithm of association rules to explore the correlation and regularity of oxygen, temperature, phosphate, nitrate and silicate in the ocean. After the marine data is interpolated, this paper utilizes the parallel FP-growth algorithm to mine the data and then briefly analyzes the mining results of the frequent itemsets and association rules. The relationship between the parallel efficiency and the core number of CPU is analyzed through datasets with different scales. The experimental results indicate that the acceleration effect is ideal when each thread scored 200,000-300,000 data, which leads to more than 1.2 times of performance improvement. [ABSTRACT FROM AUTHOR]

With the booming of social media and health informatics, there exists a pressing need for a powerful tool to sustain comprehensive analysis of public and personal health information. In particular, it should be able (1) to maximize the discovery of association rules amongst data items and (2) to handle the rapid growing data scale. The FP-Growth algorithm is a salient association rule learning method in exploring potential relation in database possibly with a lack of priori knowledge. It has the merits of low time & space complexity, whereas it cannot handle negative association rules which is necessary in comprehensive mining of health data. In order to enable comprehensive discovery of association rules, this study extends the FP-Growth algorithm to mine both positive and negative frequent patterns, namely the PNFP-Growth framework. The extended approach also adopts a prune strategy to filter out misleading patterns to the most by correlating the negative data items and the positive ones. Experiments had been performed to evaluate the performance of the PNFP-Growth over a public data set and a database consisting of thousands of people's real health examination information (collected within 5 years from the date of this publication). The results indicate that (1) the PNFP-Growth can excavate more patterns than the traditional counterpart does while it is still highly efficient, and (2) the analysis upon the health examination data is informative and well complies with the clinical practices, e.g., more than 30 % people suffering from hypertension are having high systolic pressure and liver problems. [ABSTRACT FROM AUTHOR]

In this paper, we present a sparse memory allocation data structure for sequential and parallel data mining. We explored three algorithms utilizing the proposed data structure: MASP-tree, apriori-TID, and FP-growth. We modified the data structure of apriori-TID and FP-growth algorithms to reduce memory allocation cost. Five data sets are used for comparison. The results show that the modified apriori-TID has a higher speed-up than the modified FP-growth when the proposed data structure is used. A maximum speed-up of 3.42 is observed when MASP algorithm is tested. [ABSTRACT FROM AUTHOR]

In the present scenario of global economy and World Wide Web, large sets of evolving and distributed data can be handled efficiently by incremental data mining. Frequent patterns are very important in knowledge discovery and data mining process, such as mining of association rules, correlations. FP-tree is a very versatile data structure used for mining of frequent patterns in knowledge discovery and data mining process. FP-tree is a compact representation of transaction database that contains frequency information of all relevant frequent patterns (FP) of the database. All of the existing incremental frequent pattern mining algorithms, such as AFPIM, CATS, CanTree, CP-tree, and SPO-tree, perform incremental mining by processing one transaction of the incremental part of database at a time and updating it to the FP-tree of initial (original) database. Here, in this paper, we propose a novel method that takes advantage of FP-tree representation of incremental transaction database for incremental mining. We propose a batch incremental processing algorithm BIT_FPGrowth that restructures and merges two small consecutive duration FP-trees to obtain a FP-tree of the FP-Growth algorithm. Our BIT_FPGrowth uses FP-tree as preprocessed data repository to get transactions (i.e., item-sets), unlike other sequential incremental algorithms that read transactions from database. BIT_FPGrowth algorithm takes less time for constructing FP-tree. Our experimental results show that, as the size of the database increases, increase in runtime of BIT_FPGrowth is much less and is least of all the other algorithms. [ABSTRACT FROM AUTHOR]

In this paper, a self-organization mining based hybrid evolution (SOME) learning algorithm for designing a TSK-type fuzzy model (TFM) is proposed. In the proposed SOME, group-based symbiotic evolution (GSE) is adopted in which each group in the GSE represents a collection of only one fuzzy rule. The proposed SOME consists of structure learning and parameter learning. In structure learning, the proposed SOME uses a two-step self-organization algorithm to decide the suitable number of rules in a TFM. In parameter learning, the proposed SOME uses the data mining based selection strategy and data mining based crossover strategy to decide groups and parental groups by the data mining algorithm that called frequent pattern growth. Illustrative examples were conducted to verify the performance and applicability of the proposed SOME method. [ABSTRACT FROM AUTHOR]

Recognizing attack intention is crucial for security analysis. In recent years, a number of methods for attack intention recognition have been proposed. However, most of these techniques mainly focus on the alerts of an intrusion detection system and use algorithms of low efficiency that mine frequent attack patterns without reconstructing attack paths. In this paper, a novel and effective method is proposed, which integrates several techniques to identify attack intentions. Using this method, a Bayesian-based attack scenario is constructed, where frequent attack patterns are identified using an efficient data-mining algorithm based on frequent patterns. Subsequently, attack paths are rebuilt by recorrelating frequent attack patterns mined in the scenario. The experimental results demonstrate the capability of our method in rebuilding attack paths, recognizing attack intentions as well as in saving system resources. Specifically, to the best of our knowledge, the proposed method is the first to correlate complementary intrusion evidence with frequent pattern mining techniques based on the FP-Growth algorithm to rebuild attack paths and to recognize attack intentions. [ABSTRACT FROM AUTHOR]

Generating captions or annotations automatically for still images is a challenging task. Traditionally, techniques involving higher-level (semantic) object detection and complex feature extraction have been employed for scene understanding. On the basis of this understanding, corresponding text descriptions are generated for a given image. In this paper, we pose the auto-annotation problem as that of multi-relational association rule mining where the relations exist between image-based features, and textual annotations. The central idea is to combine low-level image features such as color, orientation, intensity, etc. and corresponding text annotations to generate association rules across multiple tables using multi-relational association mining. Subsequently, we use these association rules to auto-annotate test images. In this paper we also present a multi-relational extension to the FP-tree algorithm to accomplish the association rule mining task effectively. The motivation for using multi-relational association rule mining for multimedia data mining is to exhibit the potential accorded by multiple descriptions for the same image (such as multiple people labeling the same image differently). Moreover, multi-relational association rule mining can also benefit the auto-annotation process by pruning the number of trivial associations that are generated if text and image features were combined in a single table through a join. In this paper, we discuss these issues and the results of our auto-annotation experiments on different test sets. Another contribution of this paper is highlighting a need to develop robust evaluation metrics for the image annotation task. We propose several applicable scoring techniques and then evaluate the performance of the different algorithms to study the utility of these techniques. A detailed analysis of the datasets used and the performance results are presented to conclude the paper. [ABSTRACT FROM AUTHOR]