Search

With the rapid development of AI and big data mining technologies, computerized medical decision-making has become increasingly prominent. The aim of high-utility pattern mining (HUPM) is to discover meaningful patterns in medical databases that contribute to maximizing the utility from the perspective of diagnosis. However, HUPM pays less attention to the interpretability and explainability of these patterns in medical decision-making scenarios. This paper proposes a novel algorithm called the Improved fuzzy high-utility pattern mining (IF-HUPM) to address this problem. First, the paper applies a fuzzy preprocessing method to divide the fuzzy intervals of a medical quantitative data set, which enhances the fuzziness and interpretability of the data. Next, in the process of IF-HUPM, both fuzzy tree and list structures are employed to calculate fuzzy high-utility values. By combining the characteristics of the one-stage and two-stage algorithms of HUPM, an adaptive-phase Fuzzy HUPM hybrid frame is proposed. The experimental results demonstrate that the proposed IF-HUPM algorithm enhances both accuracy and efficiency and the mining process requires less time and space on average. [ABSTRACT FROM AUTHOR]

The location planning of relief distribution centres (DCs) is crucial in humanitarian logistics as it directly influences the disaster response and service to the affected victims. In light of the critical role of facility location in humanitarian logistics planning, the study proposes a two-stage relief distribution location problem. The first stage of the model determines the minimum number of relief DCs, and the second stage find the optimal location of these DCs to minimize the total cost. To address a more realistic situation, restrictions are imposed on the coverage area and capacity of each DCs. In addition, for optimally solving this complex NP-hard problem, a novel two-phase algorithm with exploration and exploitation phase is developed in the paper. The first phase of the algorithm i.e., exploration phase identifies a near-optimal solution while the second phase i.e. exploitation phase enhances the solution quality through a close circular proximity investigation. Furthermore, the comparative analysis of the proposed algorithm with other well-known algorithms such as genetic algorithm, pattern search, fmincon, multistart and hybrid heuristics is also reported and computationally tested from small to large data sets. The results reveal that the proposed two-phase algorithm is more efficient and effective when compared to the conventional metaheuristic methods. [ABSTRACT FROM AUTHOR]

Collecting and analyzing data from all devices to improve the efficiency of business processes is an important task of Industrial Internet of Things (IIoT). In the age of data explosion, extensive text data generated by the IIoT have given birth to a variety of text representation methods. The task of text representation is to convert the natural language to a form that computer can understand with retaining the original semantics. However, these methods are difficult to effectively extract the semantic features among words and distinguish polysemy in natural language. Combining the advantages of convolutional neural network (CNN) and variational autoencoder (VAE), this paper proposes an intelligent CNN-VAE text representation algorithm as an advanced learning method for social big data within next-generation IIoT, which help users identify the information collected by sensors and perform further processing. This method employs the convolution layer to capture the local features of the context and uses the variational technique to reconstruct feature space to make it conform to the normal distribution. In addition, the improved word2vec model based on topical word embedding (TWE) is utilized to add topical information to word vectors to distinguish polysemy. This paper takes the social big data as an example to illustrate the way of the proposed algorithm applied in the next-generation IIoT and utilizes Cnews dataset to verify the performance of proposed method with four evaluating metrics (i.e., recall, accuracy, precision, and F1-score). Experimental results indicate that the proposed method outperforms word2vec-avg and CNN-AE in K-nearest neighbor (KNN), random forest (RF), and support vector machine (SVM) classifiers and distinguishes polysemy effectively. [ABSTRACT FROM AUTHOR]

In order to explore potential value of explosively growing data in thermal power unit, this paper proposes a big data mining method based on Hadoop-based Spark cluster-computing framework and algorithms. Firstly, positive and negative balance methods are used to accurately obtain actual net coal consumption, and maximum information coefficient method is used to select all parameters related to optimization objectives. Then, Spark-based Mini-Batch K-means algorithm and Elbow method are constructed to divide whole operating modes. After that, all data are discretized and mapped to corresponding intervals by using Spark-based Elbow method and Mini-Batch K-means algorithm. Finally, Spark-based parallel FP-growth algorithm is used to deeply mine the potential relationships and laws. To verify the proposed method, a 350-MW thermal power unit is taken as a study case. The important conclusions are as follows: (1) the proposed Spark-based Mini-Batch K-means algorithm reduces the calculation time by 57.11% compared with Mini-Batch K-means algorithm, and 85.61% calculation time compared with K-means algorithm. The proposed Spark-based FP-growth algorithm reduces computational time by 32.8% compared with FP-growth algorithm. (2) Strong association rules of whole operating modes are mined, and operating optimization guidance schemes for important parameters are obtained. Take operating mode 1 as an example: if the optimal result can be reasonably applied, it can save 2.942 g coal per kilowatt hour. (3) Besides, we have found out some other potential relationships among parameters, which have important reference value for on-site operators to analyze economy of the thermal power unit. [ABSTRACT FROM AUTHOR]

In the big data era, traditional data mining technology cannot meet the requirements of massive data processing with the background of intelligent manufacturing. Aiming at insufficient computing power and low efficiency in mining process, this paper proposes a improved K-means clustering algorithm based on the concept of distributed clustering in cloud computing environment. The improved algorithm (T.K-means) is combined with MapReduce computing framework of Hadoop platform to realize parallel computing, so as to perform processing tasks of massive data. In order to verify the practical performance of T.K-means algorithm, taking machining data of milling Ti-6Al-4V alloy as the mining object. The mapping relationship among cutting parameters, surface roughness, and material removal rate is mined, and the optimized value for cutting parameters is obtained. The results show that T.K-means algorithm can be used to mine the optimal cutting parameters, so that the best surface roughness can be obtained in milling Ti-6Al-4V titanium alloy. [ABSTRACT FROM AUTHOR]

The growing data have brought tremendous pressure for query processing and storage, so there are many studies that focus on using GPU to accelerate join operation, which is one of the most important operations in modern database systems. However, existing GPU acceleration join operation researches are not very suitable for the join operation on big data. Based on this, this paper speeds up nested loop join, hash join and theta join, combining Hadoop with GPU, which is also the first to use GPU to accelerate theta join. At the same time, after the data pre-filtering and pre-processing, using MapReduce and HDFS in Hadoop proposed in this paper, the larger data table can be handled, compared to existing GPU acceleration methods. Also with MapReduce in Hadoop, the algorithm proposed in this paper can estimate the number of results more accurately and allocate the appropriate storage space without unnecessary costs, making it more efficient. Experimental results show that comparing with GPU-based approach without Hadoop, our approach increases the speed by 1.5–2 times, and comparing with the Hadoop-based approaches without GPU, our approach increases the speed by 1.3–2 times. [ABSTRACT FROM AUTHOR]

This paper deals with the current Blockchain adopted in the various types of trading markets supporting smart contracts that process text-based transaction information only when exchanging cryptocurrencies. Even though the Blockchain itself is decentralized, services can still be provided through a centralized system in order to provide adequate services to the users. It also allows the Autochain to apply Blockchain technology to existing businesses conducted under the Fourth Industrial Revolution to realize higher productivity or competitiveness and improve profitability to a remarkable extent. The Autochain is adaptable to all Blockchain technologies, companies, or research works around the world to overcome the limitations of the existing Blockchain technology. It creates the Blockchain-platform ecology chain and ecosystem, making our lives more convenient while revolutionizing every industry. In addition, it automates all Blockchain-based systems and offers a base for constructing a system wherein everyone can easily provide, operate, and maintain Blockchain services. This paper discusses a number of Blockchain-related technologies and presents a decentralized Application (dApp) exclusively designed for mobile use, taking house rental service as an example. It also presents the existing Blockchain and the Blockchain database storage method and proposes the dApp based on it. [ABSTRACT FROM AUTHOR]

Big data analytics in cloud environments introduces challenges such as real-time load balancing besides security, privacy, and energy efficiency. This paper proposes a novel load balancing algorithm in cloud environments that performs resource allocation and task scheduling efficiently. The proposed load balancer reduces the execution response time in big data applications performed on clouds. Scheduling, in general, is an NP-hard problem. Our proposed algorithm provides solutions to reduce the search area that leads to reduced complexity of the load balancing. We recommend two mathematical optimization models to perform dynamic resource allocation to virtual machines and task scheduling. The provided solution is based on the hill-climbing algorithm to minimize response time. We evaluate the performance of proposed algorithms in terms of response time, turnaround time, throughput metrics, and request distribution with some of the existing algorithms that show significant improvements. [ABSTRACT FROM AUTHOR]

Recent trends in big data have shown that the amount of data continues to increase at an exponential rate. This trend has inspired many researchers over the past few years to explore new research direction of studies related to multiple areas of big data. The widespread popularity of big data processing platforms using MapReduce framework is the growing demand to further optimize their performance for various purposes. In particular, enhancing resources and jobs scheduling are becoming critical since they fundamentally determine whether the applications can achieve the performance goals in different use cases. Scheduling plays an important role in big data, mainly in reducing the execution time and cost of processing. This paper aims to survey the research undertaken in the field of scheduling in big data platforms. Moreover, this paper analyzed scheduling in MapReduce on two aspects: taxonomy and performance evaluation. The research progress in MapReduce scheduling algorithms is also discussed. The limitations of existing MapReduce scheduling algorithms and exploit future research opportunities are pointed out in the paper for easy identification by researchers. Our study can serve as the benchmark to expert researchers for proposing a novel MapReduce scheduling algorithm. However, for novice researchers, the study can be used as a starting point. [ABSTRACT FROM AUTHOR]

Algorithmic vision, the computational process of making meaning from digital images or visual information, has changed the relationship between the image and the human subject. In this paper, I explicate on the role of algorithmic vision as a technique of algorithmic governance, the organization of a population by algorithmic means. With its roots in the United States post-war cybernetic sciences, the ontological status of the computational image undergoes a shift, giving way to the hegemonic use of automated facial recognition technologies towards predatory policing and profiling practices. By way of example, I argue that algorithmic vision reconfigures the philosophical links between vision, image, and truth, paradigmatically changing the way a human subject is represented through imagistic data. With algorithmic vision, the relationship between subject and representation challenges the humanistic discourse around images, calling for a critical displacement of the human subject from the center of an analysis of how computational images make meaning. I will explore the relationship between the operative image, the image that acts but is not seen by human eyes, and what Louise Amoore calls an "emergent subject," a subject that is made visible through algorithmic techniques (2013). Algorithmic vision reveals subjects to power in a mode that requires a new approach towards analyzing the entanglement and invisiblization of the human in automated decision-making systems. [ABSTRACT FROM AUTHOR]

With the development of big data technologies, green cloud data centers have become a key factor in academia and industry. An energy-efficient cloud data center can save costs for cloud computing users. However, the problem of virtual machine mapping has always been a core problem. In the most existing research, the energy consumption generated by cloud data centers has become an important bottleneck restricting the technology of cloud computing. This paper establishes a consumption model of cloud data center energy and a virtual machine mapping rule. Based on this, a multi-dimensional double descending maximum padding priority (MD3MP2) virtual machine mapping algorithm is proposed. The algorithm can not only solve the one-dimensional virtual machine mapping problem of homogeneous data centers, but also successfully solve the multi-dimensional virtual machine mapping problem of homogeneous data centers. Finally, the algorithm is compared with four other algorithms. The experimental results show that the MD3MP2 algorithm is better than the compared algorithms. [ABSTRACT FROM AUTHOR]

Big data (BD) is attaining major attention in the information field due to the eruption of data in the preceding decade. Philosophical techniques of "query optimization (QO)" have an essential function in data retrieval as of a BD environment. Numerous cloud-centered distributed data processing platforms were developed to render effective as well as lucrative solutions for BD query optimization. Nevertheless, most techniques brought about higher "energy consumptions (EC)" along with low accuracy level because of the lack of deliberation of energy issues as well as query characteristics, correspondingly. To tackle the issues of query optimization process, this paper proposes an "efficient query optimization (EQO)" utilizing σ ANFIS load balancer in addition to the CaM-BW optimizer. The proposed technique comprises '2′ phases: (1) BD arrangement and (2) QO. In the initial phase, the BD is arranged by utilizing preprocessing, feature extraction, together with clustering. The MCoV-FCM algorithm takes care of the clustering. In the second phase, the σ ANFIS load balancer in addition to the CaM-BW optimizer concentrates on disregarding the energy-efficient query plans and enhancing the general query processing performance. Lastly, numerical simulation outcomes are rendered to display the proposed method's effectiveness. [ABSTRACT FROM AUTHOR]

Osmotic Computing represents a glue solution able to manage the deployment and orchestration of interconnected microelements across heterogeneous physical and virtual infrastructures (e.g., IoT, Edge and Cloud nodes) according to the behavior of hardware and software components during the time. The adoption of Osmotic Computing is challenging, but addressing networking issues is a key research topic due to the emergence of new problems in terms of QoS requirements. In this paper, we analyze how to exploit well-known networking solutions, such as the Dijkstra's algorithm, and Big Data oriented technologies, such as the Hadoop and MapReduce, to provide efficient newtorking functionalities in Osmotic Computing. In particular, our objective is to minimize the routing path computation time in the software defined network (SDN) at the basis of microelement networking, as well as to ensure a global view and a high level of dynamism of our network topology. To accomplish this task, we process routing tables through a MapReduce based implementation of the Dijkstra's algorithm whenever a topology change occurs, and we export routing results into the SDN. Our experimental results show that our networking strategy drastically reduces the best path computation time whenever the network of microelements is very large. [ABSTRACT FROM AUTHOR]

In this paper, we discuss a family of robust, high-dimensional regression models for quantile and composite quantile regression, both with and without an adaptive lasso penalty for variable selection. We reformulate these quantile regression problems and obtain estimators by applying the alternating direction method of multipliers (ADMM), majorize-minimization (MM), and coordinate descent (CD) algorithms. Our new approaches address the lack of publicly available methods for (composite) quantile regression, especially for high-dimensional data, both with and without regularization. Through simulation studies, we demonstrate the need for different algorithms applicable to a variety of data settings, which we implement in the cqrReg package for R. For comparison, we also introduce the widely used interior point (IP) formulation and test our methods against the IP algorithms in the existing quantreg package. Our simulation studies show that each of our methods, particularly MM and CD, excel in different settings such as with large or high-dimensional data sets, respectively, and outperform the methods currently implemented in quantreg. The ADMM approach offers specific promise for future developments in its amenability to parallelization and scalability. [ABSTRACT FROM AUTHOR]

The rise of high-resolution and high-throughput sequencing technologies has driven the emergence of such new fields of application as precision medicine. However, this has also led to an increase in the storage and processing requirements for the bioinformatics tools, which can only be provided by high-performance and massive data processing infrastructures. Such technologies allow the development of scalable, efficient and reliable bioinformatics tools. In this paper, a new implementation of the Basic Local Alignment Search Tool algorithm is presented. Our proposal, named Sparky-Blast, utilizes Cassandra database to store the different reference datasets and the Apache Spark processing framework to calculate the indexes and process the queries. This successful approach avoids the bottleneck that suffers the original BLAST version that is limited to the resources of a single machine. Sparky-Blast is capable of using the distributed resources of a Big-Data Cluster to process queries in parallel, thus, improving both the response time and the system throughput. At the same time, the use of a distributed architecture like Hadoop provides unlimited scalability from the point of view of both the hardware infrastructure and performance. [ABSTRACT FROM AUTHOR]

Single Linkage algorithm is a hierarchical clustering method which is most unsuitable for large dataset because of its high convergence time. The paper proposes an efficient accelerated technique for the algorithm for clustering univariate data with a merging threshold. It is a two-stage algorithm with the first one as an incremental pre-clustering step that uses the farthest neighbour principle to partially cluster the database by scanning it only once. The algorithm uses the Segment Addition Postulate as a major tool for accelerating the pre-clustering stage. The incremental approach makes it suitable for partial clustering of streaming data while collecting it. The Second stage merges these pre-clusters to produce the final set of Single Linkage clusters by comparing the biggest and the smallest data of each pre-cluster and thereby converging faster in comparison to those methods where all the members of the clusters are used for a clustering action. The algorithm is also suitable for fast-changing dynamic databases as it can cluster a newly added data without using all the data of the database. Experiments are conducted with various datasets and the result confirms that the proposed algorithm outperforms its well-known variants. [ABSTRACT FROM AUTHOR]

Due to the huge amount of data being generating from different sources, the analyzing and extracting of useful information from these data becomes a very complex task. The difficulty of dealing with big data optimization problems comes from many factors such as the high number of features, and the existing of lost data. The feature selection process becomes an important step in many data mining and machine learning algorithms to reduce the dimensionality of the optimization problems and increase the performance of the classification or clustering algorithms. In this paper, a set of hybrid and efficient genetic algorithms are proposed to solve feature selection problem, when the handled data has a large feature size. The proposed algorithms use a new gene-weighted mechanism that can adaptively classify the features into strong relative features, weak or redundant features, and unstable features during the evolution of the algorithm. Based on this classification, the proposed algorithm gives the strong features high priority and the weak features less priority when generating new candidate solutions. In the same time, the proposed algorithm tries to more concentrate on unstable features that sometimes appear and sometimes disappear from the best solutions of the population. The performance of proposed algorithms is investigated by using different datasets and feature selection algorithms. The results show that our proposed algorithms can outperform the other feature selection algorithms and effectively enhance the classification performance over the tested datasets. [ABSTRACT FROM AUTHOR]

Tremendous amounts of data are generated by sensors and connected devices with high velocity in a variety of forms and large volumes. These characteristics, defined as big data, need new models and methods to be processed in near real-time. The nature of decentralized large-scale data sources requires distributed algorithms in which it is assumed that the data sources are capable of processing their own data and collaborating with neighbor sources. The network objective is to make an optimal decision, while the data are processed in a distributed manner. New technologies, like next generation of wireless communication and 5G, introduce practical issues such as imperfect communication that should be addressed. In this paper, we study a generalized form of distributed algorithms for decision-making over decentralized data sources. We propose an optimal algorithm that uses optimal weighting to combine the resource of neighbors. We define an optimization problem and find the solution by applying the proposed algorithm. We evaluate the performance of the developed algorithm by using both mathematical methods and computer simulations. We introduce the conditions in which the convergence of proposed algorithm is guaranteed and prove that the network error decreases considerably in comparison with some of the known modern methods. [ABSTRACT FROM AUTHOR]

Workload scheduling in cloud computing is currently an active research field. Scheduling plays an important role in cloud computing performance, especially when the platform is used for big data analysis and as less predictable workloads dynamically enter the clouds. Finding the optimized scheduling solution with different parameters in different environments is still a challenging issue. In dynamic environments such as cloud, scheduling strategies should feature rapid altering to be able to adapt more easily to the changes in input workloads. However, achieving an optimized solution is an important issue, which has a trade-off with the speed of finding the solution. In this article, an ordinal optimization method is proposed that considers the volume of workloads, load balancing and the volume of exchanged messages among virtual clusters, considering the replications. The algorithm in the present paper is based on ordinal optimization (OO) and evolutionary OO. In any time periods, a criterion is calculated to determine the similarity of workloads in two-consequence time periods, which is appropriate for timely changes in the scheduling procedure. In this paper, considering more than one parameter, a proper scheduling would be created for each time period. This scheduler is an organization for the number of virtual machines for each virtual cluster, but if there is a desirable similarity between workloads of two-consequence time periods, this procedure would be ignored. The results show that a more optimized solution is obtained in comparison with the rated methods, such as blind pink, OO, Monte Carlo and eOO in a reasonable time. The suggested method is flexible and it is possible to change the weight ratio of the proposed criteria in different environments to be consistent with different environmental conditions. The results show that proposed method achieved up to 28% performance improvement in comparison with eOO. [ABSTRACT FROM AUTHOR]

Finding frequent itemsets in a continuous streaming data is an important data mining task which is widely used in network monitoring, Internet of Things data analysis and so on. In the era of big data, it is necessary to develop a distributed frequent itemset mining algorithm to meet the needs of massive streaming data processing. Apache Spark is a unified analytic engine for massive data processing which has been successfully used in many data mining fields. In this paper, we propose a distributed algorithm for mining frequent itemsets over massive streaming data named SWEclat. The algorithm uses sliding window to process streaming data and uses vertical data structure to store the dataset in the sliding window. This algorithm is implemented by Apache Spark and uses Spark RDD to store streaming data and dataset in vertical data format, so as to divide these RDDs into partitions for distributed processing. Experimental results show that SWEclat algorithm has good acceleration, parallel scalability and load balancing. [ABSTRACT FROM AUTHOR]

Evolutionary community discovery is a hot research topic related to the dynamic or temporal social networks. The communities detected in a dynamic network should get reasonable partition for the current network and do not deviate drastically from the previous ones. This paper is an extended version of our previous work in Gao et al. (in: Proceedings of the 2016 international conference on big data and smart computing (BigComp), pp 53–60, 2016). First, an evolutionary community discovery algorithm named EvoLeaders, which is inspired by TopLeaders algorithm, is proposed. Second, based on TopLeaders, an improved TopLeaders algorithm (i.e., AutoLeaders) is proposed. Experiments on three classic data sets are conducted, and experimental results show that the AutoLeaders can correctly find the number of communities and at the same time can discover reasonable communities. Third, the EvoAutoLeaders algorithm is proposed for detecting the communities in a dynamic network. Compared with the TopLeaders algorithm and EvoLeaders, experimental results over two real-world data sets demonstrate that the EvoAutoLeaders is more suitable for dynamic scenarios. [ABSTRACT FROM AUTHOR]

Data assimilation is an analysis technique which combines observations and the numerical results from theoretical models to deduce more realistic and accurate data. It is widely used in investigations of the atmosphere, ocean and land surface. Due to the complicated data structure of the inputs from dynamical models and the increase of the amount of model data, the parallelization of data assimilation suffers from high overhead on file reading and data communication. In this paper, we propose a flexible parallel data access approach for reading a large number of data from disks firstly. Using this approach, the data access conflict is avoided successfully, and the frequency of disk addressing operations is also decreased significantly. Next, we design a communication-avoiding strategy to reduce the communication volume at the cost of some additional computations. Furthermore, we present a "pipe-flow" scheme for data exchange to conduct conflict-free message passing. Consequently, a fast data-obtaining algorithm is developed for the data assimilation. Our experiments show that the fast data-obtaining algorithm gains a performance of 5 × speedup compared with the baseline, which is excellent at data-obtaining for the parallel data assimilation. Due to the reduction of disk addressing operations, the new approach achieves 6 × speedup on average for the file reading process. Since a large amount of data movement can be avoided, the new approach achieves 2.7 × speedup on average for the communication between processors. [ABSTRACT FROM AUTHOR]

Dimension reduction tools offer a popular approach to analysis of high-dimensional big data. In this paper, we propose an algorithm for sparse Principal Component Analysis for non-Gaussian data. Since our interest for the algorithm stems from applications in text data analysis we focus on the Poisson distribution which has been used extensively in analysing text data. In addition to sparsity our algorithm is able to effectively determine the desired number of principal components in the model (order determination). The good performance of our proposal is demonstrated with both synthetic and real data examples. [ABSTRACT FROM AUTHOR]

With the emergence of big data, deep learning (DL) approaches are becoming quite popular in many branches of science. Forensic science is no longer an exception. However, there are certain problems in forensic science where the solutions would hardly benefit from the recent advances in DL algorithms. Document authentication is one such problem where we can have many reference samples, and with the big data scenario probably we would have even more number of reference samples but number of defective or forged samples will remain an issue. Experts often encounter situations where there is no or hardly a scanty number of forged samples available. In such situation, employment of data-hungry algorithms would be inefficient as they will not be able to learn the forged samples properly. This paper addresses this problem and proposes a novel reference modelling framework for forensic document authentication. The approach is based on Mahalanobis space. Two questioned document examination problems have been studied to show the effectiveness of our reference modelling algorithm which has also been compared to a commonly used learning approach, namely neural network-based classification. [ABSTRACT FROM AUTHOR]

Understanding how people move from one location to another is important both for smart city planners and destination managers. Big-data generated on social media sites have created opportunities for developing evidence-based insights that can be useful for decision-makers. While previous studies have introduced observational data analysis methods for social media data, there remains a need for method development—specifically for capturing people's movement flows and behavioural details. This paper reports a study outlining a new analytical method, to explore people's activities, behavioural, and movement details for people monitoring and planning purposes. Our method utilises online geotagged content uploaded by users from various locations. The effectiveness of the proposed method, which combines content capturing, processing and predicting algorithms, is demonstrated through a case study of the Fiji Islands. The results show good performance compared to other relevant methods and show applicability to national decisions and policies. [ABSTRACT FROM AUTHOR]

The visualization of spatial data is becoming increasingly important in science, business and many other areas. There are two main reasons for this: First, the amount of spatial data is growing continuously, making it impossible for people to manually process the data in raw form. Secondly, users have very high demands on the interactive processing of big spatial data in visual form. For instance in geography, data often corresponds to a large number of point observations that should be displayed on a constrained screen with limited resolution. This causes two crucial problems: drawing a lot of points is expensive at runtime and leads to a loss of information due to an overloaded and occluded visualization. In this paper we present a new efficient visualization algorithm that avoids these problems by aggregating point data into a set of non-overlapping circles with the following properties: (i) they follow the distribution of the data, (ii) they represent the cardinality of the underlying point subset by the circle area, (iii) they reveal hot spots while simultaneously keeping outliers, and (iv) the number of circles is typically much smaller than the number of points. Based on a quadtree, our algorithm computes the circles in linear time with respect to the number of points. Experimental results confirm its excellent runtime and quality in comparison to competitors. [ABSTRACT FROM AUTHOR]

Unsupervised ensemble, or consensus clustering, consists in finding the optimal combination strategy of individual clusterings that is robust with respect to the selection of an algorithmic clustering pool. Recently an approach was proposed based on the concept of consensus graph that has profound advantages over its predecessors. Despite its robust properties this approach assigns the same weight to the contribution of each clustering to the final solution. In this paper, we propose a weighting policy for this problem that is based on internal clustering quality measures and compare against other popular approaches. Results on publicly available datasets show that weights can significantly improve the accuracy performance while retaining the robust properties. [ABSTRACT FROM AUTHOR]

The concept of the digital phenotype has been used to refer to digital data prognostic or diagnostic of disease conditions. Medical conditions may be inferred from the time pattern in an insomniac's tweets, the Facebook posts of a depressed individual, or the web searches of a hypochondriac. This paper conceptualizes digital data as an extended phenotype of humans, that is as digital information produced by humans and affecting human behavior and culture. It argues that there are ethical obligations to persons affected by generalizable knowledge of a digital phenotype, not only those who are personally identifiable or involved in data generation. This claim is illustrated by considering the health-related digital phenotypes of precision medicine and digital epidemiology. [ABSTRACT FROM AUTHOR]

With the advances in new-generation information technologies, smart process planning is becoming the focus for smart process planning with less time and lower cost. Big data-based reusing and evaluating the multi-dimensional process knowledge is widely accepted as an effective strategy for improving competitiveness of enterprises. However, there was little research on how to reuse and evaluate process knowledge with dynamical changing machining status. In this paper, we propose a novel digital twin-based approach for reusing and evaluating process knowledge. First, the digital twin-based process knowledge model which contains the geometric information and real-time process equipment status is introduced to represent the purpose and requirement of machining planning. Second, the process big data is constructed based on the three-layer and its association rules for accumulating process knowledge. Moreover, the similarity calculation algorithm of the scene model is proposed to filter the unmatched process knowledge. For accurately reusing the process knowledge, the process reusability evaluation approach of the candidate knowledge set is presented based on the real-time machining status and the calculated confidence. Finally, the diesel engine parts are applied in the developed prototype module to verify the effectiveness of the proposed method. The proposed method can promote the development and application of the smart process planning. [ABSTRACT FROM AUTHOR]

Metadata management is an essential part in Apache Hadoop. Performing optimization of metadata accesses enhances big data storing, processing and analyzing, especially in multitenant environments. Nevertheless, as environmental complexity increases, metadata management is becoming more challenging and costly because of the heavy performance issues. In this paper, we propose a novel approach to improve the performance of metadata management for Hadoop in the multitenant environment based on the prefetching mechanism. We create metadata access graphs based on historical access values, define access patterns and then perform prefetching potential items for the near-future requests to minimize the latency. We present a formal algorithm to apply the prefetching mechanism into the Hadoop system and perform the actual implementation on a recent Hadoop system. Experimental results show that the proposed approach can enable the high performance for metadata management as well as maintain advanced multitenancy features. [ABSTRACT FROM AUTHOR]

Record Matching refers to identifying pairs of records that relate to the same entities across different data sources. In many applications of data mining, record matching is usually associated to quadratic complexity. In practice, the number of non-matching record pairs always far exceeds the number of matching pairs, and this is called imbalance problem. Blocking is a technique of data reduction, which can filter unlikely matching pairs before record matching. However, for big data there is no fast and effective blocking algorithm yet. In this paper, we report on big data infrastructure to improve efficiency of blocking. Our approach runs blocking process independently and distributedly on the partitions of whole data. To improve efficiency, we adopt a probabilistic technique to balance the speed and the effect of the algorithm that we proposed for distributed blocking. Our experimental analysis endorses the superiority of our technique and shows its novel scalability. [ABSTRACT FROM AUTHOR]

In Homo Deus, Yuval Noah Harari argues that technological advances of the twenty-first century will usher in a significant shift in how humans make important life decisions. Instead of turning to the Bible or the Quran, to the heart or to our therapists, parents, and mentors, people will turn to Big Data recommendation algorithms to make these choices for them. Much as we rely on Spotify to recommend music to us, we will soon rely on algorithms to decide our careers, spouses, and commitments. Harari also predicts that next, the state will take away individuals’ rights to make their own choices about their lives. If Google knows where your children would flourish best in school, why should the state allow a fallible human parent to decide? Liberalism—which, as Harari uses this term, refers to a state of society in which human freedom to choose is respected and championed—will collapse. In this paper, I argue that Harari’s conception of the future implications of recommendation algorithms is deeply flawed, for two reasons. First, users will not rely on algorithms to make decisions for them because they have no reason to trust algorithms, which are developed by companies with their own incentives, such as profit. Second, for most of our life decisions, algorithms will not be able to be developed, because the factors relevant to the decisions we face are unique to our situation. I present an alternative depiction of the future: instead of relying on algorithms to make decisions for us, humans will use algorithms to enhance our decision-making by helping us consider the most relevant choices first and notice information we might not otherwise. Finally, I will also argue that even if computers could make many of our decisions for us, liberalism as a political system would emerge unscathed. [ABSTRACT FROM AUTHOR]

This article is based on a roundtable held at the Association of Internet Researchers (AoIR) annual conference in 2017, in Tartu, Estonia. The roundtable was organized by the Oxford Internet Institute’s (OII) Digital Ethics Lab (DELab). It was entitled “Artificial Intelligence and the Good Society”. It brought together four scholars—Michael Zimmer, Stine Lomborg, Ben Zevenbergen, and Corinne Cath—to discuss the promises and perils of artificial intelligence (AI), in particular what ethical frameworks are needed to guide AI’s rapid development and increased use in societies. The paper covers three case studies. They give a distinct overview of the ethical issues raised by the use of AI at different levels of analysis: top-down application of AI, bottom-up use of AI, and how academics and governments have reacted to these new challenges. From the case studies, four areas emerged. They represent some of the most topical ethical questions related to AI: (1) its uses, (2) its users, (3) its designers, and (4) the data that fuel it. Each of them provided a specific subset of ethical concerns that need further investigation. In conclusion, three recommendations are formulated for researchers and regulators to ensure the AI has a net-positive impact on society. [ABSTRACT FROM AUTHOR]

Scaling skyline queries over high-dimensional datasets remains to be challenging due to the fact that most existing algorithms assume dimensional independence when establishing the worst-case complexity by discarding correlation distribution. In this paper, we present HashSkyline, a systematic and correlation-aware approach for scaling skyline queries over high-dimensional datasets with three novel features: First, it offers a fast hash-based method to prune non-skyline points by utilizing data correlation characteristics and speed up the overall skyline evaluation for correlated datasets. Second, we develop $$HashSkyline_{GPU}$$ , which can dramatically reduce the response time for anti-correlated and independent datasets by capitalizing on the parallel processing power of GPUs. Third, the HashSkyline approach uses the pivot cell-based mechanism combined with the correlation threshold to determine the correlation distribution characteristics for a given dataset, enabling adaptive configuration of HashSkyline for skyline query evaluation by auto-switching of $$HashSkyline_{CPU}$$ and $$HashSkyline_{GPU}$$ . We evaluate the validity of HashSkyline using both synthetic datasets and real datasets. Our experiments show that HashSkyline consumes significantly less pre-processing cost and achieves significantly higher overall query performance, compared to existing state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

As the volume of data generated each day continues to increase, more and more interest is put into Big Data algorithms and the insight they provide.? Since these analyses require a substantial amount of resources, including physical machines, power, and time, reliable execution of the algorithms becomes critical. This paper analyzes the error resilience of a select group of popular Big Data algorithms and shows how they can effectively be made more fault-tolerant. Using KULFI (, 2013) and the LLVM (Proceedings of the 2004 international symposium on code generation and optimization (CGO 2004), San Jose, CA, USA, 2004) compiler for compilation allows injection of artificial soft faults throughout these algorithms, giving a thorough analysis of how faults in different locations can affect the outcome of the program. This information is then used to help guide incorporating fault tolerance mechanisms into the program, making them as impervious as possible to soft faults. [ABSTRACT FROM AUTHOR]

Data decompression is one of the most important techniques in data processing and has been widely used in multimedia information transmission and processing. However, the existing decompression algorithms on multicore platforms are time-consuming and do not support large data well. In order to expand parallelism and enhance decompression efficiency on large-scale datasets, based on the software thread-level speculation technique, this paper raises a speculative parallel decompression algorithm on Apache Spark. By analyzing the data structure of the compressed data, the algorithm firstly hires a function to divide compressed data into blocks which can be decompressed independently and then spawns a number of threads to speculatively decompress data blocks in parallel. At last, the speculative results are merged to form the final outcome. Comparing with the conventional parallel approach on multicore platform, the proposed algorithm is very efficiency and obtains a high parallelism degree by making the best of the resources of the cluster. Experiments show that the proposed approach could achieve 2.6 $$\times $$ speedup when comparing with the traditional approach in average. In addition, with the growing number of working nodes, the execution time cost decreases gradually, and the speedup scales linearly. The results indicate that the decompression efficiency can be significantly enhanced by adopting this speculative parallel algorithm. [ABSTRACT FROM AUTHOR]

This paper presents a new algorithm for determining radial efficiency with a large data set by using small-size linear programs (LPs). Instead of trying to 'reduce' the size of individual LPs, the proposed algorithm attempts to 'control' the size of individual LPs, e.g., no more than 100 data points each time while maintaining the solution quality. The algorithm is specifically designed to address the problem of LP size limitation. From the empirical results, we conclude that the proposed algorithm can converge within a reasonable number of iterations without incurring extra computation time and has savings of up to 60 % of the benchmarks when the data set contains 15,000 points. [ABSTRACT FROM AUTHOR]

This paper proposes three methods of association analysis that address two challenges of Big Data: capturing relatedness among real-world events in high data volumes, and modeling similar events that are described disparately under high data variability. The proposed methods take as input a set of geotemporally-encoded text streams about violent events called 'storylines'. These storylines are associated for two purposes: to investigate if an event could occur again, and to measure influence, i.e., how one event could help explain the occurrence of another. The first proposed method, Distance-based Bayesian Inference, uses spatial distance to relate similar events that are described differently, addressing the challenge of high variability. The second and third methods, Spatial Association Index and Spatio-logical Inference, measure the influence of storylines in different locations, dealing with the high-volume challenge. Extensive experiments on social unrest in Mexico and wars in the Middle East showed that these methods can achieve precision and recall as high as 80 % in retrieval tasks that use both keywords and geospatial information as search criteria. In addition, the experiments demonstrated high effectiveness in uncovering real-world storylines for exploratory analysis. [ABSTRACT FROM AUTHOR]

Big data trends have recently brought unrivalled opportunities to the cloud systems. Numerous virtual machines (VMs) have been widely deployed to enable the on-demand provisioning and pay-as-you-go services for customers. Due to the large complexity of the current cloud systems, promising VM placement algorithm are highly desirable. This paper focuses on the energy efficiency and thermal stability issues of the cloud systems. A Cross Entropy based VM Placement (CEVP) algorithm is proposed to simultaneously minimize the energy cost, total thermal cost and the number of hot spots in the data center. Simulation results indicate that the proposed CEVP algorithm can (1) achieve energy savings of 26.2 % on average, (2) efficiently reduce the temperature cost by up to 6.8 % and (3) significantly decrease the total number of the hot spots by 60.1 % on average in the cloud systems, by comparing to the Ant Colony System-based algorithm. [ABSTRACT FROM AUTHOR]

Background: Predicting novel drug–target associations is important not only for developing new drugs, but also for furthering biological knowledge by understanding how drugs work and their modes of action. As more data about drugs, targets, and their interactions becomes available, computational approaches have become an indispensible part of drug target association discovery. In this paper we apply random walk with restart (RWR) method to a heterogeneous network of drugs and targets compiled from DrugBank database and investigate the performance of the method under parameter variation and choice of chemical fingerprint methods. Results: We show that choice of chemical fingerprint does not affect the performance of the method when the parameters are tuned to optimal values. We use a subset of the ChEMBL15 dataset that contains 2,763 associations between 544 drugs and 467 target proteins to evaluate our method, and we extracted datasets of bioactivity ≤1 and ≤10 μM activity cutoff. For 1 μM bioactivity cutoff, we find that our method can correctly predict nearly 47, 55, 60% of the given drug–target interactions in the test dataset having more than 0, 1, 2 drug target relations for ChEMBL 1 μM dataset in top 50 rank positions. For 10 μM bioactivity cutoff, we find that our method can correctly predict nearly 32.4, 34.8, 35.3% of the given drug–target interactions in the test dataset having more than 0, 1, 2 drug target relations for ChEMBL 1 μM dataset in top 50 rank positions. We further examine the associations between 110 popular top selling drugs in 2012 and 3,519 targets and find the top ten targets for each drug. Conclusions: We demonstrate the effectiveness and promise of the approach—RWR on heterogeneous networks using chemical features—for identifying novel drug target interactions and investigate the performance. [ABSTRACT FROM AUTHOR]

In wireless sensor networks, sensor nodes are usually powered by battery and thus have very limited energy. Saving energy is an important goal in designing a WSN. It is known that clustering is an effective method to prolong network lifetime. Due to the development of big data, there are more sensor nodes and data needed to process. So how to cluster sensor nodes cooperatively and achieve an optimal number of clusters in a big data WSN is an open issue. In this paper, we first propose an analytical model to give the optimal number of clusters in a wireless sensor network. We then propose a centralized cluster algorithm based on spectral partitioning method. After that, we present a distributed implementation of the clustering algorithm based on fuzzy C-means method. Finally, we conduct extensive simulations, and the results show that the proposed algorithms outperform the hybrid energy-efficient distributed (HEED) clustering algorithm in terms of energy cost and network lifetime. [ABSTRACT FROM AUTHOR]

Real-time positioning of a specific object in the big data environment can improve the monitoring and management capacity for network data. For the real-time positioning of the specific object, it is necessary to quickly search the network data representing a specific object and match its pattern strings and compare the corresponding Internet protocol (IP) address of the matched network data with the IP address library in real time, so as to determine the position of the specific object. When a traditional method is used for pattern string matching, it will occupy a lot of memories and network resources, thereby reducing the positioning effect of the specific object in the big data environment. A positioning method for a specific object of high performance and multi-pattern matching based on three indexes in the big data network environment is proposed in this paper. Firstly, the initialization of Modified Wu-Manber (MWM) algorithm was carried out, and the algorithm was used to match the network data continuously. Secondly, the three indexes were used to improve the MWM algorithm, and the real-time and fast positioning of a specific object in the big data environment was completed by the Third Index Modified Wu-Manber (TMWM). The experimental results show that compared with the traditional method, the proposed algorithm reduces the pattern string matching scope of network data representing the specific object, improves the search speed of the specific object, and locates the specific object in the big data environment in an effective and rapid manner. [ABSTRACT FROM AUTHOR]