Search

Flooding disaster causes huge impacts on the socio-economic world. In the inundated area, some geo-referenced images are shared through some media posts, which assist in providing alertness to the critical volunteers and managing the financial loss crisis. In this work, the Adaptive Billiards-Inspired Optimization (A-BIO) and Optimized Ensemble-learning-based detection (OED) with map reduce framework is proposed for flood disaster detection. Initially, the big data is gathered and processed for detection. During the map phase, data preprocessing is performed to enhance the performance of the data, which helps in removing the noise or unwanted attributes. Furthermore, the reduction phase can be done through weighted feature selection, where the features to be selected and the weight is optimized through A-BIO, which assists in getting the most significant features for improving the performance and reducing the complexity of the designed model. Finally, OED is performed by a set of classifiers like Convolutional Neural Networks, Adaboost, XGBoost, Long Short-Term Memory, and Deep Neural Networks, where the parameters of ensemble learning classifiers are optimized by A-BIO algorithm. Finally, through the performance analysis, this detection model can provide high accuracy and better detection performance to avoid the huge impacts of flood disasters. [ABSTRACT FROM AUTHOR]

Major countries in the world attach great importance to the development of big data technology. China also puts big data as a national strategy, of great significance to develop in the long run. Big data technologies include data collection, transmission, management, processing, analysis, and application, forming a data life cycle as well as the data governance related to each procedure. Big data management, processing, analysis, and governance in four areas were seleceted, to identify the gap between China and the world. On the other hand, driven by diverse successful big data applications, the system architecture of computing technology is being restructured. From “computation-centric” to “data-centric”, fundamental computing theories and core technologies need to be redesigned, therefore a new type of big data system technology is becoming an important research direction. Against this background, four technical challenges and ten future development trends of big data technologies were aimed at identifying. [ABSTRACT FROM AUTHOR]

In this contribution two algorithms are developed to solve non-linear system of equations which can contain a large number of measurements. These algorithms are based on nonlinear block least-squares (BLS). Although block least squares was investigated by some researchers, the non-linear case was not examined by now. The first algorithm is proposed to solve a special case of non-linear problems that do not require linearization. Such an algorithm can be called total block least-squares. The second algorithm is based on linearization within a general nonlinear mixed model using a new notation which is in agreement with the rigorous linearization presented by Pope. Both of these algorithms can handle constraints on the parameters. By use of these algorithms, big data processing is feasible with inexpensive computers. Furthermore, expensive processors can solve systems with a large number of equations faster. Two case studies with more than 120,000 equations show that fast and accurate computations are possible by applying these algorithms without any loss of accuracy. [ABSTRACT FROM AUTHOR]

Big data and data science are the two most prominent contemporary developments. Big data analysis increases the requirement for innovative system designs, which promotes the creation of procedures that can manage massive data quantities while maintaining the ag ility, flexibility, and interactive feel that a data scientist requires. Big Data is the outcome of data being created at an exponential rate. This data is diverse and Includes unstructured, and semi-structured data types. It provides valuable information for many Sorts of stakeholders based on their needs, but it cannot be met using standard tools and procedures. Big data technologies play a critical role in handling, storing, and processing this massive quantity of data. It is further classified as text analytics, audio analytics, video analytics, and social media analytics. Big data analytics, when combined with big data analysis, has a major impact. [ABSTRACT FROM AUTHOR]

This paper develops a multi-dimensional Dynamic Time Warping (DTW) algorithm to identify varying lead-lag relationships between two different time series. Specifically, this manuscript contributes to the literature by improving upon the use towards lead-lag estimation. Our two-step procedure computes the multi-dimensional DTW alignment with the aid of shapeDTW and then utilises the output to extract the estimated time-varying lead-lag relationship between the original time series. Next, our extensive simulation study analyses the performance of the algorithm compared to the state-of-the-art methods Thermal Optimal Path (TOP), Symmetric Thermal Optimal Path (TOPS), Rolling Cross-Correlation (RCC), Dynamic Time Warping (DTW), and Derivative Dynamic Time Warping (DDTW). We observe a strong outperformance of the algorithm regarding efficiency, robustness, and feasibility. [ABSTRACT FROM AUTHOR]

Underwater data acquisition entities acquire big data that are processed aboard terrestrial data centres. However, processing the big data aboard terrestrial computing entities involves high latency data transfer. In addition, the processing of data in a terrestrial environment is challenging when there is inadequate edge node capacity. These challenges are addressed here. The paper proposes the heterogeneous edge computing paradigm to realize low latency transfer of increasing underwater big data. This is realized via the use of underwater computing entities instead of terrestrial computing entities for processing acquired big data. The proposed heterogeneous edge computing paradigm presents the multi-mode automated teller machine (ATM) as low cost terrestrial edge network entity. The multi-mode ATM is suitable when edge nodes have inadequate computing capacity. Performance evaluation shows that the use of underwater computing entities instead of terrestrial computing entities (existing work) enhances network performance and related capital costs. The number of hops, computing entity access latency and required autonomous underwater vehicle acquisition costs by an average of (5.3–88.4)%, 63.5% and (31.8–95.4)%, respectively. Evaluation shows that the use of the multi-mode ATM in the context of terrestrial cloud computing reduces the number of hops and latency by 44.4% and 37.3% on average, respectively. [ABSTRACT FROM AUTHOR]

• We apply NLP approaches in features choice for enhancing Classifier based on Random Forests approach. • We analyze medical records to retrieve features for clinical records classifications in smart environments. • We use a Machine Learning approach to build a fast multi-classification schema. • We apply the methodology to real case studies from health-care organizations in Italy. • We show accuracy of presented approach in terms of Accuracy-Rejection curves. The use of pervasive IoT devices in Smart Cities, have increased the Volume of data produced in many and many field. Interesting and very useful applications grow up in number in E-health domain, where smart devices are used in order to manage huge amount of data, in highly distributed environments, in order to provide smart services able to collect data to fill medical records of patients. The problem here is to gather data, to produce records and to analyze medical records depending on their contents. Since data gathering involve very different devices (not only wearable medical sensors, but also environmental smart devices, like weather, pollution and other sensors) it is very difficult to classify data depending their contents, in order to enable better management of patients. Data from smart devices couple with medical records written in natural language: we describe here an architecture that is able to determine best features for classification, depending on existent medical records. The architecture is based on pre-filtering phase based on Natural Language Processing, that is able to enhance Machine learning classification based on Random Forests. We carried on experiments on about 5000 medical records from real (anonymized) case studies from various health-care organizations in Italy. We show accuracy of the presented approach in terms of Accuracy-Rejection curves. [ABSTRACT FROM AUTHOR]

This paper takes education data mining as the research theme, mine the existing massive education big data, compares the analysis methods of existing data models, and proposes an improved random forest reference model. The information gain of various features is calculated by introducing the feature weighting system, and the evaluation index is used to improve the existing data analysis. The simulation results show that the improved model is highly efficient as compared to the existing models for classification. In order to resolve the performance bottleneck of a single node in multiple data classification tasks in the era of big data, a classification and prediction model of graduates' large-scale employment data, based on distributed improved RF algorithm, is proposed. The MapReduce distributed computing framework is used to complete the serial writing and deserialization loading of the training model between the local disk and the distributed file system, and realizing the distributed expansion of the large-scale data classification model based on the improved RF model. [ABSTRACT FROM AUTHOR]

In today' s big data era, big data processing frameworks such as MapReduce often appear slow and inefficient when processing data, specially related to graphs. Therefore, it is necessary to explore an efficient algorithm to handle this type of clique counting problem. Since the predecessor literatures have thoroughly explored the 3-clique counting, the extended version of the problem( the 4-clique counting problem) improves its position gradually. Under the guidance of a heuristic idea, this paper proposed a probability sampling algorithm based on neighboring edge sampling to solve the extended problem. With the usage of Chernoff inequality, the algorithm only needed a certain number of samplers as the performance guarantee of relative error under the approximate condition. Later, the experimental evaluation and comparison shows that the probability sampling algorithm loses a small amount of precision compared with the traditional precision algorithm, but it has great advantages in algorithm running time and space occupation. Finally, it comes to the conclusion that it has great practical value in practical applications. [ABSTRACT FROM AUTHOR]

In a world where every day we produce 2.5 quintillion bytes of data, sentiment analysis has been a key for making sense of that data. However, to process huge text data in real-time requires building a data processing pipeline in order to minimize the latency to process data streams. In this paper, we explain and evaluate our proposed real-time customer' sentiment analysis pipeline on the Moroccan banking sector through data from the web and social network using open-source big data tools such as data ingestion using Apache Kafka, In-memory data processing using Apache Spark, Apache HBase for storing tweets and the satisfaction indicator, and ElasticSearch and Kibana for visualization then NodeJS for building a web application. The performance evaluation of Naïve Bayesian model show that for French Tweets the accuracy has reached 76.19% while for English Tweets the result was unsatisfactory and the resulting accuracy is 56%. To remedy this problem, we used the Stanford core NLP which, for English Tweets, reaches a precision of 80.7%. [ABSTRACT FROM AUTHOR]

Background: The emergence of high throughput technologies that produce vast amounts of genomic data, such as next-generation sequencing (NGS) is transforming biological research. The dramatic increase in the volume of data, the variety and continuous change of data processing tools, algorithms and databases make analysis the main bottleneck for scientific discovery. The processing of high throughput datasets typically involves many different computational programs, each of which performs a specific step in a pipeline. Given the wide range of applications and organizational infrastructures, there is a great need for highly parallel, flexible, portable, and reproducible data processing frameworks. Several platforms currently exist for the design and execution of complex pipelines. Unfortunately, current platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, workflow frameworks that provide a platform to develop and share portable pipelines have recently arisen. We complement these new platforms by providing a graphical user interface to create, maintain, and execute complex pipelines. Such a platform will simplify robust and reproducible workflow creation for non-technical users as well as provide a robust platform to maintain pipelines for large organizations. Results: To simplify development, maintenance, and execution of complex pipelines we created DolphinNext. DolphinNext facilitates building and deployment of complex pipelines using a modular approach implemented in a graphical interface that relies on the powerful Nextflow workflow framework by providing 1. A drag and drop user interface that visualizes pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. Modules to execute and monitor pipelines in distributed computing environments such as high-performance clusters and/or cloud 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Modular process design with process revisioning support to increase reusability and pipeline development efficiency. 5. Pipeline sharing with GitHub and automated testing 6. Extensive reports with R-markdown and shiny support for interactive data visualization and analysis. Conclusion: DolphinNext is a flexible, intuitive, web-based data processing and analysis platform that enables creating, deploying, sharing, and executing complex Nextflow pipelines with extensive revisioning and interactive reporting to enhance reproducible results. [ABSTRACT FROM AUTHOR]

In-memory techniques keep data into faster and more expensive storage media for improving performance of big data processing. However, existing mechanisms do not consider how to expedite the data processing applications that access the input datasets only once. Another problem is how to reclaim memory without affecting other running applications. In this paper, we provide scheduling-aware data prefetching and eviction mechanisms based on Spark, Alluxio, and Hadoop. The mechanisms prefetch data and release memory resources based on the scheduling information. A mathematical method is proposed for maximizing the reduction of data access time. To make the mechanisms applicable in large-scale environments, we propose a heuristic algorithm to reduce the computational time. Furthermore, an enhanced version of the heuristic algorithm is also proposed to increase the amount of prefetched data. Finally, we perform real-testbed and simulation experiments to show the effectiveness of the proposed mechanisms. [ABSTRACT FROM AUTHOR]

Processing in memory (PIM), the concept of integrating processing directly with memory has been attracting a lot of attention, since PIM can assist in overcoming the throughput limitation caused by data movement between CPU and memory. The challenge, however, is that it requires the programmers to have a deep understanding of the PIM architecture to maximize the benefits such as data locality and parallel thread execution on multiple PIM devices. In this study, we present AnalyzeThat, a programmable shared-memory system for parallel data processing with PIM devices. Thematic to AnalyzeThat is a rich PIM-aware data structure (PADS), which is an encapsulation that integrally ties together the data, the analysis tasks and the runtime needed to interface with the PIM device array. The PADS abstraction provides (i) a sophisticated key-value data container that allows programmers to easily store data on multiple PIMs, (ii) a suite of parallel operations with which users can easily implement data analysis applications, and (iii) a runtime, hidden to programmers, which provides the mechanisms needed to overlay both the data and the tasks on the PIM device array in an intelligent fashion, based on PIM-specific information collected from the hardware. We have developed a PIM emulation framework called AnalyzeThat. Our experimental evaluation with representative data analytics applications suggests that the proposed system can significantly reduce the PIM programming effort without losing its technology benefits. [ABSTRACT FROM AUTHOR]

In view of the warning requirement for university students who have academic difficulty, through the student admission scores, semester grades, IC card application, morning exercises attendance and status data processing and other state data, based on large data processing, this paper puts forward KFCM (kernelized fuzzy C-means) to improve the SVM (support vector machine) data decision algorithm, and to develop students’ academic warning decision research. Teachers and students can take effective intervention measures to provide technical support. Improved Algorithm, through simulation analysis, is compared with the traditional data decision algorithms by using SVM, BP neural network, and GA-SVM, the proposed algorithm has greater advantages in solving the students’ academic warning area, and has the certain application value. [ABSTRACT FROM AUTHOR]

We aimed to control complex wind power generation parameters' contributions to achieve healthful and energy-saving operating conditions for wind turbines and improve the stability of wind power generation and processing capacity of wind grid output data. Hence, a design method of complex wind power generation parameter control system based on embedded control combined with Internet of Things (IoT) is proposed. In this method, in the IoT environment, the overall model of the system is determined, and embedded control for big data of the complex wind power generation parameters is performed to achieve adaptive fusion of control instruction information, and optimize the big data transmission and scheduling of control instruction sets. Test results show that the complex wind power generation parameter control system designed through this method has relatively strong parameter data control processing capability, good stability of control instruction transmission and high parallel processing capability of control instruction sets, and this method improves the control quality of the overall system. [ABSTRACT FROM AUTHOR]

Big data analysis is very important to support rescue activities when natural disaster happens, through understanding various situations, such as power/water outage regions. The traditional way to process big data is based on high-performance computation/storage resources in a cloud center. However, this is hard to be guaranteed in a disaster scenario due to destruction of communication infrastructure. Meanwhile, high latency between local sensing devices and cloud center sets a big obstacle enabling a near real-time big data analysis. On the other hand, movable base station, such as vehicle-based movable & deployable ICT resource unit (MDRU) developed by NTT, is a possible solution to reconstruct an emergency communication network and process data at the edge sites with reduced data transmission time. In this paper, we study the optimal overall delay in a fog/edge-computing platform constructed by vehicle-based MDRUs with guaranteed data resolution. We formalize the problem as a mixed-integer nonlinear program, which is a well-known NP-hard problem, and then relax the original problem to an mixed integer linear programing (MILP). Finally, we propose a two-stage heuristic algorithm to solve it in a time-efficient manner. Through evaluation, the effectiveness of the proposed heuristic approach has been validated in terms of minimizing overall delay with sufficient given data resolutions. [ABSTRACT FROM AUTHOR]

With the increase in the amount of data captured during the manufacturing process, monitoring systems are becoming important factors in decision making for management. Current technologies such as Internet of Things (IoT)-based sensors can be considered a solution to provide efficient monitoring of the manufacturing process. In this study, a real-time monitoring system that utilizes IoT-based sensors, big data processing, and a hybrid prediction model is proposed. Firstly, an IoT-based sensor that collects temperature, humidity, accelerometer, and gyroscope data was developed. The characteristics of IoT-generated sensor data from the manufacturing process are: real-time, large amounts, and unstructured type. The proposed big data processing platform utilizes Apache Kafka as a message queue, Apache Storm as a real-time processing engine and MongoDB to store the sensor data from the manufacturing process. Secondly, for the proposed hybrid prediction model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection and Random Forest classification were used to remove outlier sensor data and provide fault detection during the manufacturing process, respectively. The proposed model was evaluated and tested at an automotive manufacturing assembly line in Korea. The results showed that IoT-based sensors and the proposed big data processing system are sufficiently efficient to monitor the manufacturing process. Furthermore, the proposed hybrid prediction model has better fault prediction accuracy than other models given the sensor data as input. The proposed system is expected to support management by improving decision-making and will help prevent unexpected losses caused by faults during the manufacturing process. [ABSTRACT FROM AUTHOR]

Cloud computing allows execution and deployment of different types of applications such as interactive databases or web-based services which require distinctive types of resources. These applications lease cloud resources for a considerably long period and usually occupy various resources to maintain a high quality of service (QoS) factor. On the other hand, general big data batch processing workloads are less QoS-sensitive and require massively parallel cloud resources for short period. Despite the elasticity feature of cloud computing, fine-scale characteristics of cloud-based applications may cause temporal low resource utilization in the cloud computing systems, while process-intensive highly utilized workload suffers from performance issues. Therefore, ability of utilization efficient scheduling of heterogeneous workload is one challenging issue for cloud owners. In this paper, addressing the heterogeneity issue impact on low utilization of cloud computing system, conjunct resource allocation scheme of cloud applications and processing jobs is presented to enhance the cloud utilization. The main idea behind this paper is to apply processing jobs and cloud applications jointly in a preemptive way. However, utilization efficient resource allocation requires exact modeling of workloads. So, first, a novel methodology to model the processing jobs and other cloud applications is proposed. Such jobs are modeled as a collection of parallel and sequential tasks in a Markovian process. This enables us to analyze and calculate the efficient resources required to serve the tasks. The next step makes use of the proposed model to develop a preemptive scheduling algorithm for the processing jobs in order to improve resource utilization and its associated costs in the cloud computing system. Accordingly, a preemption-based resource allocation architecture is proposed to effectively and efficiently utilize the idle reserved resources for the processing jobs in the cloud paradigms. Then, performance metrics such as service time for the processing jobs are investigated. The accuracy of the proposed analytical model and scheduling analysis is verified through simulations and experimental results. The simulation and experimental results also shed light on the achievable QoS level for the preemptively allocated processing jobs. [ABSTRACT FROM AUTHOR]

Typically called big data processing, analyzing large volumes of data from geographically distributed regions with machine learning algorithms has emerged as an important analytical tool for governments and multinational corporations. The traditional wisdom calls for the collection of all the data across the world to a central data center location, to be processed using data-parallel applications. This is neither efficient nor practical as the volume of data grows exponentially. Rather than transferring data, we believe that computation tasks should be scheduled near the data, while data should be processed with a minimum amount of transfers across data centers. In this paper, we design and implement Flutter, a new task scheduling algorithm that reduces both the completion times and the network costs of big data processing jobs across geographically distributed data centers. To cater to the specific characteristics of data-parallel applications, in the case of optimizing the job completion times only, we first formulate our problem as a lexicographical min-max integer linear programming (ILP) problem, and then transform the ILP problem into a nonlinear program problem with a separable convex objective function and a totally unimodular constraint matrix, which can be further solved using a standard linear programming solver efficiently in an online fashion. In the case of improving both time- and cost- efficiency, we formulate the general problem as an ILP problem and we find out that solving an LP problem can achieve the same goal in the real practice. Our implementation of Flutter is based on Apache Spark, a modern framework popular for big data processing. Our experimental results have shown convincing evidence that Flutter can shorten both job completion times and network costs by a substantial margin. [ABSTRACT FROM AUTHOR]

The wide application of big data analysis technology cannot be separated from the support of big data platforms. Building big data platforms is an important demand of many enterprises and institutions. Building a big data platform requires sophisticated, system-wide technologies, and system performance and scalability should be considered especially. With the increasing volume of data, user needs continue to increase, and hence the scale of the planned data platform may not be able to meet the changing needs. Therefore, we design a hybrid big data platform that uses both bare metals and private cloud Virtual Machines (VM). This takes into account performance and scalability. Because bare metals generally outperform private cloud VMs, the big data platforms built on bare metals generally perform better than the big data platforms built on private cloud VMs. It is very convenient and quick to start the cloud servers in the private cloud, so the computing and storage nodes of the big data platform can be flexibly expanded to the private cloud so as to ensure that the big data platform can still have sufficient processing capacity during the peak period. We implemented this hybrid design in a production environment. Tests in the production environment also demonstrate the effectiveness of this design. [ABSTRACT FROM AUTHOR]

With the globalization of service, organizations continuously produce large volumes of data that need to be analysed over geo-dispersed locations. Traditionally central approach that moving all data to a single cluster is inefficient or infeasible due to the limitations such as the scarcity of wide-area bandwidth and the low latency requirement of data processing. Processing big data across geo-distributed datacenters continues to gain popularity in recent years. However, managing distributed MapReduce computations across geo-distributed datacenters poses a number of technical challenges: how to allocate data among a selection of geo-distributed datacenters to reduce the communication cost, how to determine the Virtual Machine (VM) provisioning strategy that offers high performance and low cost, and what criteria should be used to select a datacenter as the final reducer for big data analytics jobs. In this paper, these challenges is addressed by balancing bandwidth cost, storage cost, computing cost, migration cost, and latency cost, between the two MapReduce phases across datacenters. We formulate this complex cost optimization problem for data movement, resource provisioning and reducer selection into a joint stochastic integer nonlinear optimization problem by minimizing the five cost factors simultaneously. The Lyapunov framework is integrated into our study and an efficient online algorithm that is able to minimize the long-term time-averaged operation cost is further designed. Theoretical analysis shows that our online algorithm can provide a near optimum solution with a provable gap and can guarantee that the data processing can be completed within pre-defined bounded delays. Experiments on WorldCup98 web site trace validate the theoretical analysis results and demonstrate that our approach is close to the offline-optimum performance and superior to some representative approaches. [ABSTRACT FROM PUBLISHER]

Efficient response to search queries is very crucial for data analysts to obtain timely results from big data spanned over heterogeneous machines. Currently, a number of big-data processing frameworks are available in which search operations are performed in distributed and parallel manner. However, implementation of indexing mechanism results in noticeable reduction of overall query processing time. There is an urge to assess the feasibility and impact of indexing towards query execution performance. This paper investigates the performance of state-of-the-art clustered indexing approaches over Hadoop framework which is de facto standard for big data processing. Moreover, this study leverages a comparative analysis of nonclustered indexing overhead in terms of time and space taken by indexing process for varying volume data sets with increasing Index Hit Ratio. Furthermore, the experiments evaluate performance of search operations in terms of data access and retrieval time for queries that use indexes. We then validated the obtained results using Petri net mathematical modeling. We used multiple data sets in our experiments to manifest the impact of growing volume of data on indexing and data search and retrieval performance. The results and highlighted challenges favorably lead researchers towards improved implication of indexing mechanism in perspective of data retrieval from big data. Additionally, this study advocates selection of a non-clustered indexing solution so that optimized search performance over big data is obtained. [ABSTRACT FROM AUTHOR]

Efficient response to search queries is very crucial for data analysts to obtain timely results from big data spanned over heterogeneous machines. Currently, a number of big-data processing frameworks are available in which search operations are performed in distributed and parallel manner. However, implementation of indexing mechanism results in noticeable reduction of overall query processing time. There is an urge to assess the feasibility and impact of indexing towards query execution performance. This paper investigates the performance of state-of-the-art clustered indexing approaches over Hadoop framework which is de facto standard for big data processing. Moreover, this study leverages a comparative analysis of nonclustered indexing overhead in terms of time and space taken by indexing process for varying volume data sets with increasing Index Hit Ratio. Furthermore, the experiments evaluate performance of search operations in terms of data access and retrieval time for queries that use indexes. We then validated the obtained results using Petri net mathematical modeling. We used multiple data sets in our experiments to manifest the impact of growing volume of data on indexing and data search and retrieval performance. The results and highlighted challenges favorably lead researchers towards improved implication of indexing mechanism in perspective of data retrieval from big data. Additionally, this study advocates selection of a non-clustered indexing solution so that optimized search performance over big data is obtained. [ABSTRACT FROM AUTHOR]

Educational data mining is a research area of using data mining technology in education industry. In the research of EDM, data mining technology is used to modeling dataset samples in the field of education, which aims to study and forecast the testing data set with the help of effective statistical machine learning models. Machine learning models with distributed computing frameworks in the EDM can meet the needs of large-scale data processing meanwhile provide tailored data recommendation and then support decision-making in the future. Based on this background, this study first put all kinds of data models into the data training and predicting for simulation, propose an improved model to ameliorate the classification performance of the data model by adjusting the data model and by using an improved algorithm based on a new equation of information gain when calculating the optimal feature to split. Based on the best-performance data model in previous study combined with the application background of the "big data" era, we proposed a new random forest algorithm model focusing on giving classification to large-scale datasets based on distributed computing framework called MapReduce. By using the MapReduce, we design and realize a new system to meet this requirement. In this system, the model that has been trained can be serialized and deserialization between local disks and the distributed file system. [ABSTRACT FROM AUTHOR]

The virtual machine (VM) allocation problem in cloud computing has been widely studied in recent years, and many algorithms have been proposed in the literature. Most of them have been successfully applied to batch processing models such as MapReduce; however, none of them can be applied to streaming workflow well because of the following weaknesses: 1) failure to capture the characteristics of tasks in streaming workflow for the short life cycle of data streams; 2) most algorithms are based on the assumptions that the price of VMs and traffic among data centers (DCs) are static and fixed. In this paper, we propose a streaming workflow allocation algorithm that takes into consideration the characteristics of streaming work and the price diversity among geo-distributed DCs, to further achieve the goal of cost minimization for streaming big data processing. First, we construct an extended streaming workflow graph (ESWG) based on the task semantics of streaming workflow and the price diversity of geo-distributed DCs, and the streaming workflow allocation problem is formulated into mixed integer linear programming based on the ESWG. Second, we propose two heuristic algorithms to reduce the computational space based on task combination and DC combination in order to meet the strict latency requirement. Finally, our experimental results demonstrate significant performance gains with lower total cost and execution time. [ABSTRACT FROM PUBLISHER]

In the era of big data，both big data processing and high performance computing are of the urgent needs of a computer system. Specific to Spark big data processing and high performance computing based on GPU，this paper analyzes the Spark technology based on GPU proposed by industry. It is mainly by constructing heterogeneous parallel of CPU and GPU，making computer to obtain a powerful computing capability. Then we discuss the implementation of the Spark-GPU technology in laboratory environment，and expound the technical process of algorithm realization in detail. On this basis， we assess the performance of the Spark and Spark - GPU technology through simulation experiment. Results show Spark-GPU technology can achieve hundredfold speedup，hence，it can play an important role in promoting the development of image processing and information retrieval and other areas. [ABSTRACT FROM AUTHOR]

Compared with traditional applications, big data applications present have diffident characteristics, such as high parallelism, large volumes of data, irregular memory access mode, and poor temporal locality, which bring new challenges to traditional computer architectures. Graph500 which focuses on data intensive loads, is a rating of supercomputer systems. And the Bread First Search (BFS) algorithm is the core of the benchmark. Based on the parallel prototype system designed for big data applications, we implement the BFS algorithm from the perspectives of 1-D partitioning, optimized hybrid algorithm and convenient remote communication design. The performance achieves 803.8 MTEPS under the scale of an input graph of 222 vertexs and 226 edges. Then we analyze the experiment performance and provide a reference for future wok. [ABSTRACT FROM AUTHOR]

Nowadays, with an increasing expansion of the internet of things (IoT) that has created massive data streams, online attack detection via stream processing has become a matter of extensive attention. Current encryption and authentication methods cannot satisfy the security requirements of IoT critical infrastructures because of many heterogeneous connected devices, a wide network geographical scope, rapid software development, the possibility of security bugs, and the emergence of new attacks. Therefore, an extended method based on machine learning, which can process data stream to detect the fog layer's IoT attacks and prevent the spread of intrusion to other network segments, was proposed in this paper. The IFogLearn++ uses a fog layer to facilitate fast data stream processing in the fog layer. Using a fog layer helps the network tolerate more attacks due to the annexation of an extra security layer before the cloud layer. Based on the results, the IFogLearn++ has similar accuracy to competitors and 18 and 8 times faster performance compared to SVM and Learn++. This characteristic helps secure massive data streams in IoT. [ABSTRACT FROM AUTHOR]

This paper is devoted to a new method of using Microsoft (MS) Kinect sensors for non-contact monitoring of breathing and heart rate estimation to detect possible medical and neurological disorders. Video sequences of facial features and thorax movements are recorded by MS Kinect image, depth and infrared sensors to enable their time analysis in selected regions of interest. The proposed methodology includes the use of computational methods and functional transforms for data selection, as well as their denoising, spectral analysis and visualization, in order to determine specific biomedical features. The results that were obtained verify the correspondence between the evaluation of the breathing frequency that was obtained from the image and infrared data of the mouth area and from the thorax movement that was recorded by the depth sensor. Spectral analysis of the time evolution of the mouth area video frames was also used for heart rate estimation. Results estimated from the image and infrared data of the mouth area were compared with those obtained by contact measurements by Garmin sensors (www.garmin.com). The study proves that simple image and depth sensors can be used to efficiently record biomedical multidimensional data with sufficient accuracy to detect selected biomedical features using specific methods of computational intelligence. The achieved accuracy for non-contact detection of breathing rate was 0.26% and the accuracy of heart rate estimation was 1.47% for the infrared sensor. The following results show how video frames with depth data can be used to differentiate different kinds of breathing. The proposed method enables us to obtain and analyse data for diagnostic purposes in the home environment or during physical activities, enabling efficient human–machine interaction. [ABSTRACT FROM AUTHOR]

Big Data processing architectures are now widely recognized as one of the most significant innovations in Computing in the last decade. Their enormous potential in collecting and processing huge volumes of data scattered throughout the Internet is opening the door to a new generation of fully distributed applications that, by leveraging the large amount of resources available on the network will be able to cope with very complex problems achieving performances never seen before. However, the Internet is known to have severe scalability limitations in moving very large quantities of data, and such limitations introduce the challenge of making efficient use of the computing and storage resources available on the network, in order to enable data-intensive applications to be executed effectively in such a complex distributed environment. This implies resource scheduling decisions which drive the execution of task towards the data by taking network load and capacity into consideration to maximize data access performance and reduce queueing and processing delays as possible. Accordingly, this work presents a data-centric meta-scheduling scheme for fully distributed Big Data processing architectures based on clustering techniques whose goal is aggregating tasks around storage repositories and driven by a new concept of 'gravitational' attraction between the tasks and their data of interest. This scheme will benefit from heuristic criteria based on network awareness and advance resource reservation in order to suppress long delays in data transfer operations and result into an optimized use of data storage and runtime resources at the expense of a limited (polynomial) computational complexity. [ABSTRACT FROM AUTHOR]

The paper presents a new algorithm for processing big data streams, which mimics the surmised physical design in the brain. The algorithm is very suitable for extracting prevalent information items, even at rather low frequencies of about several percents. The developing data driven process can be used to effectually realize various types of large-scale computational intelligence operations. [ABSTRACT FROM AUTHOR]

• A high-level model for iterative and non-iterative large-scale data processing programs. • Formalizing data flow with Petri Nets and data processing operations with Monoid Algebra. • Application of the model to formalize mutation operators for mutation testing in data flow programs. • Proposal of new mutation operators for iterative data flow programs. This paper proposes a model for specifying data flow-based parallel data processing programs agnostic of target Big Data processing frameworks. The paper focuses on the formal abstract specification of non-iterative and iterative programs, generalizing the strategies adopted by data flow Big Data processing frameworks. The proposed model relies on Monoid Algebra and Petri Nets to abstract Big Data processing programs in two levels: a higher level representing the program data flow and a lower level representing data transformation operations (e.g., filtering, aggregation, join). We extend the model for data processing programs proposed in [1] , for modeling iterative data processing programs. The general specification of these programs implemented by data flow-based parallel programming models is essential given the democratization of iterative and greedy Big Data analytics algorithms. Indeed, these algorithms call for revisiting parallel programming models to express iterations. The paper gives a comparative analysis of the iteration strategies proposed by Apache Spark, DryadLINQ, Apache Beam, and Apache Flink. It discusses how the model achieves to generalize these strategies. [ABSTRACT FROM AUTHOR]

Spatial neighboring analysis is an indispensable part of geo-raster spatial analysis. In the big data era, high-resolution raster data offer us abundant and valuable information, and also bring enormous computational challenges to the existing focal statistics algorithms. Simply employing the in-memory computing framework Spark to serve such applications might incur performance issues due to its lack of native support for spatial data. In this article, we present a Spark-based parallel computing approach for the focal algorithms of neighboring analysis. This approach implements efficient manipulation of large amounts of terrain data through three steps: (1) partitioning a raster digital elevation model (DEM) file into multiple square tile files by adopting a tile-based multifile storing strategy suitable for the Hadoop Distributed File System (HDFS), (2) performing the quintessential slope algorithm on these tile files using a dynamic calculation window (DCW) computing strategy, and (3) writing back and merging the calculation results into a whole raster file. Experiments with the digital elevation data of Australia show that the proposed computing approach can effectively improve the parallel performance of focal statistics algorithms. The results also show that the approach has almost the same calculation accuracy as that of ArcGIS. The proposed approach also exhibits good scalability when the number of Spark executors in clusters is increased. [ABSTRACT FROM AUTHOR]

• We present two new unsupervised methods to classify remotely sensed data. • These methods perform classification process automatically. • The proposed methods allow to focus on specific classes in the image. • The performance of proposed method was compared with in four study areas. Unsupervised classification algorithms are methods for the analysis of remotely sensed images. Since these methods do not include a training phase, they require less time to apply and are more practical to use. Traditional unsupervised classification methods work with parameters given by the user, such as the number of classes, the stop criterion or the number of iterations of the algorithm. Determining the optimum values of these parameters to obtain successful classification result is a major problem. In this study, we propose two new methods, the weighted density based optimized classification method (DBOC-Weighted) and the automatic density based optimized classification method (DBOC-Automatic). Both work automatically without the need for parameters from the user, but the DBOC-Weighted only requires layer weights. These methods consist of data range expansion, useful data selection, segmentation and optimization stages, and perform the classification automatically. Both create new layers of data using remotely sensed images. After creating the initial classes based on density from all the data layers, the results are created by optimizing all classes in terms of quality indices. Four Sentinel 2 images are used to test the performance of the proposed methods. These images are selected from regions that have different geographical, climatic and vegetation properties. The results obtained are compared with the unsupervised classification methods frequently used in the literature. The accuracy analysis results show that the proposed classification algorithms produce satisfactory accuracy compared to the results of other algorithms. The results show that the proposed methods can be used successfully in the creation of expert and intelligent analysis systems, by eliminating user-induced error in the analysis of remotely sensed images. Thus, smart analysis tools can be created so that users from various professional disciplines can easily use them without being image processing specialists. [ABSTRACT FROM AUTHOR]

Computing shortest-path distances is a fundamental primitive in the context of graph data mining, since this kind of information is essential in a broad range of prominent applications, which include social network analysis, data routing, web search optimization, database design and route planning. Standard algorithms for shortest paths (e.g., Dijkstra's) do not scale well with the graph size, as they take more than a second or huge memory overheads to answer a single query on the distance for large-scale graph datasets. Hence, they are not suited to mine distances from big graphs, which are becoming the norm in most modern application contexts. Therefore, to achieve faster query answering, smarter and more scalable methods have been designed, the most effective of them based on precomputing and querying a compact representation of the transitive closure of the input graph, called the 2-hop-cover labeling. To use such approaches in realistic time-evolving scenarios, when the managed graph undergoes topological modifications over time, specific dynamic algorithms, carefully updating the labeling as the graph evolves, have been introduced. In fact, recomputing from scratch the 2-hop-cover structure every time the graph changes is not an option, as it induces unsustainable time overheads. While the state-of-the-art dynamic algorithm to update a 2-hop-cover labeling against incremental modifications (insertions of arcs/vertices, arc weights decreases) offers very fast update times, the only known solution for decremental modifications (deletions of arcs/vertices, arc weights increases) is still far from being considered practical, as it requires up to tens of seconds of processing per update in several prominent classes of real-world inputs, as experimentation shows. In this paper, we introduce a new dynamic algorithm to update 2-hop-cover labelings against decremental changes. We prove its correctness, formally analyze its worst-case performance, and assess its effectiveness through an experimental evaluation employing both real-world and synthetic inputs. Our results show that it improves, by up to several orders of magnitude, upon average update times of the only existing decremental algorithm, thus representing a step forward towards real-time distance mining in general, massive time-evolving graphs. [ABSTRACT FROM AUTHOR]

Complex networks are one of the main research fields in data mining. In this study, a penalised matrix decomposition-based community structure discovery algorithm (PMDCSDA) for complex networks is proposed. The complex network is firstly transformed into an adjacency matrix, which is then processed for dimension reduction via principal component analysis. Numerous clusters are produced on the basis of penalised matrix decomposition. To evaluate the performance of the proposed PMDCSDA, we compare it with several classical algorithms, such as K-means, CPM and GN, using three complex network datasets. Experimental results demonstrate that the proposed algorithm can achieve improved performance in precision, recall, F1 and Sep indicator. [ABSTRACT FROM AUTHOR]

With the development of online cars, the demand for travel prediction is increasing in order to reduce the information asymmetry between passengers and drivers of online car-hailing. This paper proposes a travel demand forecasting model named OC-CNN based on the convolutional neural network to forecast the travel demand. In order to make full use of the spatial characteristics of the travel demand distribution, this paper meshes the prediction area and creates a travel demand data set of the graphical structure to preserve its spatial properties. Taking advantage of the convolutional neural network in image feature extraction, the historical demand data of the first twenty-five minutes of the entire region are used as a model input to predict the travel demand for the next five minutes. In order to verify the performance of the proposed method, one-month data from online car-hailing of the Chengdu Fourth Ring Road are used. The results show that the model successfully extracts the spatiotemporal features of the data, and the prediction accuracies of the proposed method are superior to those of the representative methods, including the Bayesian Ridge Model, Linear Regression, Support Vector Regression, and Long Short-Term Memory networks. [ABSTRACT FROM AUTHOR]

Maritime transportation is an economic form of mass transportation, but it is associated with significant energy consumption and pollutant emissions. External forces such as tidal currents, waves, and wind strongly influence the energy efficiency of ships. The effective management of external forces can save energy and reduce emissions. This study presents a method to build an optimal speed adjustment plan for a ship to navigate a given route. The method takes a dynamic programming (DP)-based approach to finding such an optimal plan to utilize external forces. To estimate the speed changes caused by external forces, the proposed method uses the mapping information from a combined database of ship status, marine environmental conditions, and speed changes. For the efficient manipulation of externally forced speed-change information, we used MapReduce-based operations that can handle big data and support the easy retrieval of associated data in specific situations. To evaluate the applicability of the proposed method, we applied it to real navigation situations in the southwestern sea of the Korean Peninsula. In the simulation experiments, we used real automatic identification system data and marine environmental data. The proposed method built more efficient speed adjustment plans than the fixed-speed navigation in terms of energy savings and pollutant emission reduction. The results also showed that the speed adjustment exploits external forces in a beneficial manner. [ABSTRACT FROM AUTHOR]