Search

Flooding disaster causes huge impacts on the socio-economic world. In the inundated area, some geo-referenced images are shared through some media posts, which assist in providing alertness to the critical volunteers and managing the financial loss crisis. In this work, the Adaptive Billiards-Inspired Optimization (A-BIO) and Optimized Ensemble-learning-based detection (OED) with map reduce framework is proposed for flood disaster detection. Initially, the big data is gathered and processed for detection. During the map phase, data preprocessing is performed to enhance the performance of the data, which helps in removing the noise or unwanted attributes. Furthermore, the reduction phase can be done through weighted feature selection, where the features to be selected and the weight is optimized through A-BIO, which assists in getting the most significant features for improving the performance and reducing the complexity of the designed model. Finally, OED is performed by a set of classifiers like Convolutional Neural Networks, Adaboost, XGBoost, Long Short-Term Memory, and Deep Neural Networks, where the parameters of ensemble learning classifiers are optimized by A-BIO algorithm. Finally, through the performance analysis, this detection model can provide high accuracy and better detection performance to avoid the huge impacts of flood disasters. [ABSTRACT FROM AUTHOR]

Users are increasingly turning to big data processing systems to extract valuable information from massive datasets as the field of big data grows. Data analytics platforms are used by e-commerce enterprises to improve product suggestions and model business processes. In order to meet the needs of large-scale data center operation and maintenance management, Internet companies often use Flink to process log data. This paper takes the big data processing and analysis platforms built by Internet financial companies and large banks as examples, and implants a stock prediction model based on Deep Neural Network (DNN). In this context, this paper completes the following work: 1) The research status of big data processing and analysis platforms at home and abroad is introduced. 2) Drawing on the modular design idea, the commercial bank big data platform is designed and the functions of each sub-module are introduced. Then the basic principle and structure of Convolutional Neural Networks (CNN) are expounded. 3) The optimal parameters of Convolutional Neural Networks are selected through experiments, and then the trained model is used for experiments. It can be seen that the stock prediction model proposed in this article has a higher prediction accuracy compared to existing models, which also verifies the validity of the proposed model. Input the data and compare the obtained results with the actual results, and finally show that the model in this paper has a good performance on stock prediction.

Big Data technologies emerging day by day and are making drastic changes in various real-world applications. Traditional data mining tools adequate to process volumes of data but from past decades the rapid growth in data becomes difficult for processing. Due to continuous flow of data, data streams require additional computational processing than the traditional one. Big data stream processing considers different features of the data streams heterogeneity, scalability, fault tolerance and query optimization. Efficient implementation of these features in real-world applications using big data analytics is a challenging job during data storage, processing, and analysis phases. Therefore, the proposed model FRTSPS is a generic architecture which is influenced by popular big data processing Lambda architecture, based on distributed computing platform. The architecture using open-source platform Apache Flink for doing data processing. Flink is a popular platform for processing historical and stream data flows at once parallelly. Its stateful streaming can obtain more scalability and flexibility along with high throughput and low latency than the remaining stream processing programming models. [ABSTRACT FROM AUTHOR]

A successful deployment of Industry 5.0 is significantly dependent on the synergetic integration of several advanced technologies such as big data processing, Artificial Intelligence (AI) integration, and several effective digitization techniques that emphasize the uses of Robotics, Internet of Things (IoTs), Cloud Computing, etc. with active participations from human workers. Several researchers have explored the importance of big data processing in view of Industry 4.0 as it facilitated enhanced production at any smart manufacturing line by ensuring efficient process management, typically involving big data processing. Researchers presented several MapReduce-based data processing models at smart manufacturing lines to facilitate big data processing. Among several others, the Elementary Cellular Automata (ECAs)-based MapReduce model was introduced as an energy-efficient low-cost model for big data processing in view of the Industry 4.0 scenario. In the present research, an investigation is further proposed to explore the true potential (if any) for the ECAs-based MapReduce model with reference to available several MapReduce models, to migrate an existing big data processing model from Industry 4.0 into the future i.e., Industry 5.0. Investigation results as achieved from the comparison among several MapReduce models and further examinations about the inherent quality of shuffle in those MapReduce blocks, explore the inherent advantages of ECAs-based MapReduce model towards its choice for big data processing in Industry 5.0 scenario.

In the era of big data, using of data mining instead of data collection represents a new challenge for researchers and engineers. In the field of transportation, computing of the shortest path based on MapReduce using widely existing vehicle data is meaningful both in theory and practice. Therefore, this article proposes a simple shortest path approach to relieve urban traffic congestion. The objective is not to guarantee the optimality but to provide high-quality solutions in acceptable computational time. The proposed approach is based on partitioning of original graph into a set of subgraphs, and parallel solving of the shortest path for each subgraph in order to obtain a solution for the original graph. An iterative procedure is introduced to improve the accuracy. The experimental results show that proposed approach significantly reduces computational time. [ABSTRACT FROM AUTHOR]

In the dynamic realm of big data processing, conquering the challenges imposed by highdimensional datasets is imperative. This paper introduces a groundbreaking advancement in dimensionality reduction, employing Variational Auto-Encoder (VAE) within the Spark distributed framework. The deliberate selection of the "TLC" dataset, representative of New York City taxi trips with inherent high dimensionality, highlights the practicality of our approach. Our research showcases the virtuoso performance of VAE, achieving an impressive 95.12% reduction ratio and 89.26% accuracy. This highlights VAE's ability to elegantly distill essential information while discarding superfluous dimensions, achieving a harmonious balance between reduction and accuracy. Furthermore, building on the demonstrated superiority of Spark over Hadoop in prior successes, our adoption of VAE aligns with the overarching goal of enhancing big data processing. Spark's consistent advantage as a distributed framework reaffirms its reliability in handling diverse machine learning algorithms. This paper not only contributes to the advancement of machine learning in big data processing but also underscores the adaptability, versatility, and consistent performance of our approach across various methodologies and frameworks. The success of VAE in reducing dimensionality, coupled with Spark's inherent advantages, positions this research as a valuable contribution to the exploration of advanced techniques in distributed big data processing. [ABSTRACT FROM AUTHOR]

The arrival of the big data era makes the amount of data explosive growth, which puts forward new challenges and demands for computer network technology, and the integration of big data and network technology has become an important trend. This paper uses the optimization strategy and the elimination mechanism of the genetic algorithm to optimize the inertia weight and particle position speed updating mechanism of the particle swarm algorithm and combines the searching method of the Tennessee whisker algorithm with the sharing mechanism of the particle swarm algorithm to achieve the optimal data searching ability. Finally, the improved artificial intelligence algorithm and MapReduce are combined to improve the performance of the computer neural network algorithm in big data processing. The average data redundancy rate of this paper’s algorithm for big data processing is only 1.18%, and the resource integration checking rate always exceeds 85%, according to simulation experiments. In addition, the algorithm also shows good performance in practical applications, and it can achieve accurate classification of big data labels in big data label classification tasks while maintaining a low energy overhead. Meanwhile, it can accurately recognize electronic medical record data in large medical databases. Big data processing can benefit greatly from the proposed neural network algorithm in this paper.

With the arrival of the significant data era, efficiently processing large-scale multidimensional data has become challenging. As a powerful data dimensionality reduction tool, Principal Component Analysis (PCA) plays a vital role in big data processing, especially in information extraction and data simplification, showing unique advantages. The research aims to simplify the data processing process and improve the data processing efficiency by PCA method. The research method adopts the basic theory of PCA, the improvement of the weighted principal component analysis algorithm, and standardized and homogenized data processing techniques to process large-scale multidimensional data sets. The results show that the data dimensionality is significantly reduced after using PCA, for example, in the Analysis of the earnings quality of listed companies in the e-commerce industry, the cumulative variance contribution rate of the first four principal components extracted by PCA reaches 81.623%, which effectively removes the primary information of the original data. PCA not only reduces the complexity of the data, but also retains a large amount of crucial information, which is a significant application value for the processing of big data, especially in the fields of data compression and pattern recognition.

In the era of the explosion of digital content of large‐scale self‐media, user‐friendly social platforms such as Twitter and Facebook, provide opportunities for people to express their ideas and opinions freely. Due to lack of restrictions, hateful speech and its exposure can have profound psychological impacts on society. Current social networking platform is over‐reliant on the manual check, and it is labor‐intensive and time‐consuming. Although there are many machines learning methods for the detection of hate speech, short text with character limit on social platforms is more challenging for the detection of hate speech and offensive language. To address the problem of data sparsity, we have proposed a topic memory model for hate speech and offensive language detection (abbreviated as TM‐HOL). Potential topics are generated with our encoder and decoder to enrich short text features. Two memory matrices correspond to the topic words and the text, and the hate feature matrix is used to learn the syntactic features. It is demonstrated that our proposed method is effective on three datasets, performing better weighted‐F1. [ABSTRACT FROM AUTHOR]

Human life has become smarter by utilizing big data, telecommunication technologies, and wearable sensors over pervasive computing to give better healthcare services. Big data is built with the possibility to improve the healthcare industry. Big data makes the interconnection between patients, wearable sensors, healthcare caregivers, and providers through the utilization of Information and Communication Technology (ICT) and software. Most of the economic challenges in developing countries are caused by the healthcare sector, which occurs predominantly due to the increasing population requiring more quality of care concerning older people. Older people need great attention and care as they lead with irreparable damages when a minor accident or insignificant disease occurs. Therefore, the necessity of implementing new technologies and tools has arisen to support senior citizens regarding their healthcare. Various advancements in wireless technology, miniaturization, computing power, and processing made diverse healthcare innovations that led to developing the connected medical devices. Hence, this proposal develops a new healthcare monitoring system for tracking the activities of elderly people, where the Hadoop MapReduce technique for parallel processing the large-sized data. The data collected as mentioned in the available datasets is performed by using the numerous wearable sensors fixed on the “subject’s left ankle, right arm, and chest” that are transformed to the cloud platform and also to the data analytics layer according to the Internet of Medical Things (IoMT) devices. The given input undergoes data splitting to produce tiny chunks. These small chunks of the input files are then considered as Map tasks. Here, in the map phase, the features are optimally selected by the Hybrid Dingo Coyote Optimization (HDCO). The combiner phase classifies the physical activities using the developed Deep Ensemble Learning (DEL) consisting of classifiers such as “Extreme Learning Machine (ELM), deep Convolutional Neural Network (CNN), Long short-term memory (LSTM), Deep Belief Network (DBN), and Deep Neural Network (DNN)”. The parameter tuning in these classifiers is done by the same HDCO. The reducer phase extracts data from different chunks by merging the same classes. The developed HDCO-DEL has secured 13.66%, 16.01%, 17.33%, 13.6%, and 14.01% better accuracy than ELM, CNN, LSTM, DBN, DNN, and HealthFog, respectively on second dataset. The comparison with existing methods shows its better performance and also predicts physical activities with overall high accuracy.

The networks are fundamental to our modern world and they appear throughout science and society. Access to a massive amount of data presents a unique opportunity to the researcher's community. As networks grow in size the complexity increases and our ability to analyze them using the current state of the art is at severe risk of failing to keep pace. Therefore, this paper initiates a discussion on graph signal processing for large-scale data analysis. We first provide a comprehensive overview of core ideas in Graph signal processing (GSP) and their connection to conventional digital signal processing (DSP). We then summarize recent developments in developing basic GSP tools, including methods for graph filtering or graph learning, graph signal, graph Fourier transform(GFT), spectrum, graph frequency, etc. Graph filtering is a basic task that allows for isolating the contribution of individual frequencies and therefore enables the removal of noise. We then consider a graph filter as a model that helps to extend the application of GSP methods to large datasets. To show the suitability and the effeteness, we first created a noisy graph signal and then applied it to the filter. After several rounds of simulation results. We see that the filtered signal appears to be smoother and is closer to the original noise-free distance-based signal. By using this example application, we thoroughly demonstrated that graph filtration is efficient for big data analytics. [ABSTRACT FROM AUTHOR]

Resource scheduling is indispensable for enhancing the system performance during big data processing on clouds. It is highly useful for attaining significant utilization of computing resources completely concentrating towards the facilitation of resource scalability and on-demand services. The resources essential for running different applications is determined to be maximum heterogeneous in cloud computing. This heterogeneous resource demand introduces a resource gap in which some of the resource potentialities are drained on par with the other resource potentialities still available in the same server resulting in imbalanced resource utilization. This imbalanced resource allocation condition is more apparent when the computing resources are more heterogeneous. At this juncture, intelligent resource scheduling strategy becomes essential to distribute resources for big data processing by adopting a potential decision-making process that focusses on the objective of achieving necessitated tasks over time. In this paper, Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) algorithm-based efficient heterogeneous resource scheduling process is proposed for handling the challenges that are highly possible during big data processing in the Hadoop heterogenous cloud environment. This HGDGEO algorithm is proposed as an adaptive resource scheduling strategy that handles the dynamic characteristics of the resources and users' fluctuating demand during big data stream processing by mimicking the golden eagles' intelligence which alternates the speed of tuning at different spiral trajectory stages of hunting. It handles big data processing by adopting two adaptive parameters which completely concentrates on optimal resource allocation to suitable VMs in the shortest time possible depending on their requirements. The simulation results of HGDGEO algorithm confirmed its predominance in terms of makespan, load balance and throughput on par with the competitive resource scheduling algorithms. [ABSTRACT FROM AUTHOR]

Remote sensing satellites provide a vast amount of data to monitor and observe Earth’s surface and events on it. To use these data efficiently in subsequent analysis and decision-making, highly automated easy-to-use tools are needed. Here, we present Earth Observation Data Information Extractor (EODIE). EODIE is a toolkit to extract object-level time-series information from several multispectral satellite remote sensing platforms and to produce analysis-ready products for subsequent data analysis. EODIE has a modular design that makes it adjustable for end-user requirements. Users have a possibility to exchange and add modules in EODIE for flexible processing in different computing environments. With EODIE, remote sensing data can be processed to object level array, geotiff or statistics information of different (vegetation) indices or plain wavelength intervals.

This article is devoted to the use and further application of Python libraries in the medical industry. These libraries include NumPy, Pandas, Scikit-learn, Keras and TensorFlow, Matplotlib, Seaborn, and Plotly. On the example of the Keras library, the problem associated with the use of medical data analysis was considered.

This study uses a neural network to propose a methodology for creating digital bathymetric models for shallow water areas that are partially covered by a mix of hydroacoustic and photogrammetric data. A key challenge of this approach is the preparation of the training dataset from such data. Focusing on cases in which the training dataset covers only part of the measured depths, the approach employs generalized linear regression for data optimization followed by multilayer perceptron neural networks for bathymetric model creation. The research assessed the impact of data reduction, outlier elimination, and regression surface-based filtering on neural network learning. The average values of the root mean square (RMS) error were successively obtained for the studied nearshore, middle, and deep water areas, which were 0.12 m, 0.03 m, and 0.06 m, respectively; moreover, the values of the mean absolute error (MAE) were 0.11 m, 0.02 m, and 0.04 m, respectively. Following detailed quantitative and qualitative error analyses, the results indicate variable accuracy across different study areas. Nonetheless, the methodology demonstrated effectiveness in depth calculations for water bodies, although it faces challenges with respect to accuracy, especially in preserving nearshore values in shallow areas.

The study deals with detection of the occupation of Intelligent Building (IB) using data obtained from indirect methods with Big Data Analysis within IoT. In the area of daily living activity monitoring, one of the most challenging tasks is occupancy prediction, giving us information about people's mobility in the building. This task can be done via monitoring of CO2 as a reliable method, which has the ambition to predict the presence of the people in specific areas. In this paper, we propose a novel hybrid system, which is based on the Support Vector Machine (SVM) prediction of the CO2 waveform with the use of sensors that measure indoor/outdoor temperature and relative humidity. For each such prediction, we also record the gold standard CO2 signal to objectively compare and evaluate the quality of the proposed system. Unfortunately, this prediction is often linked with a presence of predicted signal activities in the form of glitches, often having an oscillating character, which inaccurately approximates the real CO2 signals. Thus, the difference between the gold standard and the prediction results from SVM is increasing. Therefore, we employed as the second part of the proposed system a smoothing procedure based on Wavelet transformation, which has ambitions to reduce inaccuracies in predicted signal via smoothing and increase the accuracy of the whole prediction system. The whole system is completed with an optimization procedure based on the Artificial Bee Colony (ABC) algorithm, which finally classifies the wavelet's response to recommend the most suitable wavelet settings to be used for data smoothing.

As the explosive growth of the data volume, data center is playing a critical role to store and process huge amount of data. Traditional single data center can no longer to adapt into incredibly fast-growing data. Recently, some researches have extended the tasks such data processing to geographically distributed data centers. However, since the joint consideration of task placement and data transfer, it is complex and difficult to design a proper scheduling approach with the goal of minimizing makespan under the constraint of task dependencies, processing capability and network, etc. Therefore, our work proposes $JHTD$ : an efficient joint scheduling framework based on hypergraph for task placement and data transfer across geographically distributed data centers. Generally, there are two crucial stages in $JHTD$ . Initially, due to the outstanding of hypergraphs in modeling complex problems, we have leveraged a hypergraph-based model to establish the relationship between tasks, data files, and data centers. Thereafter, a hypergraph-based partition method has been developed for task placement within the first stage. In the second stage, a task reallocation scheme has been devised in terms of each task-to-data dependency. Meanwhile, a data dependency aware transferring scheme has been designed to minimize the makespan. Last, the real-world model China-VO project has been used to conduct a variety of simulation experiments. The results have demonstrated that $JHTD$ effectively optimizes the problems of task placement and data transfer across geographically distributed data centers. $JHTD$ has been compared with three other state-of-the-art algorithms. The results have demonstrated that $JHTD$ can reduce the makespan by up to 20.6%. Also, various impacts (data transfer volume and load balancing) have been taken into account to show and discuss the effectiveness of $JHTD$ .

In this contribution two algorithms are developed to solve non-linear system of equations which can contain a large number of measurements. These algorithms are based on nonlinear block least-squares (BLS). Although block least squares was investigated by some researchers, the non-linear case was not examined by now. The first algorithm is proposed to solve a special case of non-linear problems that do not require linearization. Such an algorithm can be called total block least-squares. The second algorithm is based on linearization within a general nonlinear mixed model using a new notation which is in agreement with the rigorous linearization presented by Pope. Both of these algorithms can handle constraints on the parameters. By use of these algorithms, big data processing is feasible with inexpensive computers. Furthermore, expensive processors can solve systems with a large number of equations faster. Two case studies with more than 120,000 equations show that fast and accurate computations are possible by applying these algorithms without any loss of accuracy. [ABSTRACT FROM AUTHOR]

Big data and data science are the two most prominent contemporary developments. Big data analysis increases the requirement for innovative system designs, which promotes the creation of procedures that can manage massive data quantities while maintaining the ag ility, flexibility, and interactive feel that a data scientist requires. Big Data is the outcome of data being created at an exponential rate. This data is diverse and Includes unstructured, and semi-structured data types. It provides valuable information for many Sorts of stakeholders based on their needs, but it cannot be met using standard tools and procedures. Big data technologies play a critical role in handling, storing, and processing this massive quantity of data. It is further classified as text analytics, audio analytics, video analytics, and social media analytics. Big data analytics, when combined with big data analysis, has a major impact. [ABSTRACT FROM AUTHOR]

This paper develops a multi-dimensional Dynamic Time Warping (DTW) algorithm to identify varying lead-lag relationships between two different time series. Specifically, this manuscript contributes to the literature by improving upon the use towards lead-lag estimation. Our two-step procedure computes the multi-dimensional DTW alignment with the aid of shapeDTW and then utilises the output to extract the estimated time-varying lead-lag relationship between the original time series. Next, our extensive simulation study analyses the performance of the algorithm compared to the state-of-the-art methods Thermal Optimal Path (TOP), Symmetric Thermal Optimal Path (TOPS), Rolling Cross-Correlation (RCC), Dynamic Time Warping (DTW), and Derivative Dynamic Time Warping (DDTW). We observe a strong outperformance of the algorithm regarding efficiency, robustness, and feasibility. [ABSTRACT FROM AUTHOR]

Underwater data acquisition entities acquire big data that are processed aboard terrestrial data centres. However, processing the big data aboard terrestrial computing entities involves high latency data transfer. In addition, the processing of data in a terrestrial environment is challenging when there is inadequate edge node capacity. These challenges are addressed here. The paper proposes the heterogeneous edge computing paradigm to realize low latency transfer of increasing underwater big data. This is realized via the use of underwater computing entities instead of terrestrial computing entities for processing acquired big data. The proposed heterogeneous edge computing paradigm presents the multi-mode automated teller machine (ATM) as low cost terrestrial edge network entity. The multi-mode ATM is suitable when edge nodes have inadequate computing capacity. Performance evaluation shows that the use of underwater computing entities instead of terrestrial computing entities (existing work) enhances network performance and related capital costs. The number of hops, computing entity access latency and required autonomous underwater vehicle acquisition costs by an average of (5.3–88.4)%, 63.5% and (31.8–95.4)%, respectively. Evaluation shows that the use of the multi-mode ATM in the context of terrestrial cloud computing reduces the number of hops and latency by 44.4% and 37.3% on average, respectively. [ABSTRACT FROM AUTHOR]

In this era of massive knowledge, Hadoop, jointly of the foremost unremarkably used massive processing platforms, features a variety of parameters that are closely associated with resource utilization, particularly mainframe and memory. Calibration of these parameters through improvement may increase Hadoop's resource utilization. Manual calibration of those parameters is just about not possible, thanks to the time price. Within the massive knowledge business, there's a requirement to mechanically set up parameters and thereby maximize resource usage. The previous automatic calibration strategies take an extended time to realize the optimum configuration, reducing the cluster's overall performance. With the help of a novel perceptive procedure, each genetic programming and a genetic rule are supported with a view to enhancing the performance of a Hadoop MapReduce work, we propose an assistant in nursing the best configuration finder. We have a tendency to use the algorithms on top to search out the simplest values for parameter settings. Experiments were performed on four common applications, WordCount, TeraSort, Index and Grep, and eight virtual machines (VMs) in a very typical Hadoop cluster. Our projected methodology will increase MapReduce job potency by 53.63% for a one GB dataset and by 67.4% for a five GB dataset, in keeping with the findings, and by 73.68% on a 10 GB dataset; additionally, for a TeraSort programmed application, MapReduce job potency will increase by 52.62% for an one GB dataset, 61.2% for a five GB dataset, and 55.17% for a 10 GB dataset. MapReduce jobs boost the performance of Grep applications by 44.4% for a one GB dataset, 56.25% for a five GB dataset, and 49.44% for a 10 GB dataset. [ABSTRACT FROM AUTHOR]

Abstract Nowadays a large amount of data is originated by complex systems, such as social networks, transportation systems, computer and service networks. These systems can be modeled by using graphs and studied by exploiting graph metrics, such as betweenness centrality (BC), a popular metric to analyze node centrality of graphs. In spite of its great potential, this metric requires long computation time, especially for large graphs. In this paper, we present a very fast algorithm to compute BC of undirected graphs by exploiting clustering. The algorithm leverages structural properties of graphs to find classes of equivalent nodes: by selecting one representative node for each class, we are able to compute BC by significantly reducing the number of single-source shortest path explorations adopted by Brandes’ algorithm. We formally prove the graph properties that we exploit to define the algorithm and present an implementation based on Scala for both sequential and parallel map-reduce executions. The experimental evaluation of both versions, conducted with synthetic and real graphs, reveals that our solution largely outperforms Brandes’ algorithm and significantly improves known heuristics.

A data analytics application submitted to a Spark cluster often has to finish executing by a specified time target. To use cluster resources effectively, the key challenge is having the ability to gain quick insights on how the execution time of any given application is likely to be impacted by the resources allocated to the application, e.g., the number of Spark executor cores assigned, and the size of the data to be processed. Such insights can be used to quickly estimate the required resources and configure a Spark application for a desired execution time using the least amount of resources. Our paper proposes an automated execution time estimation approach called PERIDOT that involves executing a given application under a fixed resource setting with two different-sized, small subsets of its input data to offer fast, lightweight execution time predictions. It analyzes logs from these two executions to estimate the dependencies between internal stages of the application. Information on these dependencies combined with knowledge of Spark's data partitioning mechanisms is used to derive an analytic model that can estimate execution times for other resource settings and input data sizes. Our results from a wide range of applications and multiple Spark clusters show that PERIDOT can accurately estimate the execution time of an application from limited historical data, and suggest the minimum amount of resources required to closely meet an execution time target.

Big data systems are sufficiently stable to store and process a massive volume of rapidly changing data. However, big data systems are composed of large-scale hardware resources that make their subspecies easily fail. Fault tolerance is the main property of such systems because it maintains availability, reliability, and constant performance during faults. Achieving an efficient fault tolerance solution in a big data system is challenging because fault tolerance must meet some constraints related to the system performance and resource consumption. This study aims to provide a consistent understanding of fault tolerance in big data systems and highlights common challenges that hinder the improvement in fault tolerance efficiency. The fault tolerance solutions applied by previous studies intended to address the identified challenges are reviewed. The paper also presents a perceptive discussion of the findings derived from previous studies and proposes a list of future directions to address the fault tolerance challenges.

Summary: Swarm intelligence meta‐heuristic optimization algorithms for optimizing engineering applications have become increasingly popular. The whale optimization algorithm (WOA) is a recent and effective swarm intelligence optimization algorithm that mimics humpback whales' behaviors when optimizing a problem. Applying the algorithm to achieve optimal solutions has shown good results compared to most meta‐heuristic optimization algorithms. However, complex applications might require the processing of large‐scale computations, which results in down‐scaling computational throughput of WOA. Apache Spark, a well‐known parallel data processing framework, is the most recent distributed computing framework and has been proven to be the most efficient. In this article, we propose a WOA implementation on top of Apache Spark, represented as SBWOA, to enhance its computational performance while providing higher scalability of the algorithm for handling more complex problems. Compared with the recently reported MapReduce WOA (MR‐WOA), and serial implementation of WOA, our approach achieves significant enhancements with respect to computational performance for the highest population size with the maximum number of iterations. SBWOA successfully handles higher‐complexity problems which require complex computations. [ABSTRACT FROM AUTHOR]

Big data frameworks play a vital role in storing, processing, and analysing large datasets. Apache Spark has been established as one of the most popular big data engines for its efficiency and reliability. However, one of the significant problems of the Spark system is performance prediction. Spark has more than 150 configurable parameters, and configuration of so many parameters is challenging task when determining the suitable parameters for the system. In this paper, we proposed two distinct parallelisation models for performance prediction. Our insight is that each node in a Hadoop cluster can communicate with identical nodes, and a certain function of the non-parallelisable runtime can be estimated accordingly. Both models use simple equations that allows us to predict the runtime when the size of the job and the number of executables are known. The proposed models were evaluated based on five HiBench workloads, Kmeans, PageRank, Graph (NWeight), SVM, and WordCount. The workload’s empirical data were fitted with one of the two models meeting the accuracy requirements. Finally, the experimental findings show that the model can be a handy and helpful tool for scheduling and planning system deployment. [ABSTRACT FROM AUTHOR]

Abstract Background The emergence of high throughput technologies that produce vast amounts of genomic data, such as next-generation sequencing (NGS) is transforming biological research. The dramatic increase in the volume of data, the variety and continuous change of data processing tools, algorithms and databases make analysis the main bottleneck for scientific discovery. The processing of high throughput datasets typically involves many different computational programs, each of which performs a specific step in a pipeline. Given the wide range of applications and organizational infrastructures, there is a great need for highly parallel, flexible, portable, and reproducible data processing frameworks. Several platforms currently exist for the design and execution of complex pipelines. Unfortunately, current platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, workflow frameworks that provide a platform to develop and share portable pipelines have recently arisen. We complement these new platforms by providing a graphical user interface to create, maintain, and execute complex pipelines. Such a platform will simplify robust and reproducible workflow creation for non-technical users as well as provide a robust platform to maintain pipelines for large organizations. Results To simplify development, maintenance, and execution of complex pipelines we created DolphinNext. DolphinNext facilitates building and deployment of complex pipelines using a modular approach implemented in a graphical interface that relies on the powerful Nextflow workflow framework by providing 1. A drag and drop user interface that visualizes pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. Modules to execute and monitor pipelines in distributed computing environments such as high-performance clusters and/or cloud 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Modular process design with process revisioning support to increase reusability and pipeline development efficiency. 5. Pipeline sharing with GitHub and automated testing 6. Extensive reports with R-markdown and shiny support for interactive data visualization and analysis. Conclusion DolphinNext is a flexible, intuitive, web-based data processing and analysis platform that enables creating, deploying, sharing, and executing complex Nextflow pipelines with extensive revisioning and interactive reporting to enhance reproducible results.

The combination of powerful parallel frameworks and on-demand commodity hardware in distributed computing has made both analytics and decision support systems canonical to enterprises of all sizes. The unprecedented volumes of data stacked by companies present challenges to process analytical queries efficiently. This data is often organised as star schema, in which star join and group-by are ubiquitous and expensive operations. Although parallel frameworks such as Apache Spark facilitate join and group-by, the implementation can only process two tables at a time and fail to handle the excessive network communication, disk spills and multiple scans of data. In this paper, we present Distributed ATrie Group Join (DATGJ), a fast distributed star join and group-by algorithm for column-stores. DATGJ uses divide and broadcast-based joining technique where the fact table columns are partitioned equally and fast hash table (FHT) for each dimension table are broadcasted. This technique helps it avoid cross communication between workers and disk spills. DATGJ performs a single scan of partitioned fact table columns and use FHT to join data. FHT uses Robin Hood hashing with the upper limit on number of probes and achieve significant speed up during join. DATGJ performs group-by and aggregation leveraging progressive materialisation and realising grouping attributes as a tree shaped deterministic finite automation known as Aggregate Trie or ATrie. We evaluated our algorithm using Star Schema Benchmark (SSBM) to show that it is 1.5X to 6X faster than the most prominent approaches while having zero data shuffle and consistently perform well with addition of resources and in memory-constrained scenarios.

Particle swarm optimization (PSO) algorithms have low-quality initial particle swarm, which is generated by a random method when handling the problem of task scheduling in networked data centres. Such algorithms also fall easily into local optimum when searching for the optimal solution. To address these problems, this study proposes combining opposition-based learning (OBL) and tentative perception (TP) with PSO; the proposed method is called OBL-TP-PSO. This algorithm uses reverse learning to generate the initial population, such that the quality of the initial particle swarm can be improved. Before the particle speed and location are updated, the TP method is used to search for the individual optimum around each particle, thereby reducing the possibility of missing the potential optimal solution during the process of searching. In this manner, the problem in which the PSO algorithm easily falls into the local optimal is effectively solved. To evaluate the performance of the proposed algorithm, simulation experiments are performed on CloudSim toolkit. Experimental results show that in comparison with other algorithms (namely, Min-Min, Max-Min and PSO algorithm), the proposed OBL-TP-PSO algorithm has better performance in terms of the total execution time, load balancing and quality of service.

Today's scientific data analysis very often requires complex Data Analysis Workflows (DAWs) executed over distributed computational infrastructures, e.g., clusters. Much research effort is devoted to the tuning and performance optimization of specific workflows for specific clusters. However, an arguably even more important problem for accelerating research is the reduction of development, adaptation, and maintenance times of DAWs. We describe the design and setup of the Collaborative Research Center (CRC) 1404 "FONDA -– Foundations of Workflows for Large-Scale Scientific Data Analysis", in which roughly 50 researchers jointly investigate new technologies, algorithms, and models to increase the portability, adaptability, and dependability of DAWs executed over distributed infrastructures. We describe the motivation behind our project, explain its underlying core concepts, introduce FONDA's internal structure, and sketch our vision for the future of workflow-based scientific data analysis. We also describe some lessons learned during the "making of" a CRC in Computer Science with strong interdisciplinary components, with the aim to foster similar endeavors. [ABSTRACT FROM AUTHOR]

This paper develops a multi-dimensional Dynamic Time Warping (DTW) algorithm to identify varying lead-lag relationships between two different time series. Specifically, this manuscript contributes to the literature by improving upon the use towards lead-lag estimation. Our two-step procedure computes the multi-dimensional DTW alignment with the aid of shapeDTW and then utilises the output to extract the estimated time-varying lead-lag relationship between the original time series. Next, our extensive simulation study analyses the performance of the algorithm compared to the state-of-the-art methods Thermal Optimal Path (TOP), Symmetric Thermal Optimal Path (TOPS), Rolling Cross-Correlation (RCC), Dynamic Time Warping (DTW), and Derivative Dynamic Time Warping (DDTW). We observe a strong outperformance of the algorithm regarding efficiency, robustness, and feasibility.

• We apply NLP approaches in features choice for enhancing Classifier based on Random Forests approach. • We analyze medical records to retrieve features for clinical records classifications in smart environments. • We use a Machine Learning approach to build a fast multi-classification schema. • We apply the methodology to real case studies from health-care organizations in Italy. • We show accuracy of presented approach in terms of Accuracy-Rejection curves. The use of pervasive IoT devices in Smart Cities, have increased the Volume of data produced in many and many field. Interesting and very useful applications grow up in number in E-health domain, where smart devices are used in order to manage huge amount of data, in highly distributed environments, in order to provide smart services able to collect data to fill medical records of patients. The problem here is to gather data, to produce records and to analyze medical records depending on their contents. Since data gathering involve very different devices (not only wearable medical sensors, but also environmental smart devices, like weather, pollution and other sensors) it is very difficult to classify data depending their contents, in order to enable better management of patients. Data from smart devices couple with medical records written in natural language: we describe here an architecture that is able to determine best features for classification, depending on existent medical records. The architecture is based on pre-filtering phase based on Natural Language Processing, that is able to enhance Machine learning classification based on Random Forests. We carried on experiments on about 5000 medical records from real (anonymized) case studies from various health-care organizations in Italy. We show accuracy of the presented approach in terms of Accuracy-Rejection curves. [ABSTRACT FROM AUTHOR]

Nowadays a large amount of data is originated by complex systems, such as social networks, transportation systems, computer and service networks. These systems can be modeled by using graphs and studied by exploiting graph metrics, such as betweenness centrality (BC), a popular metric to analyze node centrality of graphs. In spite of its great potential, this metric requires long computation time, especially for large graphs. In this paper, we present a very fast algorithm to compute BC of undirected graphs by exploiting clustering. The algorithm leverages structural properties of graphs to find classes of equivalent nodes: by selecting one representative node for each class, we are able to compute BC by significantly reducing the number of single-source shortest path explorations adopted by Brandes' algorithm. We formally prove the graph properties that we exploit to define the algorithm and present an implementation based on Scala for both sequential and parallel map-reduce executions. The experimental evaluation of both versions, conducted with synthetic and real graphs, reveals that our solution largely outperforms Brandes' algorithm and significantly improves known heuristics. [ABSTRACT FROM AUTHOR]

Abstract In recent years, the rapid growth of the Internet has led to creation of massively large graphs. Since databases have become very large nowadays, they cannot be processed by a simple machine at an acceptable time anymore; therefore, traditional graph partitioning methods, which are often based on having a complete image of the entire graph, are not applicable to large datasets. This challenge has led to the appearance of a new approach called streaming graph partitioning. In streaming graph partitioning, a stream of input data is received by a partitioner, and partitioner decides which computational machine the data should be transferred to. Often, streaming partitioner does not have any information about the whole graph, and usually distributes the vertices based on some greedy heuristics which may not be optimal for incoming vertices. Hence, partitioner’s decision can be significantly improved if more information about the graph is utilized. In this paper, we present a new vertex-cut streaming graph partitioning approach. The proposed method uses the idea of postponing the decision for some of the edges (by means of an intelligent buffering) and corrects some of the past decisions to improve the quality of the graph partitioning. The proposed approach is evaluated using from real-world graphs. The experimental results show that the performance of the proposed method is superior in comparison with the previous HDRF method.

This article presents and analyses the modular architecture and capabilities of CODE-DE (Copernicus Data and Exploitation Platform – Deutschland, www.code-de.org), the integrated German operational environment for accessing and processing Copernicus data and products, as well as the methodology to establish and operate the system. Since March 2017, CODE-DE has been online with access to Sentinel-1 and Sentinel-2 data, to Sentinel-3 data shortly after this time, and since March 2019 with access to Sentinel-5P data. These products are available and accessed by 1,682 registered users as of March 2019. During this period 654,895 products were downloaded and a global catalogue was continuously updated, featuring a data volume of 814 TByte based on a rolling archive concept supported by a reload mechanism from a long-term archive. Since November 2017, the element for big data processing has been operational, where registered users can process and analyse data themselves specifically assisted by methods for value-added product generation. Utilizing 195,467 core and 696,406 memory hours, 982,948 products of different applications were fully automatically generated in the cloud environment and made available as of March 2019. Special features include an improved visualization of available Sentinel-2 products, which are presented within the catalogue client at full 10 m resolution.

Researchers and researched populations are actively involved in participatory epidemiology. Such studies collect many details about an individual. Recent developments in statistical inferences can lead to sensitive information leaks from seemingly insensitive data about individuals. Typical safeguarding mechanisms are vetted by ethics committees; however, the attack models are constantly evolving. Newly discovered threats, change in applicable laws or an individual's perception can raise concerns that affect the study. Addressing these concerns is imperative to maintain trust with the researched population. We are implementing Lohpi: an infrastructure for building accountability in data processing for participatory epidemiology. We address the challenge of data-ownership by allowing institutions to host data on their managed servers while being part of Lohpi. We update data access policies using gossips. We present Lohpi as a novel architecture for research data processing and evaluate the dissemination, overhead, and fault-tolerance.

Handling large data in geographically distributed information centers with resource and cost optimization is a key challenge. With several approaches being designed, handling a large volume of data in multiple datacenters in an inappropriate manner yet is considered to be a time-consuming process. To address these issues, a Multivariate Metaphor based Metaheuristic Glowworm Swarm Map-Reduce Optimization (MM-MGSMO) technique is presented. Here, with search space and large data volume as input for geo-distributed datacenters, glowworm (i.e. virtual machine) population is initialized. With each glowworm possessing a certain amount of luciferin (i.e. objective function), multiple objective functions (i.e. bandwidth, storage capacity, energy and computation cost) are defined for each virtual machine. Next, the glowworm position is updated according to the neighboring factor by means of probability. Followed by this, MapReduce function identifies the optimal virtual machine and accordingly allocation is performed, therefore improving data allocation efficiency. Besides, the workload is assigned across datacenters, reduction in computation cost and storage capacity is guaranteed. Experimental evaluation of MM-MGSMO approach with existing methods attained improved performances with factors such as data allocation efficiency, false-positive rate, storage capacity compared with other cutting edge technologies such as Joint optimization algorithm and Game theory-based dynamic resource allocation strategy. [ABSTRACT FROM AUTHOR]

The uneven distribution of classes in any dataset poses a tendency of biasness toward the majority class when analyzed using any standard classifier. The instances of the significant class being deficient in numbers are generally ignored and their correct classification which is of paramount interest is often overlooked in calculating overall accuracy. Therefore, the conventional machine learning approaches are rigorously refined to address this class imbalance problem. This challenge of imbalanced classes is more prevalent in big data scenario due to its high volume. This study deals with acknowledging a sampling solution based on cluster computing in handling class imbalance problems in the case of big data. The newly proposed approach hybrid sampling algorithm (HSA) is assessed using three popular classification algorithms namely, support vector machine, decision tree and k-nearest neighbor based on balanced accuracy and elapsed time. The results obtained from the experiment are considered promising with an efficiency gain of 42% in comparison to the traditional sampling solution synthetic minority oversampling technique (SMOTE). This work proves the effectiveness of the distribution and clustering principle in imbalanced big data scenarios. [ABSTRACT FROM AUTHOR]

Nowadays, healthcare applications necessitate maximum volume of medical data to be fed to help the physicians, academicians, pathologists, doctors and other healthcare professionals. Advancements in the domain of Wireless Sensor Networks (WSN) andMultimediaWireless Sensor Networks (MWSN) are tremendous. M-WMSN is an advanced form of conventional Wireless Sensor Networks (WSN) to networks that use multimedia devices. When compared with traditional WSN, the quantity of data transmission in M-WMSN is significantly high due to the presence of multimedia content. Hence, clustering techniques are deployed to achieve low amount of energy utilization. The current research work aims at introducing a new Density BasedClustering (DBC) technique to achieve energy efficiency inWMSN. The DBC technique is mainly employed for data collection in healthcare environment which primarily depends on three input parameters namely remaining energy level, distance, and node centrality. In addition, two static data collector points called Super Cluster Head (SCH) are placed, which collects the data from normal CHs and forwards it to the Base Station (BS) directly. SCH supports multi-hop data transmission that assists in effectively balancing the available energy. Adetailed simulation analysiswas conducted to showcase the superior performance of DBC technique and the results were examined under diverse aspects. The simulation outcomes concluded that the proposed DBC technique improved the network lifetime to a maximum of 16,500 rounds, which is significantly higher compared to existing methods. [ABSTRACT FROM AUTHOR]

This paper takes education data mining as the research theme, mine the existing massive education big data, compares the analysis methods of existing data models, and proposes an improved random forest reference model. The information gain of various features is calculated by introducing the feature weighting system, and the evaluation index is used to improve the existing data analysis. The simulation results show that the improved model is highly efficient as compared to the existing models for classification. In order to resolve the performance bottleneck of a single node in multiple data classification tasks in the era of big data, a classification and prediction model of graduates' large-scale employment data, based on distributed improved RF algorithm, is proposed. The MapReduce distributed computing framework is used to complete the serial writing and deserialization loading of the training model between the local disk and the distributed file system, and realizing the distributed expansion of the large-scale data classification model based on the improved RF model. [ABSTRACT FROM AUTHOR]

Sub-aquatic data processing is a procedure that exchanges datasets through underwater sensory devices in the distributed computing environment. This paradigm has evolved techniques of data exchange and signal processing over time and uses big data frameworks to store processed datasets at edge nodes. Also, it uses modern IoT devices that capture sensory data tuples of water temperature, turbidity, speed, and pressure levels. Recently, we observe that the edge nodes that acquire the dataset of heterogeneous IoT devices are becoming overwhelmed with the issue of tuple non-classification at the level of data encapsulation. This issue raises a few concerns such as (a) ineffective tuple wrapup, (b) bundle compression failovers, (c) bundle block placement latency, and (d) end-of-file replica build latency. This paper proposes a fine-grained processing framework that normalizes tuple non-classification through enhanced false-positive function and assembles IoT sensory tuples with the in-memory capacity to rectify compression failovers. This solution leads to a tremendous decrease in bundle block placement and end-of-file replica latencies. The simulation results depict the effectiveness of fine-grained processing framework through easing the edge nodes in the sub-aquatic distributed environment. [ABSTRACT FROM AUTHOR]

Cloud computing has emerged as a powerful paradigm for delivering data-intensive services over the Internet. Cloud computing has enabled the implementation and success of big data, a recent phenomenon handling huge data being generated from different sources. Competing clouds have made it challenging to select a cloud provider that guarantees quality of cloud service (QoCS). Also, cloud providers' claims of guaranteeing QoCS are exaggerated for marketing purposes; hence, they cannot often be trusted. Therefore, a comprehensive trust model is necessary to evaluate the QoCS prior to making any selection decision. In this paper, we propose a multi-dimensional trust model for big data workflow processing over different clouds. It evaluates the trustworthiness of cloud providers based on: the most up-to-date cloud resource capabilities, the reputation evidence measured by neighboring users, and a recorded personal history of experiences with the cloud provider. The ultimate goal is to ensure an efficient selection of trustworthiness cloud provider who eventually will guarantee high QoCS and fulfills key big data workflow requirements. Various experiments were conducted to validate our proposed model. The results show that our model captures the different components of trust, ensures high QoCS, and effectively adapts to the dynamic nature of the cloud.

In a world where every day we produce 2.5 quintillion bytes of data, sentiment analysis has been a key for making sense of that data. However, to process huge text data in real-time requires building a data processing pipeline in order to minimize the latency to process data streams. In this paper, we explain and evaluate our proposed real-time customer' sentiment analysis pipeline on the Moroccan banking sector through data from the web and social network using open-source big data tools such as data ingestion using Apache Kafka, In-memory data processing using Apache Spark, Apache HBase for storing tweets and the satisfaction indicator, and ElasticSearch and Kibana for visualization then NodeJS for building a web application. The performance evaluation of Naïve Bayesian model show that for French Tweets the accuracy has reached 76.19% while for English Tweets the result was unsatisfactory and the resulting accuracy is 56%. To remedy this problem, we used the Stanford core NLP which, for English Tweets, reaches a precision of 80.7%. [ABSTRACT FROM AUTHOR]

Background: The emergence of high throughput technologies that produce vast amounts of genomic data, such as next-generation sequencing (NGS) is transforming biological research. The dramatic increase in the volume of data, the variety and continuous change of data processing tools, algorithms and databases make analysis the main bottleneck for scientific discovery. The processing of high throughput datasets typically involves many different computational programs, each of which performs a specific step in a pipeline. Given the wide range of applications and organizational infrastructures, there is a great need for highly parallel, flexible, portable, and reproducible data processing frameworks. Several platforms currently exist for the design and execution of complex pipelines. Unfortunately, current platforms lack the necessary combination of parallelism, portability, flexibility and/or reproducibility that are required by the current research environment. To address these shortcomings, workflow frameworks that provide a platform to develop and share portable pipelines have recently arisen. We complement these new platforms by providing a graphical user interface to create, maintain, and execute complex pipelines. Such a platform will simplify robust and reproducible workflow creation for non-technical users as well as provide a robust platform to maintain pipelines for large organizations. Results: To simplify development, maintenance, and execution of complex pipelines we created DolphinNext. DolphinNext facilitates building and deployment of complex pipelines using a modular approach implemented in a graphical interface that relies on the powerful Nextflow workflow framework by providing 1. A drag and drop user interface that visualizes pipelines and allows users to create pipelines without familiarity in underlying programming languages. 2. Modules to execute and monitor pipelines in distributed computing environments such as high-performance clusters and/or cloud 3. Reproducible pipelines with version tracking and stand-alone versions that can be run independently. 4. Modular process design with process revisioning support to increase reusability and pipeline development efficiency. 5. Pipeline sharing with GitHub and automated testing 6. Extensive reports with R-markdown and shiny support for interactive data visualization and analysis. Conclusion: DolphinNext is a flexible, intuitive, web-based data processing and analysis platform that enables creating, deploying, sharing, and executing complex Nextflow pipelines with extensive revisioning and interactive reporting to enhance reproducible results. [ABSTRACT FROM AUTHOR]

This paper presents a super-resolution (SR) technique for enhancement of infrared (IR) images. The suggested technique relies on the image acquisition model, which benefits from the sparse representations of low-resolution (LR) and high-resolution (HR) patches of the IR images. It uses bicubic interpolation and minimum mean square error (MMSE) estimation in the prediction of the HR image with a scheme that can be interpreted as a feed-forward neural network. The suggested algorithm to overcome the problem of having only LR images due to hardware limitations is represented with a big data processing model. The performance of the suggested technique is compared with that of the standard regularized image interpolation technique as well as an adaptive block-by-block least-squares (LS) interpolation technique from the peak signal-to-noise ratio (PSNR) perspective. Numerical results reveal the superiority of the proposed SR technique. [ABSTRACT FROM AUTHOR]