Search

High utility itemset (HUI) mining is a powerful data mining technique to discover profitable patterns. The utility of an item is computed by using two measures named quantity and per-unit profit. All existing HUI mining algorithms consider a single value of external utility (per unit profit) for the entire database. However, the per-unit profit of items might fluctuate over time in many applications. This research introduces three novel strategies to comprise the external utilities of items as input for the HUI mining algorithm. Traditional HUI mining algorithms have been developed for the standalone system and do not fit for big data processing due to the limited computing resources (CPU, memory). Big data are efficiently processed on distributed frameworks like Apache Hadoop, Spark, etc. This paper introduces a distributed HUI mining algorithm named Spark-based Top-k high utility itemset (k-SHUI) miner. We also propose a fair load distribution strategy to divide the search space equally among the cluster nodes. The k-SHUI produces top-k HUIs without the requirement of the minimum utility threshold. We conducted extensive experiments on six real-life datasets to compare the proposed algorithm's performance with the existing algorithm. The experimental results demonstrate that the proposed algorithm outperforms the existing algorithms.

One of the most challenging aspects of using MapReduce to parallelize and distribute large-scale data processingis detecting straggler tasks. Identifying ongoing tasks on weak nodes is how it’s described. The total computation time isthe sum of the execution times of the two stages in the Map process (copy, combine) and the three stages in the Reducephase (shuffle, sort, and reduce). The main aim of this paper is to estimate the accurate execution time in each location. Theproposed approach uses a backpropagation neural network on Hadoop to detect straggler tasks and calculate the remainingtask execution time, which is crucial in straggler task identification. The comparative analysis is done with some efficientmodels in this domain, such as LATE, ESAMR, and the real remaining time for WordCount and Sort benchmarks. It wasfound that the proposed model is capable of detecting straggler tasks in accurately estimating execution time. It also helpsin reducing the execution time that it takes to complete a task.

The paper addresses the development of technology for controlling access to digital portals and platforms based on assessments of personal characteristics of user behavior built into the interface. In distributed digital platforms and portals using personal data, big data is collected and processed using specialized applications using computer networks. In accordance with the law, the data is stored on internal corporate servers and data centers. Special attention is paid to the tasks of differentiation and control of access in modern information systems. Wide availability and mass scale of services should be accompanied by more careful control and user verification. Access control to such systems cannot be ensured only through technologies and information security tools; efficiency can be increased through software and hardware architectural solutions. The paper proposes to expand the currently developing SIEM technology (Security information and event management), which combines the concept of security event management and information security management, with blocks of user behavior analysis. As a characteristic that can be measured without overloading communication channels and is independent of the type of device used, the psychomotor reaction time is proposed, measured as the performance of actions with the interface. A technological solution has been developed for implementation in a wide range of digital platforms: banking, medical, educational, etc. The results of experimental research using a digital platform of mass psychological research are presented. For the research, data from a mass survey were used when answering (in the form of a choice from the available options) to questions about the level of education. Analysis of the reaction time data showed the possibility of standardization and the same indicators of specific users when answering different questions.

A distributed big data network is the integration of big data and the underlying distributed network. This emerging paradigm brings the potential to divide big data processing tasks into smaller ones so that they can be intelligently processed in parallel with machine learning based on distributed network resources. Such a pattern requires strict system integrity, especially machine learning integrity against data tampering or network control by malicious nodes. In this article, we propose a secure architecture consisting of one HaSi scheme and two data tampering detection schemes for protecting the machine learning integrity in distributed big data networking. Illustrative results demonstrate the effect of our proposed schemes, and show that they can ensure the learning accuracy even when 30-40 percent of processing nodes are maliciously controlled. When the figure raises to 40-50 percent, the accuracy of our proposed schemes begins to fall visibly, but still outperforms the scenario without protection by up to 70-80 percent.

In this paper, the technical aspects of big data processing of multimedia communication are deeply studied. Firstly, we propose a cloud implementation method of radial basis function neural network which is based on Map-Reduce on cloud computing cluster. Secondly, in order to meet the needs of big data processing of multimedia communication, the map-Reduce-based error back-propagation algorithm is trained to effectively map the effective mapping mechanism of multi-layer neural networks. For a cloud algorithm on a cloud computing cluster and a serial algorithm on a single processor, the time required to implement the algorithm is theoretically derived, and the cloud algorithm and performance parameters (acceleration ratio) on the cloud cluster are evaluated, the optimal number and minimum number of data nodes). Finally, the experimental results show that compared with the existing algorithms, the cloud algorithm proposed in this paper has better acceleration speed, faster convergence speed and fewer iterations.

Cloud computing has become an important alternative for solving big data processing. Nowadays, cloud service providers usually offer users a virtual machine with various combinations of prices. As each user has different circumstances, the problem of choosing the cost-minimized combination under a deadline constraint as well as user's preference is becoming more complex. This article is concerned with the investigation of adapting a user's preference policies for scheduling real-time divisible loads in a cloud computing environment. The workload allocation approach used in this research is using Divisible Load Theory. The proposed algorithm aggregates resources into groups and optimally distributes the fractions of load to the available resources according to user's preference. The proposed algorithm was evaluated by simulation experiments and compared with the baseline approach. The result obtained from the proposed algorithm reveals that a significant reduction in computation cost can be attained when the user's preferences are low priority.

In contrast to HPC clusters, when big data is processing in a distributed, particularly dynamic and opportunistic environment, the overall performance must be impaired and even bottlenecked by the dynamics of overlay and the opportunism of computing nodes. The dynamics and opportunism are caused by churn and unreliability of a generic distributed environment, and they cannot be ignored or avoided. Understanding impact factors, their impact strength and the relevance between these impacts is the foundation of potential optimization. This paper derives the research background, methodology and results by reasoning the necessity of distributed environments for big data processing, scrutinizing the dynamics and opportunism of distributed environments, classifying impact factors, proposing evaluation metrics and carrying out a series of intensive experiments. The result analysis of this paper provides important insights to the impact strength of the factors and the relevance of impact across the factors. The production of the results aims at paving a way to future optimization or avoidance of potential bottlenecks for big data processing in distributed environments.

The article considers the concept and features of Big Data management systems and their differences from traditional DBMS’s. The authors describe changes in the intruder model and new vulnerabilities in data management systems. A new threat model is developed. The article presents new problems of information security in a distributed Big Data processing and storage system.

This article considers the solution to the problem of building secure Big Data management systems using a consistent approach. The concept and features of Big Data management systems and their differences from traditional DBMS’s are presented. The principles of a new, consistent approach to building secure Big Data management systems are given and substantiated. The security subsystem architecture is proposed.

Big data frameworks play a vital role in storing, processing, and analysing large datasets. Apache Spark has been established as one of the most popular big data engines for its efficiency and reliability. However, one of the significant problems of the Spark system is performance prediction. Spark has more than 150 configurable parameters, and configuration of so many parameters is challenging task when determining the suitable parameters for the system. In this paper, we proposed two distinct parallelisation models for performance prediction. Our insight is that each node in a Hadoop cluster can communicate with identical nodes, and a certain function of the non-parallelisable runtime can be estimated accordingly. Both models use simple equations that allows us to predict the runtime when the size of the job and the number of executables are known. The proposed models were evaluated based on five HiBench workloads, Kmeans, PageRank, Graph (NWeight), SVM, and WordCount. The workload’s empirical data were fitted with one of the two models meeting the accuracy requirements. Finally, the experimental findings show that the model can be a handy and helpful tool for scheduling and planning system deployment.

Deep learning has made extraordinary progress in the last few years, focusing on improving the accuracy and speed of standard deep learning benchmarks. Nevertheless, datasets in production environments are often messy, which makes data cleaning crucial for DNN model training and inference. Existing solutions that combine big data processing systems and deep learning systems to accomplish the data cleaning, DNN model training and inference are internally tied to one of Spark or Flink. However, Spark and Flink usually show different performance under batch and stream processing workloads. In order to employ Spark in batch training and Flink in streaming inference, existing solutions incur the burden of maintaining two data cleaning programs. In this demonstration, we showcase Beamer: an end-to-end deep learning framework for unifying the data cleaning program when employing Spark in training and Flink in inference, respectively.

This article presents the implementation details related to the distributed storage and processing of big datasets in fog computing cluster environment. The implementation details of fog computing framework using Apache Spark for big data applications in a resource-constrained environment are given. The results related to Big Data processing, modeling, and prediction in a resource-constraint fog computing framework are presented by considering the evaluation of case studies using the e-commerce customer dataset and bank loan credit risk datasets.

Machine learning models can detect hidden trends and patterns in datasets, leading to valuable insights and data-driven decision-makers to improve general performance. However, it is also possible to scale up the machine learning algorithms by paralyzing such algorithms and using big data processing synchronically. The validation dataset is usually 20-25% of the original dataset, while the original dataset is considered as a massive dataset it would be 1% of the original size. The measurement or criteria for calculating error in classification and regression algorithms are discussed in the following. However, in some literature, the validation dataset and testing dataset are considered the same, and the original dataset is not divided into three phases to build a machine learning model. In this chapter, the most common supervised learning algorithms, classification, and regression are introduced.

In today's telecom industry, fraud detection is a significant research problem. As part of this research, we propose a methodology to detect fraud cases using machine learning models with big data processing and analysis platforms based on customer data in telecommunications. The prototype implementation of the proposed methodology has been designed, developed, and applied to the usage data set of telecommunications company subscribers. We perform performance tests on the developed prototype application to understand how successful the proposed methodology is and its scalability. The obtained results demonstrate the usability of the proposed method in the telecommunications sector.

The paper presents data mining as a suitable tool for analyzing data from industrial processes. The data mining methods offer a wide range of uses in the current age of digitalization, big data processing and analysis. Apart from discovering patterns and detecting relationship between individual characteristics, assuring quality of products, prediction and optimization of process performance, data mining techniques also contribute to the transition from a reactive to a predictive approach in problem solving. The first part of the paper presents the possibilities of utilization of data mining methods and techniques to analyze data from industrial processes. The second part of the paper deals with a selection of proper data mining method and its practical application on data from manufacturing industry.

New information - modeling instrumental approach is proposed to realize the diagnostics of hydrological and hydrochemical systems and processes. Decision making procedure is based on the big data processing algorithms. Real environmental problems are considered as the examples of the using this approach.

When clustering efficient distributed database, the conventional algorithm has long time cost and low clustering accuracy. To solve the above problems, an efficient distributed database clustering algorithm for big data processing is designed. Calculating the eigenvalues of the database, and linking the efficient distributed database with similar characteristics. The cross correlation matrix is used to ensure the consistency of cluster label. To improve the performance of K-means algorithm, input the database to be clustered, output $k$ clustering centers, and divide the clustering groups. Mapping database to clustering center, clustering low dimensional big data. Experimental results show that the proposed algorithm can reduce the running time and mean square error of data clustering, and improve the efficiency and accuracy of clustering.

The uneven distribution of classes in any dataset poses a tendency of biasness toward the majority class when analyzed using any standard classifier. The instances of the significant class being deficient in numbers are generally ignored and their correct classification which is of paramount interest is often overlooked in calculating overall accuracy. Therefore, the conventional machine learning approaches are rigorously refined to address this class imbalance problem. This challenge of imbalanced classes is more prevalent in big data scenario due to its high volume. This study deals with acknowledging a sampling solution based on cluster computing in handling class imbalance problems in the case of big data. The newly proposed approach hybrid sampling algorithm (HSA) is assessed using three popular classification algorithms namely, support vector machine, decision tree and k-nearest neighbor based on balanced accuracy and elapsed time. The results obtained from the experiment are considered promising with an efficiency gain of 42% in comparison to the traditional sampling solution synthetic minority oversampling technique (SMOTE). This work proves the effectiveness of the distribution and clustering principle in imbalanced big data scenarios.

As big data processing is being widely adopted by many domains, massive amount of generated data become more reliant on the parallel computing platforms for analysis, wherein Spark is one of the most widely used frameworks. Spark’s abnormal tasks may cause significant performance degradation, and it is extremely challenging to detect and diagnose the root causes. To that end, we propose an innovative tool, named LADRA, for log-based abnormal tasks detection and root-cause analysis using Spark logs. In LADRA, a log parser first converts raw log files into structured data and extracts features. Then, a detection method is proposed to detect where and when abnormal tasks happen. In order to analyze root causes we further extract pre-defined factors based on these features. Finally, we leverage General Regression Neural Network (GRNN) to identify root causes for abnormal tasks. The likelihood of reported root causes are presented to users according to the weighted factors by GRNN. LADRA is an off-line tool that can accurately analyze abnormality without extra monitoring overhead. Four potential root causes, i.e., CPU, memory, network, and disk I/O, are considered. We have tested LADRA atop of three Spark benchmarks by injecting aforementioned root causes. Experimental results show that our proposed approach is more accurate in the root cause analysis than other existing methods.

Bioinformatics computing is a kind of big data processing problem, which usually has the characteristics of large data scale, large computational load and long computational time. Therefore, the use of big data technology in bioinformatics computing has gradually become a research hotspot, and using Hadoop for gene sequence alignment is one of it. It is a common way to use various tools to complete a job in the field of Biocomputing. In most studies of parallel alignment of gene sequences using Hadoop, third-party tools are also needed. However, there are few methods using Hadoop independently to complete gene sequences alignment. Adding data processing with other tools to Hadoop workflow not only affects the improvement of computing performance, but also complicates the application. In this paper, a parallel alignment model of gene sequences based on multiple inputs and outputs is proposed, which can independently complete parallel alignment of gene sequences in Hadoop platform without using other tools. This model not only simplifies the process flow of gene sequence alignment, but also improves the performance compared with other methods. This paper describes in detail the method of manipulating gene sequences with multiple inputs and outputs modes on Hadoop platform and the design of a computing model based on this method, and proves the superiority of this model through experiments.

The paper offers a new approach to improving the performance of the materials knowledge analysis based on Big Data processing and machine learning. We consider a framework in which thread functioning of five machine learning mechanisms intended for solving the classification problem is realized. Classifier operation results are exposed to majority voting. The experimental assessment of performance and accuracy of framework operation is made on the data set containing technological data of the production line. Assessment showed that the offered framework provides a scoring on productivity of materials knowledge processing by 7.4 times.

The analysis of Big data technologies was provided. An example of MapReduce paradigm application, uploading of big volumes of data, processing and analyzing of unstructured information and its distribution into the clustered database was provided. The article summarizes the concept of "big data". Examples of methods for working with arrays of unstructured data are given. The parallel system Resilient Distributed Datasets (RDD) is organized. The class of basic database operations was realized: database con-nection, table creation, getting in line id, returning all elements of the database, update, delete and create the line.

Recently, many researches on positioning technology using LBS (Location-Based Services) have been conducted with the development of wearable devices. In addition, the data used in these devices is helping to perform LBS using Big data technology. And the existing method of finding a specific location is not suitable for collecting and processing all the data [1] [2] [3] [4]. Therefore, in order to process all streaming data in real time, it is necessary to use Big data processing technology. In this paper, we use NoSQL technology to solve this problem. We propose a framework for improving the performance of LBS using NoSQL.

Apache Spark is a framework providing fast computations on Big Data using MapReduce model. With cloud environments Big Data processing becomes more flexible since they allow to create virtual clusters on-demand. One of the most powerful open-source cloud environments is Openstack. The main goal of this project is to provide an ability to create virtual clusters with Apache Spark and other Big Data tools in Openstack. There exist three approaches to do it. The first one is to use Openstack REST APIs to create instances and then deploy the environment. This approach is used by Apache Spark core team to create clusters in propriatary Amazon EC2 cloud. Almost the same method has been implemented for Openstack environments. Although since Openstack API changes frequently this solution is deprecated since Kilo release. The second approach is to integrate virtual clusters creation as a built-in service for Openstack. ISP RAS has provided several patches implementing universal Spark Job engine for Openstack Sahara and Openstack Swift integration with Apache Spark as a drop-in replacement for Apache Hadoop. This approach allows to use Spark clusters as a service in PaaS service model. Since Openstack releases are less frequent than Apache Spark this approach may be not convenient for developers using the latest releases. The third solution implemented uses Ansible for orchestration purposes. We implement the solution in loosely coupled way and provide an ability to add any auxiliary tool or even to use another cloud environment. Also, it provides an ability to choose any Apache Spark and Apache Hadoop versions to deploy in virtual clusters. All the listed approaches are available under Apache 2.0 license.

In the present situation, the healthcare sector is one of the leading areas in which technologies and data are easily improved. The vast volume of medical data is impossible to manage. Big data processing makes it easy to manage these data. There are several medical methods worldwide for many diseases. Machine learning is an emerging solution to disease detection and prediction. This document explains the diagnosis of disease through machine learning based on symptoms. Algorithms such as Naive Bayes, Decision Tree and the Random Forest are used in a given data collection and estimate of disease machines. The python programming language is used to execute it. The analysis demonstrates the proper detailed algorithm. By the data set the accuracy of an algorithm is determined.

This project aims to implement a tool that allows applications following the map-reduce principle and the AES encryption algorithm. The map-reduce policy is used in big data processing for speeding up the treatment of large data sets (large files). AES encryption algorithm is used to secure the data that is transmitted over the network. This principle is to split a large file into blocks (files) that are distributed on several nodes. An application (we call it Map) can be executed on each of these blocks in parallel. All the results (files) from these computations are then gathered on the original node where another application (we call it Reduce) is executed to compute the final result (one file). The typical application we will use is the word count application. This application counts the occurrences of each word in a big file. Each Map counts the presence of words in each block, producing a table . Then the Reduce aggregates the created tables to obtain the final result (a table )

The use of pervasive IoT devices in Smart Cities, have increased the Volume of data produced in many and many field. Interesting and very useful applications grow up in number in E-health domain, where smart devices are used in order to manage huge amount of data, in highly distributed environments, in order to provide smart services able to collect data to fill medical records of patients. The problem here is to gather data, to produce records and to analyze medical records depending on their contents. Since data gathering involve very different devices (not only wearable medical sensors, but also environmental smart devices, like weather, pollution and other sensors) it is very difficult to classify data depending their contents, in order to enable better management of patients. Data from smart devices couple with medical records written in natural language: we describe here an architecture that is able to determine best features for classification, depending on existent medical records. The architecture is based on pre-filtering phase based on Natural Language Processing, that is able to enhance Machine learning classification based on Random Forests. We carried on experiments on about 5000 medical records from real (anonymized) case studies from various health-care organizations in Italy. We show accuracy of the presented approach in terms of Accuracy-Rejection curves.

Measuring the traffic parameters may be considered as one of the key operations in traffic management. With the contemporary monitoring systems, this issue transformed from the problem of gathering data to the issue of big data processing. In this chapter, we propose the methodological approach to the processing and analysing of the real-time flow of information from the traffic toll system. The core classes library developed in Python programming language implements the methodology of data processing. The results of the data analysis from the viaTOLL system (Poland) illustrate the use of the proposed approach.

Triboelectric nanogenerator based sensor has excellent material compatibility, low cost, and flexibility, which is a unique candidate technology for artificial intelligence. Triboelectric nanogenerators effectively provide critical infrastructure for new generation of sensing systems that collect information by large amounts of self-powered sensors. This review mainly discusses capability and prospect of triboelectric nanogenerators being applied to intelligent sports, security, touch control, and document management systems. The above fields have paid increasing attention in artificial intelligence technologies, such as machine learning, big data processing and cloud computing, demanding huge amount of sensors and complicated sensors network.

We live in an era in which a myriad of computer systems produce immense amounts of (raw) data every day. This big data must be processed efficiently to gain valuable and hidden knowledge. Complex processing pipelines need to be designed for filtering out irrelevant data, also for efficient data mining and machine learning methods must be used to discover useful correlations in the big data. The purpose of this PhD research is the implementation of multi-objective evolutionary-based dimensionality reduction on a high volume of image/video data with the support of distributed multi-label classification algorithms.

In the digital bio-economy like in many other sectors, standards play an important role. With “Standards”, we refer here to the protocols that describe how data and the data-exchange are defined to enable digital exchange of data between devices. This chapter evaluates how Big Data, cloud processing, and app stores together form a new market that allows exploiting the full potential of geospatial data. This chapter focuses on the essential cornerstones that help make Big Data processing a more seamless experience for bioeconomy data. The described approach is domain-independent, thus can be applied to agriculture, fisheries, and forestry as well as earth observation sciences, climate change research, or disaster management. This flexibility is essential when it comes to addressing real world complexities for any domain, as no single domain has sufficient data available within its own limits to tackle the major research challenges our world is facing.

Database technology has been developing to exploit the next-generation hardware in the era of big data processing. At the same time, astronomical data size has been steadily increasing, and astronomical source catalogs obtained from largescale surveys with a wide-field camera, such as Subaru/Hyper Suprime-Cam (HSC), are a good test bench for evaluating the new database technology with a large data set. Such archive systems often employ a highly versatile relational database management system (RDBMS), but reducing the time required for data transaction and complex analysis has come to an important challenge. To tackle this difficulty, we aim to develop astronomical applications with a new catalog database using a next-generation RDBMS technology, where the query engine is designed to efficiently use computing infrastructures for processing big data. Demonstrations with science applications are essential to evaluate the new database. We verify query performance with the current HSC source catalog. For application to huge astronomical catalog databases, we are pursuing and verifying the capabilities of new database technologies. It will, in turn, enable fast ad hoc search and efficient detection of a wide range of variable events with the technology. Our pilot tests using typical astronomical queries on a cluster system shows significant improvements in response times with the aid of distributed query engines. We report performance of the test database for typical astronomical queries, and discuss optimizing the schema based on query workloads.

Spatial neighboring analysis is an indispensable part of geo-raster spatial analysis. In the big data era, high-resolution raster data offer us abundant and valuable information, and also bring enormous computational challenges to the existing focal statistics algorithms. Simply employing the in-memory computing framework Spark to serve such applications might incur performance issues due to its lack of native support for spatial data. In this article, we present a Spark-based parallel computing approach for the focal algorithms of neighboring analysis. This approach implements efficient manipulation of large amounts of terrain data through three steps: (1) partitioning a raster digital elevation model (DEM) file into multiple square tile files by adopting a tile-based multifile storing strategy suitable for the Hadoop Distributed File System (HDFS), (2) performing the quintessential slope algorithm on these tile files using a dynamic calculation window (DCW) computing strategy, and (3) writing back and merging the calculation results into a whole raster file. Experiments with the digital elevation data of Australia show that the proposed computing approach can effectively improve the parallel performance of focal statistics algorithms. The results also show that the approach has almost the same calculation accuracy as that of ArcGIS. The proposed approach also exhibits good scalability when the number of Spark executors in clusters is increased.

With the wide application of intelligent sensors and internet of things (IoT) in the smart job shop, a large number of real-time production data is collected. Accurate analysis of the collected data can help producers to make effective decisions. Compared with the traditional data processing methods, artificial intelligence, as the main big data analysis method, is more and more applied to the manufacturing industry. However, the ability of different AI models to process real-time data of smart job shop production is also different. Based on this, a real-time big data processing method for the job shop production process based on Long Short-Term Memory (LSTM) and Gate Recurrent Unit (GRU) is proposed. This method uses the historical production data extracted by the IoT job shop as the original data set, and after data preprocessing, uses the LSTM and GRU model to train and predict the real-time data of the job shop. Through the description and implementation of the model, it is compared with KNN, DT and traditional neural network model. The results show that in the real-time big data processing of production process, the performance of the LSTM and GRU models is superior to the traditional neural network, K nearest neighbor (KNN), decision tree (DT). When the performance is similar to LSTM, the training time of GRU is much lower than LSTM model.

The rapid growth of data, the increasing number of network based applications, and the advent of the omnipresence of internet and connected devices have affected the importance of information security. Hence, a security system such as an Intrusion Detection System (IDS) becomes a fundamental requirement. However, the complexity of the generated data and their huge size, plus, the variation of Cyber-attacks on: the network traffic, wireless network traffic, worldwide network traffic, connected devices and 5 G communication media, lead to hinder the IDS’s efficiency. Dealing with this huge amount of traffic is challenging and requires deploying new big data security solutions. This paper proposes an overview of intrusion detection which offers a review of IDS that deploy big data technologies and provides interesting recommendations for further study.

With the development of mobile Internet, association rules data mining is still a research hotspot. In this paper, the traditional mining algorithm of association rules for Apriori is analyzed, which has low efficiency and poor expansibility, because it scans the database many times and produces a large number of redundant frequent itemsets when dealing with big data. Therefore, it is proposed that the Apriori algorithm is improved by using the MapReduce model of Hadoop platform to parallelize processing, and the experimental results show that the improved Apriori algorithm has high efficiency and good stability in big data processing, it has great potential to excavate. Finally, the algorithm is applied to the data mining of student scores, which verifies its effectiveness and can provide services for education management.

The HACE Theorem has emerged as a way to characterize big data. Over the years it has become fundamental to big data characterization as the Newton’s Laws are to Physics. Associated with the HACE theorem is the Big Data Processing Framework for storing, managing, analyzing and sharing massive amounts of heterogenous, autonomous and distributed data with complex and evolving relationships. This paper examines the security and privacy aspects for the HACE theorem. It argues that what is needed is a Policy-Aware Big Data Processing Framework for the collection, storage, management, mining, and sharing of the massive amounts of data. It also examines knowledge graphs to represent the big data and determines ways to reason about the graphs and yet maintain security and privacy.

Processing big data requires serious computing resources. Because of this challenge, big data processing is an issue not only for algorithms but also for computing resources. This article analyzes a large amount of data from different points of view. One perspective is the processing of reduced collections of big data with less computing resources. Therefore, the study analyzed 40 GB data to test various strategies to reduce data processing. Thus, the goal is to reduce this data, but not to compromise on the detection and model learning in machine learning. Several alternatives were analyzed, and it is found that in many cases and types of settings, data can be reduced to some extent without compromising detection efficiency. Tests of 200 attributes showed that with a performance loss of only 4%, more than 80% of the data could be ignored. The results found in the study, thus provide useful insights into large data analytics.

Big data analytics (BDA) is progressively becoming a popular practice implemented by many organizations, because of its potential to discover treasured insights for improved decision-making. Machine Learning (ML) algorithms are one of the effective tools used for BDA, however, their computational complexity increases with an increase in data size. Therefore, this paper introduces a boundary detection and instance selection technique for improving the speed of MLbased big data classification models. The proposed technique (called ACOISA_ML) is inspired by edge selection in ant colony optimization. ACOISA_ML is evaluated on five ML algorithms and ten large- or medium-scale datasets, and the results show that it has the potential to reduce the training speed of ML algorithms by over 94% without significantly affecting their prediction accuracy. Moreover, the results show that it reduced the storage size of big datasets by over 55% (in most cases), thus improving the speed of big data processing.

The method of load balancing for indivisible network resources when parallelizing Big Data processing is considered. A method for finding the optimal partitioning of a processing service into parallel processes is proposed.

The large amount of real time medical measurement parameters stored in the SQL server needs processing using a specific algorithm. One of the big data processing techniques is available for medical data is Genetic algorithm. The acquired medical parameters are combined together to predict or diagnose the disease using the genetic algorithm. In this paper, the genetic algorithm is used to process the medical measurements data. The medical parameters are posted temporarily in the Representational Structure (REST) Application Program Interface (API) using a gateway protocol MQTT. The genetic algorithm can easily diagnose the disease using the existing stored parameters. The medical parameters of the patient like ECG, Blood pressure and skin temperature are posted frequently in the cloud server for continuous monitoring, and the huge data is also processed using this proposed method.