Search

Abstract The traditional methods used in big data, like cluster creation and query-based data extraction, fail to yield accurate results on massive networks. To address such issues, the proposed approach involves using the Hadoop Distributed File System (HDFS) for data processing, the map-reduce programming paradigm for data processing, and query optimization techniques to quickly and effectively extract accurate outcomes from a variety of options with a high processing capacity. The methodology proposed in this work makes use of Gaussian Mixture Model (GMM) for data clustering and the Salp Swarm Algorithm (SSA) for optimization. The security of preprocessed data stored on networked clusters with interconnections has been ensured by SHA algorithms. Finally, incorporating into consideration the important parameters, evaluation findings for the experimental performance of the model in the indicated methodology are produced. For this work, the estimated range of input file sizes is 60–100 MB. The processing of 100 MB of input files yielded an accuracy of 96% and results for specificity and sensitivity of 90% and 93%, respectively. The outcomes have been compared with well-known methods like fuzzy C-means and K-means approaches, and the results show that the proposed method effectively distributes accurate data processing to cluster nodes with low latency. Moreover, it uses the least amount of memory resources possible when operating on functional CPUs. As a result, the proposed approach outperforms existing techniques.

The traditional methods used in big data, like cluster creation and query-based data extraction, fail to yield accurate results on massive networks. To address such issues, the proposed approach involves using the Hadoop Distributed File System (HDFS) for data processing, the map-reduce programming paradigm for data processing, and query optimization techniques to quickly and effectively extract accurate outcomes from a variety of options with a high processing capacity. The methodology proposed in this work makes use of Gaussian Mixture Model (GMM) for data clustering and the Salp Swarm Algorithm (SSA) for optimization. The security of preprocessed data stored on networked clusters with interconnections has been ensured by SHA algorithms. Finally, incorporating into consideration the important parameters, evaluation findings for the experimental performance of the model in the indicated methodology are produced. For this work, the estimated range of input file sizes is 60–100 MB. The processing of 100 MB of input files yielded an accuracy of 96% and results for specificity and sensitivity of 90% and 93%, respectively. The outcomes have been compared with well-known methods like fuzzy C-means and K-means approaches, and the results show that the proposed method effectively distributes accurate data processing to cluster nodes with low latency. Moreover, it uses the least amount of memory resources possible when operating on functional CPUs. As a result, the proposed approach outperforms existing techniques. [ABSTRACT FROM AUTHOR]

Clustering algorithms are part of algorithms of unsupervised-learning that are commonly used in different fields. Two major impacts on the data clustering algorithm have been the rapid developments and creation of electronic data. This includes the view of how the big data is stored as well as how it is processed. A cluster has a high level of resemblance to the same cluster and a low level of resemblance to other clusters. Clustering algorithms are commonly used in all fields, such as retail, banking, development, etc.. Even though different methodologies are proposed with different scenarios, but none of the existing methodologies are proven to be a better approaches. Here, in our work we have adopted an improved Map- Reduce programming model to combine the Canopy clustering and K-means clustering algorithms to process the Microarray data with the available commodity hardware with an aim to achieve better performance. The results obtained from different scenarios shown that the proposed method was capable of improving computational speed significantly by increasing the nodes as required. In this research, this paper explored the efficiency by evaluating and implementing the proposed Improved k-means with Improved Map Reduce algorithm which runs on Hadoop Frame work using Microarray dataset along with different datasets. [ABSTRACT FROM AUTHOR]

Big data security is becoming increasingly important in today's data-driven world. Big data refers to large amounts of data generated from applications like airlines, hospitals, and government organizations, including social media and banking. This data contains insightful information that can be used for data analytics, research, and decision-making. However, because the data may contain sensitive information such as personally identifiable information, trade secrets, and confidential business data, it poses significant security risks. Big data security entails protecting the data's confidentiality, authenticity, and accessibility. MapReduce is the part of big data that can process large datasets in a distributed computing environment. Google initially developed it, which is now widely used in big data processing. MapReduce works by dividing the extensive data set into smaller chunks and distributing the processing across a cluster of computers. The map function converts the given input information into key-value pairs. The second phase is the reduced phase focused on generating the intermediate results from the map phase and combined as the final results. The reduce function condenses the key-value pairs produced by the map function into more minor key-value pairs. This paper describes an advanced detection system (ADS) to predict cyber Attacks from two publically datasets, KDD Cup 1999 and UNSW-NB15 Dataset. The performance of ADS is improved by adopting the rough set theory for the effective prediction of cyber Attacks. [ABSTRACT FROM AUTHOR]

Medical care services are changing to address problems with the development of big data frameworks as a result of the widespread use of big data analytics. Covid illness has recently been one of the leading causes of death in people. Since then, related input chest X-ray image for diagnosing COVID illness have been enhanced by diagnostic tools. Big data technological breakthroughs provide a fantastic option for reducing contagious Covid disease. To increase the model's confidence, it is necessary to integrate a large number of training sets, however handling the data may be difficult. With the development of big data technology, a unique method to identify and categorise covid illness is now found in this research. In order to manage incoming big data, a massive volume of chest x-ray images is gathered and analysed using a distributed computing server built on the Hadoop framework. In order to group identical groups in the input x-ray images, which in turn segments the dominating portions of an image, the fuzzy empowered weighted k-means algorithm is then employed. A hybrid quantum dilated convolution neural network is suggested to classify various kinds of covid instances, and a Black Widow-based Moth Flame is also shown to improve the performance of the classifier pattern. The performance analysis of COVID-19 detection makes use of the COVID-19 radiography dataset. The suggested HQDCNet approach has an accuracy of 99.01. The experimental results are evaluated in Python using performance metrics such as accuracy, precision, recall, f-measure, and loss function. [ABSTRACT FROM AUTHOR]

Information granules are fundamental and effective constructs in human cognitive and decision-making activities. It is unavoidable to automatically describe and extract some valuable information from large-scale datasets. In this study, we suggest a data-adaptive information granulation method to reinforce the data description mechanisms. First, a dynamic programming-based attribute-value clustering method is proposed to adaptively partition the numeric attribute values to several optimal data fragments for different condition attributes. Then, some interval information granules are constructed from the clustered data fragments by the principle of justifiable granularity for some specific tasks in machine learning. Furthermore, we introduce the distributed implementation in the framework of Map-Reduce to overcome some restrictions in large-scale datasets. The experiments on several benchmark datasets are provided to demonstrate the adaptivity on different datasets and the feasibility in machine learning tasks. Furthermore, the distributed performance are tested on some artificial datasets. (left).figa [ABSTRACT FROM AUTHOR]

Objectives: In the large-scale of virtual reality scene, it is difficult to add all graphics data into the video memory for rendering. Removing the occluded objects in advance by visible query technology can reduce the amount of data loaded on the display end to improve the rendering efficiency. Therefore, the research of visible query method for regional objects has important application value for real-time rendering of large-scale urban scene. Methods: We put forward a distributed visible query method based on Map-Reduce. In the map phase, we apply a hierarchical axis-aligned bounding box as viewpoint space partition. When the number of 3D objects in viewpoint space partition exceeds the threshold, the axis-aligned bounding box continues to be divided into sub- boxes. After the above process, the map tasks produce GeoTuples with the VSPID as key and visible query candidate set as value. In the reduce phase, a viewpoint is created for each leaf axis-aligned bounding box where the binary space partitioning trees are build and the visible set is calculated using real-time occlusion algorithm. Results: The experiment results with a building compound, containing more than 200 000 geometric solids, in Shenzhen, China show that: (1) There is no simple linear relationship between the running time of distributed visible query and the number of viewpoint space partitions. (2) Running time and parallelism are not simply inversely proportional. The computational efficiency of each process first increases and then decreases with the increase of parallelism. About 48 parallelism, the process has the highest efficiency. (3) Whether the distributed approach is better than the traditional approach depends on the number of 3D objects. After the amount of 3D objects reaches about 40 000, the distributed algorithm begins to be better than the traditional algorithm. Conclusions: The computational experiments reveal the proposed algorithms outperform competitors in terms of the processing efficiency and feasibility, which can meet the requirement of visible query in large-scale scenarios. [ABSTRACT FROM AUTHOR]

Indexing is one of the key components of any search tool to be optimized for searching documents. Among the existing indexing techniques, inverted indexing is one of the best methods used at a larger scale for various applications. Under this method, the index is designed using a signature file, hash tree and B-tree to retrieve the required document in efficient time. B-tree is popular due to its searching efficiency, but its performance degrades with increasing data set size. The wavelet tree has become a popular and versatile data structure in the last decade, used in various domains such as sequences, indexing, compression, and grid-point with surprising results. This study proposes a parallel wavelet tree algorithm with hybridization of the Map-Reduce concept to construct an index for textual search. The proposed algorithm reduces the index construction time considerably. Experiments show that the proposed algorithm takes a reasonable trade-off with existing indexing approaches. For large data sets, index construction time has been reduced with respect to other existing state-of-art schemes. Also, results show that the algorithm performs well when the data-set scales up to up-to-the full utilization of available cores. It is possible due to the use of multiple threads working in parallel. Our experiment demonstrated consistent performance with 2-core, 4-core, 8-core, 12-core and results of 16-core show increase in index construction time due to parallel overhead when the data-set in not sufficiently large. [ABSTRACT FROM AUTHOR]

The goal of this article is to identify fundamental limitations on how efficiently algorithms implemented on platforms such as MapReduce and Hadoop can compute the central problems in motivating application domains, such as graph connectivity problems. We introduce an abstract model of massively parallel computation, where essentially the only restrictions are that the "fan-in" of each machine is limited to s bits, where s is smaller than the input size n, and that computation proceeds in synchronized rounds, with no communication between different machines within a round. Lower bounds on the round complexity of a problem in this model apply to every computing platform that shares the most basic design principles of MapReduce-type systems. We prove that computations in our model that use few rounds can be represented as low-degree polynomials over the reals. This connection allows us to translate a lower bound on the (approximate) polynomial degree of a Boolean function to a lower bound on the round complexity of every (randomized) massively parallel computation of that function. These lower bounds apply even in the "unbounded width" version of our model, where the number of machines can be arbitrarily large. As one example of our general results, computing any nontrivial monotone graph property—such as connectivity—requires a super-constant number of rounds when every machine receives only a subpolynomial (in n) number of input bits s. Finally, we prove that, in two senses, our lower bounds are the best one could hope for. For the unbounded-width model, we prove a matching upper bound. Restricting to a polynomial number of machines, we show that asymptotically better lower bounds would separate P from NC1. [ABSTRACT FROM AUTHOR]

In Many domains, data is increasing day by day. Moreover, the large amount of data creates the best challenges for the users. Big data is one of the high focus domains in data science. In many organizations, big data plays a major role in processing large datasets and extracts useful information from the data. Security becomes more important to all the domains and applications. Many applications use different types of security approaches to secure sensitive data from attackers. Many real-time applications are providing security for applications such as banking, trading, and e-commerce applications using better security algorithms. Processing a large amount of real-time data gets a lot of benefits to analyzing the threats in cyber-security systems. The data is collected from online real-time sources and this cyber security data consists of network information, sensor data, threat information, analysis of intrusion detection, and identifying the sensitive data in real-time applications. In this collected information various vulnerabilities and attacks are becoming prevalent and develop security solutions accordingly. In this paper, the Map-reduce-based Ensemble Intrusion Detection System (MR-EIDS) is developed to detect intruders and attackers from the real-time datasets. Two benchmark datasets are used to analyze the intrusion data. The proposed system analyzes the dataset and finds interesting patterns in the datasets. Performance is also measured by showing improved results. [ABSTRACT FROM AUTHOR]

Educational systems have advanced with the use of electronic learning (e-learning), which is a promising solution for long-distance learners. Students who engage in e-learning can access tests and exams online, making education more flexible and accessible. This work reports on the design of an e-learning system that makes recommendations to students to improve their learning. This artificial intelligence-based student assessment and recommendation (AISAR) system consists of score estimation, clustering, performance prediction, and recommendation. In addition, the importance of student authentication is recognised in situations in which students must authenticate themselves prior to using the e-learning system using their identity, password, and personal identification number. Individual scores are determined using a recurrent neural network (RNN) based on student engagement and examination scores. Then, a density-based spatial clustering algorithm (DBSCAN) using Mahalanobis distance clustering is implemented to group students based on their obtained score values. The constructed clusters are validated by estimating purity and entropy. Student performance is predicted using a threshold-based MapReduce (TMR) procedure from the score-based cluster. When predicting student performance, students are classified into two groups: average and poor, with the former being divided into below- and above-average students and the latter into poor and very poor students. This categorisation aims to provide useful recommendations for learning. A recommendation reinforcement learning algorithm, the rule-based state–action–reward–state–action (R-SARSA) algorithm, is incorporated for evaluation. Students were required to work on their subjects according to the provided recommendations. This e-learning recommendation system achieves better performance in terms of true-positives, false-positives, true-negatives, false-negatives, precision, recall, and accuracy. [ABSTRACT FROM AUTHOR]

With the increasing number of heterogeneous resource-constrained devices populating the current wireless ecosystem, enabling ubiquitous computing at the edge of the network requires moving part of the computing burden back to the edge to reduce user-side latency and relieve the backhaul network. Motivated by this challenge, this work investigates edge-facilitated collaborative fog computing to augment the computing capabilities of individual devices while optimizing for energy-efficiency. Collaborative-computing is modeled using the Map-Reduce framework, consisting in two computing rounds and a communication round. The computing load is optimally distributed among devices, taking into account their diversity in terms of computing and communication capabilities. Devices local parameters such as CPU frequency and RF transmit power are also optimized for energy-efficiency. The corresponding optimization problem is shown to be convex and optimality conditions are obtained through Lagrange duality theory. A waterfilling-like interpretation for the size of the computing load assigned to each device is given. Numerical experiments demonstrate the benefits of the proposed collaborative-computing scheme over various other schemes in several respects. Most notably, the proposed scheme exhibits increased probability of successfully dealing with more demanding computations in time, along with significant energy-efficiency gains. Both improvements come from the scheme ability to advantageously leverage devices diversity.

This paper presents the design, implementation, and operation of a novel distributed fault-tolerant middleware. It uses interconnected WSNs that implement the Map-Reduce paradigm, consisting of several low-cost and low-power mini-computers (Raspberry Pi). Specifically, we explain the steps for the development of a novice, fault-tolerant Map-Reduce algorithm which achieves high system availability, focusing on network connectivity. Finally, we showcase the use of the proposed system based on simulated data for crowd monitoring in a real case scenario, i.e. a historical building in Greece (M. Hatzidakis' residence). The technical novelty of this article lies in presenting a viable low-cost and low-power solution for crowd sensing without using complex and resource-intensive AI structures or image/video recognition techniques. [ABSTRACT FROM AUTHOR]

Sentiment analysis is the process of analyzing a person’s perception or belief about a particular subject matter. However, finding correct opinion or interest from multi-facet sentiment data is a tedious task. In this paper, a method to improve the sentiment accuracy by utilizing the concept of categorized dictionary for sentiment classification and analysis is proposed. A categorized dictionary is developed for the sentiment classification and further calculation of sentiment accuracy. The concept of categorized dictionary involves the creation of dictionaries for different categories making the comparisons specific. The categorized dictionary includes words defining the positive and negative sentiments related to the particular category. It is used by the mapper reducer algorithm for the classification of sentiments. The data is collected from social networking site and is pre-processed. Since the amount of data is enormous therefore a reliable open-source framework Hadoop is used for the implementation. Hadoop hosts various software utilities to inspect and process any type of big data. The comparative analysis presented in this paper proves the worthiness of the proposed method.

Over the last few years, the huge amount of data represented a major obstacle to data analysis. Big data implies that the volume of data undergoes a faster progress than computational speeds, thereby demanding a larger data storage capacity. The Internet of Things (IoT) is a main source of data that is closely related to big data, as the former extends to a variety of fields such as healthcare, entertainment, and disaster control. Despite the different advantages associated with the composition of Big Data analytics and IoT, there are a number of complex difficulties and issues involved that need to be resolved and managed to ensure an accurate data analysis. Some of these solutions include the utilization of map-reduce techniques, processing, and large data scale, particularly for the relatively less time that this method requires to process large data from the Internet of Things. Machine learning algorithms of this kind are often implemented in the healthcare sector. Medical facilities need to be advanced so that more appropriate decisions can be made in terms of patient diagnosis and treatment options. In this work, two datasets have been used: the first set, used in the prediction of heart diseases, obtained an accuracy rate of 84.5 for RF and 83 for J48, whereas the second dataset is related to weather stations (automated sensors) and obtained accuracy rates of 88.5 and 86.5 for RF and J48, respectively.

More and more tourists are sharing their travel feelings and posting their real experiences on the Internet, generating tourism big data. Online travel reviews can fully reflect tourists’ emotions, and mining and analyzing them can provide insight into the value of them. In order to analyze the potential value of online travel reviews by using big data technology and machine learning technology, this paper proposes an improved support vector machine (SVM) algorithm based on travel consumer sentiment analysis and builds an Hadoop Distributed File System (HDFS) system based on Map-Reduce model. Firstly, Internet travel reviews are pre-processed for sentiment analysis of the review text. Secondly, an improved SVM algorithm is proposed based on the main features of linear classification and kernel functions, so as to improve the accuracy of sentiment word classification. Then, HDFS data nodes are deployed on the basis of Hadoop platform with the actual tourism application context. And based on the Map-Reduce programming model, the map function and reduce function are designed and implemented, which greatly improves the possibility of parallel processing and reduces the time consumption at the same time. Finally, an improved SVM algorithm is implemented under the built Hadoop platform. The test results show that online travel reviews can be an important data source for travel big data recommendation, and the proposed method can quickly and accurately achieve travel sentiment classification.

An optimal approach to anonymization using small data is proposed in this study. Map Reduce is a big data processing framework used across distributed applications. Prior to the development of a map reduce framework, data are distributed and clustered using a hybrid clustering algorithm. The algorithm used for grouping together similar techniques utilises the k-means clustering algorithm, along with the MFCM clustering algorithm. Clustered data is then fed into the map reduce frame work after it has been clustered. In order to guarantee privacy, the optimal k anonymization method is recommended. When using generalisation and randomization, there are two techniques that can be employed: K-anonymity, which is unique to each, depends on the type of the quasi identifier attribute. Our method replaces the standard k anonymization process by employing an optimization algorithm that dynamically determines the optimal k value. This algorithm uses the Modified Grey Wolf Optimization (MGWO) algorithm for optimization. The memory, execution time, accuracy, and error value are used to assess the recommended method's practise. This experiment has shown that the suggested method will always finish ahead of the existing method by using the least amount of time while ensuring the greatest level of security. The current technique gets the lowest accuracy and the privacy proposed achieves the maximum accuracy while compared to the current technique. The solution is implemented in Java with Hadoop Map-Reduce, and it is tested and deployed in the cloud on Google Cloud Platform. [ABSTRACT FROM AUTHOR]

More and more tourists are sharing their travel feelings and posting their real experiences on the Internet, generating tourism big data. Online travel reviews can fully reflect tourists' emotions, and mining and analyzing them can provide insight into the value of them. In order to analyze the potential value of online travel reviews by using big data technology and machine learning technology, this paper proposes an improved support vector machine (SVM) algorithm based on travel consumer sentiment analysis and builds an Hadoop Distributed File System (HDFS) system based on Map-Reduce model. Firstly, Internet travel reviews are pre-processed for sentiment analysis of the review text. Secondly, an improved SVM algorithm is proposed based on the main features of linear classification and kernel functions, so as to improve the accuracy of sentiment word classification. Then, HDFS data nodes are deployed on the basis of Hadoop platform with the actual tourism application context. And based on the Map-Reduce programming model, the map function and reduce function are designed and implemented, which greatly improves the possibility of parallel processing and reduces the time consumption at the same time. Finally, an improved SVM algorithm is implemented under the built Hadoop platform. The test results show that online travel reviews can be an important data source for travel big data recommendation, and the proposed method can quickly and accurately achieve travel sentiment classification. [ABSTRACT FROM AUTHOR]

Clustering is one of the major operations to analyse genome sequence data. Sophisticated sequencing technologies generate huge DNA sequence data; consequently, the complexity of analysing sequences is also increased. So, there is an enormous need for faster sequence analysis algorithms. Most of the existing tools focused on alignment‐based approaches, which are slow‐paced for sequence comparison. Alignment‐free approaches are more successful for fast clustering. The state‐of‐the‐art methods have been applied to cluster small genome sequences of various species; however, they are sensitive to large size sequences. To subdue this limitation, we propose a novel alignment‐free method called DNA sequence clustering with map‐reduce (DCMR). Initially, MapReduce paradigm is used to speed up the process of extracting eight different types of repeats. Then, the frequency of each type of repeat in a sequence is considered as a feature for clustering. Finally, K‐means (DCMR‐Kmeans) and K‐median (DCMR‐Kmedian) algorithms are used to cluster large DNA sequences by using extracted features. The two variants of proposed method are evaluated to cluster large genome sequences of 21 different species and the results show that sequences are very well clustered. Our method is tested for different benchmark data sets like viral genome, influenza A virus, mtDNA, and COXI data sets. Proposed method is compared with MeshClust, UCLUST, STARS, and ClustalW. DCMR‐Kmeans outperforms MeshClust, UCLUST, and DCMR‐Kmedian with respect to purity and NMI on virus data sets. The computational time of DCMR‐Kmeans is less than STARS, DCMR‐Kmedian, and much less than UCLUST on COXI data set. [ABSTRACT FROM AUTHOR]

Big data has ushered in the era of the tera, during which massive amounts of data are being gathered at accelerating rates. The size of the world's data is increasing in zeta-bytes as a result of improvements in processing speed, storage capacity, and data availability. One of the big data technologies is Hadoop, which uses the Map-Reduce and Hadoop Distributed File System to analyse data. An essential task for effective cluster resource management is job scheduling. Schedulers for Hadoop are pluggable parts that allocate resources to jobs. The default FIFO, Fair, and Capacity schedulers are prevalent in a variety of schedulers. A thorough analysis of the various work scheduling algorithms has been carried out in this paper. Additionally, their comparative parametric analysis was conducted while highlighting the essential features that these schedulers have in common. [ABSTRACT FROM AUTHOR]

We have been studying the problem of clustering data objects. As we have implemented a newalgorithm EMaRC which is An Efficient Map Reduce algorithm for Clustering Data. In clusters Featureselection is the most important part of the clustering process that involves and identifying the set of features of asubset, at which they produces accurate and accordant results with the original set of features. The main conceptbehind this paper is that, to give the effective outcomes of clustering features. In this the nature of clustering andsome more concepts serves for processing large data sets. A map-reduce concept is involved followed by featureselection algorithm which affects the entire process of clustering to get the most effective and features producesefficiently. While efficiency concerns, the time complexity is desirable component, which the time required to findeffective features, where effectiveness is related to the quality of the features of subsets. Based on these criteria, acluster based map-reduce feature selection approach, is proposed and evaluated in this paper.

Abstract The DBSCAN algorithm is a prevalent method of density-based clustering algorithms, the most important feature of which is the ability to detect arbitrary shapes and varied clusters and noise data. Nevertheless, this algorithm faces a number of challenges, including failure to find clusters of varied densities. On the other hand, with the rapid development of the information age, plenty of data are produced every day, such that a single machine alone cannot process this volume of data; hence, new technologies are required to store and extract information from this volume of data. A large volume of data that is beyond the capabilities of existing software is called Big data. In this paper, we have attempted to introduce a new algorithm for clustering big data with varied density using a Hadoop platform running MapReduce. The main idea of this research is the use of local density to find each point’s density. This strategy can avoid the situation of connecting clusters with varying densities. The proposed algorithm is implemented and compared with other algorithms using the MapReduce paradigm and shows the best varying density clustering capability and scalability.