Search

Deep neural network (DNN) comprises multiple stages of data processing sub-systems with one of the primary sub-systems is a fully connected neural network (FCNN) model. This fully connected neural network model has multiple layers of neurons that need to be implemented using arithmetic units with suitable number representation to optimize area, power, and speed. In this work, the network parameters are analyzed, and redundancy in weights is eliminated. A pipelined and parallel structure is designed for the fully connected network information. The proposed FCNN structure has 16 inputs, 3 hidden layers, and an output layer. Each hidden layer consists of 4 neurons and describes how the inputs are connected to hidden layer neurons to process the raw data. A hardware description language (HDL) model is developed for the proposed structure and the verified model is implemented on Xilinx field-programmable gate array (FPGA). The modified structure comprises registers, demultiplexers, weight registers, multipliers, adders, and read-only memory lookup table (ROM/LUT). The modified architecture implemented on FPGA is estimated to reduce area by 87.5% and improve timing by 3x compared with direct implementation methods. [ABSTRACT FROM AUTHOR]

Contemporary industry and science expectations towards technological solutions set the bar high. Current approaches to increasing the computing power of standard systems are reaching the limits of physics known to humankind. Fast, programmable systems with relatively low power consumption are a different concept for performing complex calculations. Highly parallel processing opens up a number of possibilities in the context of accelerating calculations. Application of SoC (System On Chip) with FPGA (Field-Programmable Gate Array) enables the delegating of a part of computations to the gates matrix, thereby expediting processing by using parallelization of hardware operations. This paper presents the general concept of using SoC FPGA systems to support the CPU (Central Processing Unit) in many modern tasks. While some tasks might be really hard to implement on an FPGA in a reasonable time, the SoC FPGA platform allows for easy low-level interconnections, and with such virtualized access to the hardware computing resources, it is seen as making FPGAs, or hardware in general, more accessible to engineers accustomed to high-level solutions. The concept presented in the article takes into account the limited resources of cheaper educational platforms, which, however, still provide an interesting and alternative hybrid solution to the problem of parallelization and acceleration of data processing. This allows encountered limitations to be overcome and the flexibility known from high-level solutions and high performance achieved with low-level programming to be maintained without the need for a high financial background. [ABSTRACT FROM AUTHOR]

The randomization-based feedforward neural network has raised great interest in the scientific community due to its simplicity, training speed, and accuracy comparable to traditional learning algorithms. The basic algorithm consists of randomly determining the weights and biases of the hidden layer and analytically calculating the weights of the output layer by solving a linear overdetermined system using the Moore–Penrose generalized inverse. When processing large volumes of data, randomization-based feedforward neural network models consume large amounts of memory and drastically increase training time. To efficiently solve the above problems, parallel and distributed models have recently been proposed. Previous reviews of randomization-based feedforward neural network models have mainly focused on categorizing and describing the evolution of the algorithms presented in the literature. The main contribution of this paper is to approach the topic from the perspective of the handling of large volumes of data. In this sense, we present a current and extensive review of the parallel and distributed models of randomized feedforward neural networks, focusing on extreme learning machine. In particular, we review the mathematical foundations (Moore–Penrose generalized inverse and solution of linear systems using parallel and distributed methods) and hardware and software technologies considered in current implementations. [ABSTRACT FROM AUTHOR]

The object of research is the process of automating systems for structuring data from several sources. The subject of the research is methods and algorithms for implementing a complete system for automated and parallel processing, validation and structuring of data. One of the most problematic areas is the merging of databases with different structures and several common fields into a generalized structure. The research was aimed at developing a system to increase the efficiency of automation of big data processing. As a result of the work, optimization methods were studied, the influence of their internal parameters on the operation of algorithms was analyzed, their main advantages and disadvantages were determined, and software was developed in which the corresponding methods were implemented. An algorithm for structuring data before processing has been obtained. Data structuring is achieved by performing the «mapping» operation. Mapping can take place by indexes of already cleaned data or using a defined dictionary with a given set of keys, which allows not to care about the sequence of storing values and their possible shift. The practical significance of the developed system lies in the improvement of methods of collecting and processing information for the purpose of its further validation, cleaning and accumulation in the following categories: geographic addresses and geo-coordinates, validation and automated addition of a mobile phone number to the international format, processing of car numbers (in modern and outdated format), VIN code of the engine and car brand, validation of urls of social networks, passport data and processing of personal data. Compared to similar methods for processing large volumes of data, the possibility of splitting the input file or stream into separate parts was used, the cleaned data from which is combined at the end of the system operation. Thanks to this, it is possible to process data whose size exceeds the available volume of the device’s RAM, and the method of working with loosely structured text files in CSV format has been improved. [ABSTRACT FROM AUTHOR]

The act of gathering, looking over, and analyzing a lot of data to find patterns, insights, and market trends that can help businesses make more effective choices is known as big data analysis (BDA). Quick and effective access to this data allows businesses to be flexible in developing strategies to hold onto their competitive edge. To analyze massive amounts of data quickly through parallel processing, the structure of the Hadoop software employs the MapReduce methodology. Computational solid resources are necessary for BDA, although they are not always available. Developing new clustering techniques that could handle this kind of data processing became crucial. Therefore, in this research, we presented a novel, effective fuzzy-based Improved Multiview Fuzzy C-Means Algorithm (IMV-FCA) to boost the clustering strategy. To summarize, fuzzy-based IMV-FCA clustering presents the ensemble of the MobileNet V2 model, and three-layered stacked Bidirectional LSTM (MVSBiLSTM) to increase computing speed and effectiveness. It also presents a function that calculates the separation among the cluster center and the particular instance, to assist with better clustering. By simulating shared memory space and parallelizing on the framework known as MapReduce on the Hadoop cloud computing platform, the distributed database is utilized to improve the method’s effectiveness while reducing its time complexities. The experimental investigation was conducted on existing approaches, and the proposed approach was analyzed using three standard datasets. While differentiating from existing approaches, the presented approach yields greater performances in terms of various metrics. [ABSTRACT FROM AUTHOR]

In a location tracking system for a moving object, accuracy and real-time processing are essential factors. The Kalman filter, as a recursive function, stands out as one of the prominent algorithms for object tracking. It continuously compares measured data with predicted values based on the system characteristics in real-time. Then it corrects the error of the predicted values while considering the noise of both system and measured data. This paper focuses on designing a hardware based Kalman filter for object tracking in two dimensions. Following an analysis of the Kalman filter algorithm, the blocks capable of parallel processing are identified and configured to be processed in parallel, effectively reducing data processing time. The clock speed is enhanced by using the pipeline technique. In addition, the time-sharing technique is applied to increase the utilization of hardware resources and reduce the area. Data was processed at 32-bit floating points to uphold accuracy comparable to software-implemented Kalman filters. The proposed Kalman filter architecture is designed using Verilog HDL and then simulated in Synopsys VCS/Verdi. The accuracy is verified by comparing results with a software based Kalman filter designed using MATLAB. It was implemented using Zynq ZYNQ-7 ZC702 Programmable logic via Xilnix Vivado and can operate at 33MHz. It takes 44 clocks, or 1.32 us, to process one data. Therefore, it was confirmed that the designed Kalman filter hardware is suitable for real-time processing. [ABSTRACT FROM AUTHOR]

Currently, cryptographic hash functions are widely used in various applications, including message authentication codes, cryptographic random generators, digital signatures, key derivation functions, and post-quantum algorithms. Notably, they play a vital role in establishing secure communication between servers and clients. Specifically, servers often need to compute a large number of hash functions simultaneously to provide smooth services to connected clients. In this paper, we present highly optimized parallel implementations of Lightweight Secure Hash (LSH), a hash algorithm developed in Korea, on server sides. To optimize LSH performance, we leverage two parallel architectures: AVX-512 on high-end CPUs and NVIDIA GPUs. In essence, we introduce a word-level parallel processing design suitable for AVX-512 instruction sets and a data parallel processing design appropriate for the NVIDIA CUDA platform. In the former approach, we parallelize the core functions of LSH using AVX-512 registers and instructions. As a result, our first implementation achieves a performance improvement of up to 50.37% compared to the latest LSH AVX-2 implementation. In the latter approach, we optimize the core operation of LSH with CUDA PTX assembly and apply a coalesced memory access pattern. Furthermore, we determine the optimal number of blocks/threads configuration and CUDA streams for RTX 2080Ti and RTX 3090. Consequently, in the RTX 3090 architecture, our optimized CUDA implementation achieves about a 180.62% performance improvement compared with the initially ported LSH implementation to the CUDA platform. As far as we know, this is the first work on optimizing LSH with AVX-512 and NVIDIA GPU. The proposed implementation methodologies can be used alone or together in a server environment to achieve the maximum throughput of LSH computation. [ABSTRACT FROM AUTHOR]

The digital era brings up on one hand massive amounts of available data and on the other hand the need of parallel computing architectures for efficient data processing. String similarity evaluation is a processing task applied on large data volumes, commonly performed by various applications such as search engines, biomedical data analysis and even software tools for defending against viruses, spyware, or spam. String similarities are also used in musical industry for matching playlist records with repertory records composed of song titles, performer artists and producers names, aiming to assure copyright protection of mass-media broadcast materials. The present paper proposes a novel GPU-based approach for parallel implementation of the Jaro–Winkler string similarity metric computation, broadly used for matching strings over large datasets. The proposed implementation is applied in musical industry for matching playlist with over 100k records with a given repertory which includes a collection of over 1 million right owner records. The global GPU RAM memory is used to store multiple string lines representing repertory records, whereas single playlist string comparisons with the raw data are performed using the maximum number of available GPU threads and the stride operations. Further on, the accuracy of the Jaro–Winkler approach for the string matching procedure is increased using both an adaptive neural network approach guided by a novelty detection classifier (aNN) and a multiple-features neural network implementation (MF-NN). Thus, the aNN approach yielded an accuracy of 92% while the MF-NN approach achieved an accuracy of 99% at the cost of increased computational complexity. Timing considerations and the computational complexity are detailed for the proposed approaches compared with both the general-purpose processor (CPU) implementation and the state-of-the-art GPU approaches. A speed-up factor of 21.6 was obtained for the GPU-based Jaro–Winkler implementation compared with the CPU one, whereas a factor of 3.72 was obtained compared with the existing GPU implementation of string matching procedure based on Levenstein distance metrics. [ABSTRACT FROM AUTHOR]

Data processing is a key challenge for computationally limited Ground Users (GUs) in various applications. Unmanned Aerial Vehicles (UAVs) equipped with Multi-access Edge Computing (MEC) servers can assist GUs by offloading their computing tasks. However, existing work ignores fairness when multiple GUs compete for limited computing resources, which may result in UAV underserving certain GUs. In this paper, we investigate a flight trajectory optimization based on reinforcement learning for UAV selection of target GUs for task computation, which provides low latency and fair offloading computing services for GUs by jointly training UAV flight trajectories and task offloading decisions. We formulate UAV flight and offloading as a mixed integer non-convex optimization problem with high-dimensional state and action spaces. The problem is then transformed into a Markov Decision Processes (MDPs) problem and the Maximizing Service Efficiency Proximal Policy Optimization (MSE-PPO) algorithm is proposed to find the optimal solution. The algorithm adopts an actor-critic-based parallel architecture to handle the parameterized action space. Specifically, the UAV position sequence is updated while ensuring an optimal offloading policy between the UAV and the GUs. Simulation results verify that the average system rewards including computational energy efficiency and fairness index are improved by 35.06 % and 12.10 % respectively compared to DDPG and PPO algorithms. [ABSTRACT FROM AUTHOR]

Summary: Recent studies have shown that a large amount of energy consumption in the communication medium occurs at the data processing plane. The data plane executes the forwarding decisions in the routers and switches. Through initial studies, the line card was identified as the most granular component of the router and later forward engine, a sub component of line card has been recognized as the most power intensive component of routers. Fundamental techniques like link rate adaption, sleep and wake‐up have been imposed to reduce the energy at line cards and forward engines. On the other hand, router's architectures undergo drastic changes like empowering data plane with parallel processing due to the rise of gigabit routers. In this paper, to increase the degree of local autonomy and better energy savings, we have adopted the parallel processing design and imposed a variable amount of traffic on different line cards. The existing parallel forward engines perform load balancing in synchronous manner. In our design, asynchronous behavior is imposed such that every line card could independently decide to wake up or turn off their forward engine based on the traffic parameters. We have done simulations and witnessed increased energy savings (average sleeping time) without compromising the completion of lookup process. [ABSTRACT FROM AUTHOR]

Background and Objectives: From the multiview recorded video, free viewpoint video provides flexible viewpoint navigation. Thus, a lot of views need to be sent to the receivers in an encoded format. The scalable nature of the coded bitstream is one method of lowering the volume of data. However, adhering to the limitations of the free viewpoint application heavily relies on the kind of scalable modality chosen. The perceptual quality of the received sequences and the efficiency of the compression technique are significantly impacted by the scalable modality that was chosen. Methods: In order to address the primary issues with free-viewpoint video, such as high bandwidth requirements and computational complexity, this paper suggests a scalable framework. The two components of the suggested framework are as follows: 1) introducing appropriate scalable modality and data assignment to the base and enhancement layers; and 2) bit budget allocation to the base and enhancement layers using a rate control algorithm. In our novel scalable modality, termed Tile-based scalability, the idea of Region of Interest (ROI) is employed, and the region of interest is extracted using the tile coding concept first presented in the MV-HEVC. Results: When compared to the state-of-the-art techniques, our approach's computational complexity can be reduced by an average of 44% thanks to the concept of tile-coding with parallel processing capabilities. Furthermore, in comparison to standard MV-HEVC, our suggested rate control achieves an average 17.7 reduction in bandwidth and 1.2 improvement in video quality in the Bjøntegaard-Bitrate and Bjøntegaard-PSNR scales. Conclusion: Using new tile-based scalability, a novel scalable framework for freeviewpoint video applications is proposed. It assigns appropriate regions to the base and enhancement layers based on the unique features of free viewpoint scalability. Next, a rate control strategy is put forth to allocate a suitable bitrate to both the base and enhancement layers. According to experimental results, the suggested method can achieve a good coding efficiency with significantly less computational complexity than state-of-the-art techniques that used the 1-domain rate control method. [ABSTRACT FROM AUTHOR]

Transfer learning approaches in natural language processing have been explored and evolved as a potential solution for solving many problems in recent days. The current research on aspect-based summarization shows unsatisfactory accuracy and low-quality generated summaries. Additionally, the potential advantages of combining language models with parallel processing have not been explored in the existing literature. This paper aims to address the problem of aspect-based extractive text summarization using a transfer learning approach and an optimization method based on map reduce. The proposed approach utilizes transfer learning with language models to extract significant aspects from the text. Subsequently, an optimization process using map reduce is employed. This optimization framework includes an in-node mapper and reducer algorithm to generate summaries for important aspects identified by the language model. This enhances the quality of the summary, leading to improved accuracy, particularly when applied to electrical power system documents. By leveraging the strengths of natural language models and parallel data processing techniques, this model presents an opportunity to achieve better text summary generation. The performance metric used is accuracy, measured with the ROUGE tool, incorporating precision, recall and f-measure. The proposed model demonstrates a 6% improvement in scores compared to state-of-the-art techniques. [ABSTRACT FROM AUTHOR]

This paper proposes an improved method for diagnosing errors of computer devices, which process economic data in computer systems, by increasing efficiency. The object of research is the processes of control and diagnosis of data errors, in the class of residuals (CR). The improved method of diagnosing economic data represented in the non-positional system of calculation in CR is based on the application of orthogonal bases of partial sets of bases. The use of these bases makes it possible to organize the process of parallel processing of the projections of the output number of the non-positional code structure. This makes it possible to increase the efficiency of data diagnostics by n times, depending on the length of the bit grid of computer systems. For a single-byte computer system, the efficiency of data diagnosis increases by 1.2 times compared to the existing method based on the zeroing principle. At the same time, the effectiveness of operational control and diagnostics systems for data processing in CR with the growth of bit grids has been proven. Based on the results, it is shown that, unlike the correction codes used in the positional counting system, the arithmetic codes in CR have additional correction capabilities. An example of a specific implementation of the process of applying operational control and diagnosing errors in the processing of economic data presented in CR is given. The improved diagnostic method will make it possible (in comparison with existing diagnostic methods) to reduce the time, which increases the efficiency of the diagnostic procedure of data operating in CR. The results give grounds for asserting that, based on the improved method and the developed algorithm for the implementation of data diagnostics, it is possible to synthesize a device for reliable and operational control and diagnostics of economic data operating in CR. [ABSTRACT FROM AUTHOR]

The aim of this research work is to reduce the time complexity of the backpropagation algorithm, used for training neural networks. While the traditional approach to neural network training involves the well-established backpropagation algorithm, this paper introduces modifications to it. The study employs OpenMP and message passing interface technologies to parallelize the gradient descent and stochastic gradient descent algorithms. The study parallelizes the gradient descent algorithm and stochastic gradient descent algorithm using OpenMP and message passing interface technologies. The parallel gradient descent, employing OpenMP’s incremental parallelism, demonstrates a threefold increase in speed, compared to its sequential counterpart. The paper also addresses the issue of ‘false sharing’, which can adversely impact the performance and scalability of the parallel algorithm. Moreover, by utilizing message passing interface technology on six cores, the parallel gradient descent runs nearly five times faster, while the stochastic gradient descent achieves a speed improvement of 4.5 times. Consequently, the parallel mode of the backpropagation algorithm proves to be beneficial for processing Big dataset enabling faster retraining of neural networks for streaming data processing. However, it is worth noting that the exchange of information between processes in the parallel versions can introduce some overhead, potentially slowing down the parallelization process. [ABSTRACT FROM AUTHOR]

Edge networks are providing services for an increasing number of companies, and they can be used for communication between edge devices and edge gateways. However, the performance of edge devices varies greatly, and it is not easy to upgrade low-performance edge devices. Therefore, cyber attackers can use the vulnerability of edge devices to implement advanced persistent threat attacks. This article proposes a network verification framework for edge networks that can minimize the upgrades needed to strengthen edge network security. First, the communication parties use the data transmitted by the given edge network. Our method uses our proposed PacketVerifier to attach verification information to the packet after it is sent and to verify and restore the packet before it reaches the receiver. Second, due to the performance requirements of edge networks, we design a new data processing structure, namely, a sliding window double ring, to improve the performance of strict sequential protocols in parallel validation. Finally, experimental simulations show that our parallel processing algorithm has good performance in terms of network bandwidth compared with two existing packet processing algorithms. Furthermore, the proposed packet with verification information is compatible with the existing network topology, which helps PacketVerifier establish trustworthy transmission in a zero-trust environment. [ABSTRACT FROM AUTHOR]

Large-scale spiking neural network (SNN) parallel simulation is an important means to explore brain function. The difficulty lies in properly mapping the load to the multi-process parallel computing platform and improving the simulation speed. In order to solve this problem, this paper proposed a SNN load balancing method based on united weight hypergraph partitioning, which could solve the problem of balancing computing load and communication load between processes in parallel computing, and improve the speed of SNN simulation. And it used sparse communication instead of collective communication to solve the problem of data redundancy in the process of event communication and improve communication efficiency. The experimental results show that this method makes the simulation time of the cortical microcircuit model with STDP synapse scale of 20% shortened by about 64.5% compared with the standard cyclic allocation algorithm, and about 57.4% compared with the ordinary supermap allocation algorithm. At the same time, the communication time is reduced by more than 90%. [ABSTRACT FROM AUTHOR]

Increased chip densities offer massive computation power to deal with fundamental big data operations such as searching and sorting. At the same time, the proliferation of processing elements (PEs) in such multicore chips together with the employment of more aggressive parallel algorithms cause the amount of space needed for interprocessor communications to dominate the overall chip space, potentially resulting in reduced computational efficiency. To overcome this issue, this paper introduces a new architecture that uses simple crosspoint switches to pair PEs instead of a complex interconnection network. This new architecture may be viewed as a 'quadratic' array of processors as it uses O (n 2) PEs rather than O (n) PEs as in linear array processor models. The switches between adjacent PEs are created using a cyclic permutation wiring idea with n (n − 1) / 2 + 1 PEs and as many crosspoints. We demonstrate the versatility of this new parallel architecture by designing fast algorithms to sort and search a list of n elements with it. In particular, we show that finding a minimum, maximum, and searching a list of n elements can all be performed on this parallel architecture in O (1) time with additional O (n) elementary logic gates with O (n) fan-in and in O (lg ⁡ n) time with O (1) fan-in. We further show that sorting a list of n elements can also be carried out in O (1) time using O (n) additional elementary logic gates with O (n) fan-in and threshold logic gates on the same parallel architecture. The sorting time increases to O ((lg ⁡ n) lg ⁡ lg ⁡ n) if only elementary logic gates with O (1) fan-in are used. In addition, we establish how similar queries can be handled within the same order of time complexities. We use this new parallel architecture to perform sorting and searching on big data on three different models. The first of these models provides an efficient implementation of enumeration sorting and searching for moderate size big data sets. The second model offers increased parallelism by replication of the new parallel architecture but its hardware complexity limits its use to moderate size big data sets as well. The third model removes this limitation by introducing a tradeoff parameter between the time and hardware complexity of the overall computation, thereby providing an optimal use of available resources within a given chip-set space. [ABSTRACT FROM AUTHOR]

The digital twin city seeks to restore virtually the geographic scene that is close to reality and provide the audience an immersive experience. Three dimensional (3D) real urban scene is one of the key technologies to achieve the goal. Nowadays, the combination of geographic information system and game engine becomes an industrial trend and provides new opportunities for 3D real urban scene. Oblique photogrammetric data, as being a fundamental data source for 3D urban modeling, is able to represent large-scale and high-precision scene. The game engine plays an excellent role in offering realistic effects and post-production systems that are greatly useful for geospatial data visualization. However, the diversity of specifications and formats of oblique photogrammetric data has caused many different data loading and transformation methods, resulting in a problem to be solved in 3D real urban scene. Methods: This paper aims to propose a data transformation approach from oblique photogrammetric data to rendering resources in game engine. To stress, the proposed approach can allow us to load instantly massive amount of oblique photogrammetric data in Unreal Engine, which is one of the most popular game engines. We further compared oblique photogrammetric data with rendering resources in game engine and discussed the significance of such a data transformation. The approach reorganizes the original structure of the data by parsing data into a tree structure that better suits transformation purposes. Also, the proposed approach transforms respectively the geometry and texture data when iterating tree nodes, as well as considers the essential physical properties of rendering assets. All these features are combined into a package data format. Results: In the case study of Futian district, Shenzhen, the results show that the proposed approach successfully converts oblique photogrammetric data to rendering resources in Unreal Engine 4. In the transformation process, the geometrical structure and collision properties of triangular mesh are guaranteed, and color, bump and other visual elements of original data are kept in materials. The rendering resources are loaded to simulate the real urban scenarios at different time periods of a day in Unreal Engine 4. Conclusions: This paper devises a novel approach which can convert the oblique photogrammetric data into rendering resource for a game engine to load and render, and meantime maintain the consistency of data's geometry and texture. For its scalability, it also supports parallel processing of large-scale data transformation when data is organized in the same directory. The proposed approach may be of great theoretical and practical value for other related 3D model data transformation problems. [ABSTRACT FROM AUTHOR]

In order to solve the problem of accounting data encryption processing behavior uncontrollable, accounting data encryption processing based on virtual reality and multimedia technology is proposed in medical sector. In this article, according to the actual topology form of BP neural network, the encryption processing module and data reading and writing interface module are connected as needed, and the hardware execution environment of encryption system based on neural network is built in medical sector. On this basis, the chaotic mapping relationship is defined, and the encryption execution process is improved by authenticating the parallel identity processing of accounting data, so as to build the execution environment of application system software. Combined with the structure system of hardware equipment at all levels, the design of accounting data security encryption system based on neural network is completed. The experimental results show that the initial value of the memory ratio index in the experimental group is completely consistent with the initial value of the ideal curve. When the amount of accounting data input is equal to 60 Mb, the difference between the memory ratio index in the experimental group and the ideal index is the smallest, only 1.2%. Conclusion: This method not only solves the problem of uncontrollable accounting data encryption behavior in medical sector, but also realizes the secure transmission and processing of accounting data encryption information. [ABSTRACT FROM AUTHOR]

As phenomics data volume and dimensionality increase due to advancements in sensor technology, there is an urgent need to develop and implement scalable data processing pipelines. Current phenomics data processing pipelines lack modularity, extensibility, and processing distribution across sensor modalities and phenotyping platforms. To address these challenges, we developed PhytoOracle (PO), a suite of modular, scalable pipelines for processing large volumes of field phenomics RGB, thermal, PSII chlorophyll fluorescence 2D images, and 3D point clouds. PhytoOracle aims to (i) improve data processing efficiency; (ii) provide an extensible, reproducible computing framework; and (iii) enable data fusion of multi-modal phenomics data. PhytoOracle integrates open-source distributed computing frameworks for parallel processing on high-performance computing, cloud, and local computing environments. Each pipeline component is available as a standalone container, providing transferability, extensibility, and reproducibility. The PO pipeline extracts and associates individual plant traits across sensor modalities and collection time points, representing a unique multi-system approach to addressing the genotype-phenotype gap. To date, PO supports lettuce and sorghum phenotypic trait extraction, with a goal of widening the range of supported species in the future. At the maximum number of cores tested in this study (1,024 cores), PO processing times were: 235 minutes for 9,270 RGB images (140.7 GB), 235 minutes for 9,270 thermal images (5.4 GB), and 13 minutes for 39,678 PSII images (86.2 GB). These processing times represent end-to-end processing, from raw data to fully processed numerical phenotypic trait data. Repeatability values of 0.39-0.95 (bounding area), 0.81-0.95 (axis-aligned bounding volume), 0.79-0.94 (oriented bounding volume), 0.83-0.95 (plant height), and 0.81-0.95 (number of points) were observed in Field Scanalyzer data. We also show the ability of PO to process drone data with a repeatability of 0.55-0.95 (bounding area). [ABSTRACT FROM AUTHOR]

The microseismic signals released by rock mass fracture can be captured via microseismic monitoring to evaluate the development of geological disasters. This is crucial for underground engineering construction, underground mining, and earthquake and geological disaster evaluation. However, extracting information effectively is difficult due to the low signal-to-noise ratio of microseismic signals caused by complex environmental factors. Therefore, denoising and detection (onset time picking) are essential to processing microseismic signals and extracting source information. To improve the efficiency and accuracy of microseismic signal processing, we propose a parallel dual-tasking network, which is an advanced deep learning model that can simultaneously perform microseismic denoising and detection tasks. The network, comprising one encoder and two parallel decoders, is customised to extract input data features, and two outputs can be simultaneously generated to denoise and detect microseismic signals. The model exhibits excellent denoising and detection performance for microseismic signals containing various types of noise. Compared with traditional methods, the signal-to-noise ratio of the denoised signal is greatly improved, and the waveform distortion of the denoised signal is small. Even when the signal-to-noise ratio is low, the proposed model can maintain good onset time pickup performance. This method obviates the need for different denoising methods for different types of noise and precludes setting thresholds artificially to improve the denoising effect and detection accuracy. Moreover, the dual processing characteristics of the model facilitate simultaneous denoising and detection, which improves the processing efficiency of microseismic data and meets the demand for automatically processing massive microseismic data. Therefore, this method has excellent data processing potential in exploration seismology, and earthquake and disaster assessment. [ABSTRACT FROM AUTHOR]

In recent years, one of the biggest concerns of researchers has been environmental knowledge. This concern can be resolved by collecting and processing remote sensing data in the shortest possible time and cost with the highest accuracy and efficiency. In remote sensing, various types of satellite data are processed for different purposes and applications. In this process, data storage and processing methods, resource management, scalability, performance improvement, and efficiency are among the issues and challenges in this scope. This paper presents a service-oriented framework using big data and parallel processing in remote sensing to address these challenges. The proposed framework provides scalability, flexibility, and generalization without dependency on specific data or processing techniques. In addition, it provides reasonable results to quality criteria such as response time, efficiency, and performance. The evaluation results of the proposed framework show the effectiveness of the framework for various types of analysis of remote sensing data with acceptable accuracy. [ABSTRACT FROM AUTHOR]

Conventional machine learning techniques may have lesser performance when they deal with complex data. For addressing this issue, it is important to build data mining frameworks coupled with robust knowledge discovery mechanisms. One of such frameworks, which addresses these issues is ensemble learning. It fuses data, builds models and mines data into a single framework. In spite of the work done on ensemble learning, there remain issues like how to manage the complexity, how to optimize the model, and how to fine-tune the model. Natural data processing schemes use parallel processing and are robust and efficient, hence are successful. Taking a cue from natural data processing architectures, we propose a parallelized CNN tree ensemble approach. The proposed approach is compared against the baseline which is the deep network used in the ensemble. The ResNet50 architecture is utilized for initial experimentation. The datasets used for this task are the ImageNet and natural images datasets. The proposed approach outperforms the baseline on all experiments on the ImageNet dataset. Further, benchmarking of the proposed approach against different types of CNNs is done on various datasets including CIFAR-10, CIFAR-100, Fashion-MNIST, FEI face recognition, and MNIST digits. Since our approach is adaptable for CNNs, it outperforms the baseline CNNs as well as the state-of-the-art techniques on these datasets. The CNNs architectures used for benchmarking are ResNet-50, DenseNet, WRN-28-10 and NSGANetV1. The code for the paper is available in https://github.com/mueedhafiz1982/CNNTreeEnsemble.git. [ABSTRACT FROM AUTHOR]

The best region search (BRS) is one of the major research problems in geospatial data processing applications. The BRS problem objective is to discover the ideal location of a particular size specified rectangle, with a predetermined end goal of maximizing the user-defined scoring function. The existing solutions for finding the top-k best regions have focused on designing algorithms for centralized settings. These solutions are not suitable for processing massive datasets. In this paper, we enable a Hadoop MapReduce-based parallel and distributed computation to obtain significant improvement in the performance. In addition to the parallel and distributed setting, we also incorporate early pruning strategies to eliminate the need to process rectangles that are not part of the output to minimize the communication cost involved in computing k-BRS. We later introduced a redistribution strategy over the initially proposed methodology that handles skew inherited from the dataset. Our results are obtained from extensive experimentation, both synthetic and real-world datasets. [ABSTRACT FROM AUTHOR]

Featured Application: Controlling access to a large set of medical and social media data processed in a cloud environment using open-source Hadoop storage and Spark parallel processing. There is an increasing demand for processing large volumes of unstructured data for a wide variety of applications. However, protection measures for these big data sets are still in their infancy, which could lead to significant security and privacy issues. Attribute-based access control (ABAC) provides a dynamic and flexible solution that is effective for mediating access. We analyzed and implemented a prototype application of ABAC to large dataset processing in Amazon Web Services, using open-source versions of Apache Hadoop, Ranger, and Atlas. The Hadoop ecosystem is one of the most popular frameworks for large dataset processing and storage and is adopted by major cloud service providers. We conducted a rigorous analysis of cybersecurity in implementing ABAC policies in Hadoop, including developing a synthetic dataset of information at multiple sensitivity levels that realistically represents healthcare and connected social media data. We then developed Apache Spark programs that extract, connect, and transform data in a manner representative of a realistic use case. Our result is a framework for securing big data. Applying this framework ensures that serious cybersecurity concerns are addressed. We provide details of our analysis and experimentation code in a GitHub repository for further research by the community. [ABSTRACT FROM AUTHOR]

Data-intensive computing frameworks typically split job workload into fixed-size chunks, allowing them to be processed as parallel tasks on distributed machines. Ideally, when the machines are homogeneous and have identical speed, chunks of equal size would finish processing at the same time. However, such determinism in processing time cannot be guaranteed in practice. Diverging processing times can result from various sources such as system dynamics, machine heterogeneity, and variable network conditions. Such variation, together with dynamics and uncertainty during task processing, can lead to significant performance degradation at job level, due to long tails in job completion time resulted from residual chunk workload and stragglers. In this paper, we propose Forseti , a novel processing scheme that is able to reshape data chunk size on the fly with respect to heterogeneous machines and a dynamic execution environment. Forseti mitigates residual workload and stragglers to achieve significant improvement in performance. We note that Forseti is a fully online scheme and does not require any a priori knowledge of the machine configuration nor job statistics. Instead, it infers such information and adjusts data chunk sizes at runtime, making the solution robust even in environments with high volatility. In its implementation, Forseti also exploits a virtual machine reuse feature to avoid task start-up and initialization cost associated with launching new tasks. We prototype Forseti on a real-world cluster and evaluate its performance using several realistic benchmarks. The results show that Forseti outperforms a number of baselines, including default Hadoop by up to 68% and SkewTune by up to 50% in terms of average job completion time. • Forseti enables efficient data processing by reshaping data chunk sizes on the fly. • Forseti mitigates residual workload and stragglers to improve performance. • Forseti does not require any a priori knowledge. • Forseti infers tasks statistics and adjusts data chunk sizes at runtime. • Forseti exploits a virtual machine reuse feature to avoid task start-up time. [ABSTRACT FROM AUTHOR]

Regarding parallel processing, a study brings a parallel computing approach by integrating low-cost microcomputers. In the given research, the main aspects linked to parallel processing and its motivation are exposed, widespread today by the need for data mining integrated with large-scale databases, prominent data characteristics, large volumes of data, lack of agile processing for real-time analysis, and high variability of data. For a given proposal, a case study scenario is addressed, where the database composed of more than twelve years of continuous records was processed, requiring processing and processing of the data present for the feasibility of a machine learning project. In the study, the main gains of the implemented system are presented, mainly the use of only 18% of the time required if it was considered a traditional model in the processing of the base. At the end of the article, considerations under the research and proposals for future works are presented. [ABSTRACT FROM AUTHOR]

• This article proposed a new complete straggler mitigation classification taxonomy that classifies the techniques into proactive and reactive approaches based on the previously proposed strategies by the researchers and industrial experts. • It also discusses the homogeneity assumptions made by Hadoop MapReduce, which was never being discussed in earlier literature studies. • Further, it has performed a detailed comparative study of different strategies with the year, avoidance, detection and prediction criteria, nodes used, testing environments, results, etc. • Finally, it has analyzed various articles based on the proposed classification taxonomy and presented the graph studies carried out in each category. It has proposed numerous hidden open issues for straggler mitigation and management in a heterogeneous Hadoop environment with the potential directions for possible future work. Hadoop is the most economical and cheap software framework that allows distributed storage and parallel processing of more extensive data sets. Hadoop distributed file system (HDFS) allows distributed storage and parallel processing of vast data sets using MapReduce. However, Hadoop's current implementation believes that computing nodes connected in a cluster are homogeneous and distribute the tasks equally. This equal load distribution creates the load imbalance during storage, resource contention during task scheduling, hardware degradation by its excess use, and software misconfiguration during cluster management which are the leading causes of stragglers in heterogeneous Hadoop clusters. Due to hardware heterogeneity, Hadoop's performance degrades in the heterogeneous environment. In our study, the paper reviews and analyzes significant studies. It presents the new classification taxonomy to broadly classify the existing straggler management and mitigation techniques into two approaches: proactive and reactive. It analyses and compares the state of art studies and identifies their limitation based on their results. Finally, the systematic review discusses the open issues and the potential directions for future work to manage and mitigate stragglers from the heterogeneous Hadoop clusters. [ABSTRACT FROM AUTHOR]

It is well known that network-based parallel data processing algorithms are well suited to implementation in reconfigurable hardware recurring to either Field-Programmable Gate Arrays (FPGA) or Programmable Systems-on-Chip (PSoC). The intrinsic parallelism of these devices makes it possible to execute several data-independent network operations in parallel. However, the approaches to designing the respective systems vary significantly with the experience and background of the engineer in charge. In this paper, we analyze and compare the pros and cons of using an embedded processor, high-level synthesis methods, and register-transfer low-level design in terms of design effort, performance, and power consumption for implementing a parallel algorithm to find the two smallest values in a dataset. This problem is easy to formulate, has a number of practical applications (for instance, in low-density parity check decoders), and is very well suited to parallel implementation based on comparator networks. [ABSTRACT FROM AUTHOR]

Learning mappings of data on manifolds is an important topic in contemporary machine learning, with applications in astrophysics, geophysics, statistical physics, medical diagnosis, biochemistry, and 3D object analysis. This paper studies the problem of learning real-valued functions on manifolds through filtered hyperinterpolation of input–output data pairs where the inputs may be sampled deterministically or at random and the outputs may be clean or noisy. Motivated by the problem of handling large data sets, it presents a parallel data processing approach which distributes the data-fitting task among multiple servers and synthesizes the fitted sub-models into a global estimator. We prove quantitative relations between the approximation quality of the learned function over the entire manifold, the type of target function, the number of servers, and the number and type of available samples. We obtain the approximation rates of convergence for distributed and non-distributed approaches. For the non-distributed case, the approximation order is optimal. [ABSTRACT FROM AUTHOR]

Fog computing can be an effective way to improve quality of services and solve network problems, with the demand for real-time, latency-sensitive applications increasing as well as limitations such as network bandwidth and Internet of Things users' resources. Due to the fact that different tasks in a network can create overhead that can reduce the quality of service, dynamic voltage, and frequency scaling along with a ranking function and a high number of physical and virtual machines were used in this research. A profit function phase is used to analyze network tasks in order to improve QoS by sending them to physical machines and sending them via the network to physical machines. The simulation results demonstrate that this method is the most effective in allocating radio and computational resources to IoT devices in fog computing. A comparison is presented in the results section between the proposed method and the SPA, Markov-Fog, and TRAM methods. Criteria for evaluating performance include the response time for heterogeneous environments, energy consumption against tasks and users, memory processing, energy consumption for physical and virtual machines, and network profitability. [ABSTRACT FROM AUTHOR]

Parallel processing of large spatial datasets over distributed systems has become a core part of modern data analytic systems like Apache Hadoop and Apache Spark. The general-purpose design of these systems does not natively account for the data's spatial attributes and results in poor scalability, accuracy, or prolonged runtimes. Spatial extensions remedy the problem and introduce spatial data recognition and operations. At the core of a spatial extension, a locality-preserving spatial partitioner determines how to spatially group the dataset's objects into smaller chunks using the distributed system's available resources. Existing spatial extensions rely on data sampling and often mismanage non-spatial data by either overlooking their memory requirements or excluding them entirely. This work discusses the various challenges that face spatial data partitioning and proposes a novel spatial partitioner for effectively processing spatial queries over large spatial datasets. For evaluation, the proposed partitioner is integrated with the well-known k-Nearest Neighbor (k NN) spatial join query. Several experiments evaluate the proposal using real-world datasets. Our approach differs from existing proposals by (1) accounting for the dataset's unique spatial traits without sampling, (2) considering the computational overhead required to handle non-spatial data, (3) minimizing partition shuffles, (4) computing the optimal utilization of the available resources, and (5) achieving accurate results. This contributes to the problem of spatial data partitioning through (1) providing a comprehensive discussion of the problems facing spatial data partitioning and processing, (2) the development of a novel spatial partitioning technique for in-memory distributed processing, (3) an effective, built-in, load-balancing methodology that reduces spatial query skews, and (4) a Spark-based implementation of the proposed work with an accurate k NN spatial join query. Experimental tests show up to 1.48 times improvement in runtime as well as the accuracy of results. [ABSTRACT FROM AUTHOR]

With the rapid growth of Internet of Things (IoT) based models, and the lack amount of data makes cloud computing resources insufficient. Hence, edge computing-based techniques are becoming more popular in present research domains that makes data storage, and processing effective at the network edges. There are several advanced features like parallel processing and data perception are available in edge computing. Still, there are some challenges in providing privacy and data security over networks. To solve the security issues in Edge Computing, Hash-based Message Authentication Code (HMAC) algorithm is used to provide solutions for preserving data from various attacks that happens with the distributed network nature. This paper proposed a Trust Model for Secure Data Sharing (TM-SDS) with HMAC algorithm. Here, data security is ensured with local and global trust levels with the centralized processing of cloud and by conserving resources effectively. Further, the proposed model achieved 84.25% of packet delivery ratio which is better compared to existing models in the resulting phase. The data packets are securely transmitted between entities in the proposed model and results showed that proposed TM-SDS model outperforms the existing models in an efficient manner. [ABSTRACT FROM AUTHOR]

Simulated quantum annealing (SQA) is a probabilistic approximation method to find a solution for a combinatorial optimization problem using a digital computer. It is possible to simulate large-scale optimization problems on a CPU due to its high external memory capacity. However, the processing time increases exponentially with the number of variables, and parallel implementation is difficult due to the serial nature of the quantum Monte Carlo algorithm used in SQA. In this paper, we propose a method to accelerate SQA on a multicore CPU, based on temporal and spatial parallel processing with high data localization. According to the experimental results using 16-core CPU, we achieved from 8 to 16 times speedup compared to single-core CPU implementations. The proposed method can be used to solve combinatorial optimization problems that have more than 64,000 variables, which was not possible using previous GPU- and FPGA-based accelerators. [ABSTRACT FROM AUTHOR]

This paper introduces the Hopsworks platform to the entire Earth Observation (EO) data community and the Copernicus programme. Hopsworks is a scalable data-intensive open-source Artificial Intelligence (AI) platform that was jointly developed by Logical Clocks and the KTH Royal Institute of Technology for building end-to-end Machine Learning (ML)/Deep Learning (DL) pipelines for EO data. It provides the full stack of services needed to manage the entire life cycle of data in ML. In particular, Hopsworks supports the development of horizontally scalable DL applications in notebooks and the operation of workflows to support those applications, including parallel data processing, model training, and model deployment at scale. To the best of our knowledge, this is the first work that demonstrates the services and features of the Hopsworks platform, which provide users with the means to build scalable end-to-end ML/DL pipelines for EO data, as well as support for the discovery and search for EO metadata. This paper serves as a demonstration and walkthrough of the stages of building a production-level model that includes data ingestion, data preparation, feature extraction, model training, model serving, and monitoring. To this end, we provide a practical example that demonstrates the aforementioned stages with real-world EO data and includes source code that implements the functionality of the platform. We also perform an experimental evaluation of two frameworks built on top of Hopsworks, namely Maggy and AutoAblation. We show that using Maggy for hyperparameter tuning results in roughly half the wall-clock time required to execute the same number of hyperparameter tuning trials using Spark while providing linear scalability as more workers are added. Furthermore, we demonstrate how AutoAblation facilitates the definition of ablation studies and enables the asynchronous parallel execution of ablation trials. [ABSTRACT FROM AUTHOR]

The extensive usage of social media polarity analysis claims the need for real-time analytics and runtime outcomes on dashboards. In data analytics, only 30% of the time is consumed in modeling and evaluation stages and 70% is consumed in data engineering tasks. There are lots of machine learning algorithms to achieve a desirable outcome in prediction points of view, but they lack in handling data and their transformation so-called data engineering tasks, and reducing its time remained still challenging. The contribution of this research paper is to encounter the mentioned challenges by presenting a parallelly, scalable, effective, responsive and fault-tolerant framework to perform end-to-end data analytics tasks in real-time and batch-processing manner. An experimental analysis on Twitter posts supported the claims and signifies the benefits of parallelism of data processing units. This research has highlighted the importance of processing mentioned URLs and embedded images along with post content to boost the prediction efficiency. Furthermore, this research additionally provided a comparison of naive Bayes, support vector machines, extreme gradient boosting and long short-term memory (LSTM) machine learning techniques for sentiment analysis on Twitter posts and concluded LSTM as the most effective technique in this regard. [ABSTRACT FROM AUTHOR]

Educational big data analysis is facilitated by the significant amount of unstructured data found in education institutions. Python has various toolkits for both structured and unstructured data processing. However, its ability for processing large-scale data is limited. On the other hand, Spark is a big data processing framework, but it does not have the needed toolkits for processing unstructured rich text documents, 3D model and image processing. In this study, we develop a generic framework that integrates Python toolkits and Spark based on service-oriented architecture. The framework automatically extends the serial algorithm written in Python to distributed algorithm to accomplish parallel processing tasks seamlessly. First, our focus is on achieving non-intrusive deployment to Spark servers and how to run Python codes in Spark environment to process rich text documents. Second, we propose a compression-based schema to address the poor performance of small sized files in HDFS. Finally, we design a generic model that can process different types of poly-structured data such as 3D models and images. We published the services used in the system for sharing them at https level for constructing different systems. It is evaluated through simulation experiments using large-scale rich text documents, 3D models and images. According to the results, the speedup is 49 times faster than the standalone Python-docx in the simulations of extracting 232 GB docx files when eight physical nodes with 128 cores are used. It reaches about 89 times after further compression schema is applied. In addition, simulations for 3D model descriptors' extraction show that the simulation achieves a speedup of about 116 times. In the large-scale image's HOG features extraction simulation task of up to 256.7 GB (6,861,024 images), a speedup of up to 110 times is achieved. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The deep learning (DL) training process consists of multiple phases—data augmentation, training, and validation of the trained model. Traditionally, these phases are executed either on the central processing units or graphics processing units in a serial fashion due to lack of additional computing resources to offload independent phases of DL training. Recently, Mellanox/NVIDIA introduced the BlueField-2 data processing units (DPUs), which combine the advanced capabilities of traditional application-specific-integrated-circuit-based network adapters with an array of ARM processors. In this article, we explore how to take advantage of the additional ARM cores on the BlueField-2 DPUs. We propose and evaluate multiple novel designs to efficiently offload the phases of DL training to the DPUs. Our experimental results show that the proposed designs are able to deliver up to 17.5% improvement in overall DL training time. To the best of our knowledge, this is the first work to explore the use of DPUs to accelerate DL training. [ABSTRACT FROM AUTHOR]

Support for efficient spatial data storage and retrieval has become a vital component in almost all spatial database systems. While GPUs have become a mainstream platform for high-throughput data processing in recent years, exploiting the massively parallel processing power of GPUs is non-trivial. Current approaches that parallelize one query at a time have low work efficiency and cannot make good use of GPU resources. On the other hand, many spatial database systems could receive a large number of queries simultaneously. In this paper, we present a comprehensive framework named G-PICS for parallel processing of concurrent spatial queries on GPUs. G-PICS encapsulates efficient parallel algorithms for constructing a variety of spatial trees with different space partitioning methods. G-PICS also provides highly optimized programs for processing major spatial query types, and such programs can be accessed via an API that could be further extended to implement user-defined algorithms. While support for dynamic data inputs is missing in existing work, G-PICS implements efficient parallel algorithms for bulk updates of data. Furthermore, G-PICS is designed to work in a Multi-GPU environment to support datasets beyond the size of a single GPU’s global memory. Empirical evaluation of G-PICS shows significant performance improvement over the state-of-the-art GPU and parallel CPU-based spatial query processing systems. In particular, G-PICS achieves double-digit speedup over such systems in tree construction (up to 53X) and query processing (up to 80X). [ABSTRACT FROM AUTHOR]

With the rapid growth of artificial intelligence (AI), the Internet of Things (IoT) and big data, emerging applications that cross stacks with different techniques bring new challenges to parallel computing systems. These cross-stack functionalities require one system to possess multiple characteristics, such as the ability to process data under high throughput and low latency, the ability to carry out iterative and incremental computation, transparent fault tolerance, and the ability to perform heterogeneous tasks that evolve dynamically. However, high-performance computing (HPC) and big data computing, as two categories of parallel computing architecture, are incapable of meeting all these requirements. Therefore, by performing a comparative analysis of HPC and big data computing from the perspective of the parallel programming model layer, middleware layer, and infrastructure layer, we explore the design principles of the two architectures and discuss a converged architecture to address the abovementioned challenges. [ABSTRACT FROM AUTHOR]

Spin waves (SWs) have tremendous application potential in wave-based computation utilizing a broad frequency spectrum spanning from the gigahertz-to-terahertz ranges. Like optical and other electromagnetic waves, SWs also promise to usher in a new era of parallel data processing with low-power consumption without Joule heating. However, this potential is undermined by the lack of investigation on multichannel networking and operation on single chips under a uniform bias magnetic field. This study proposes a multifrequency SW propagation based on shape anisotropy in microstructured rectangular waveguides made of yttrium iron garnet (YIG). The width-dependent transmission properties of magnetostatic surface SWs in the YIG waveguides were experimentally demonstrated. We revealed that the smaller width of the waveguide results in lower SWs frequency due to the demagnetizing field along the width direction. Multifrequency SW propagation was demonstrated in a device where three waveguides with widths of 10, 20, and $100~\mu \text{m}$ were connected to the common antennas. SWs propagation with the frequencies of 1.98, 2.11, and 2.18 GHz have been transmitted under a uniform bias magnetic field. Furthermore, we investigated SWs transmission in a device where three waveguides with different widths were interconnected at their ends and under one side of the antenna. We observed that the interconnected waveguides result in a single resonant frequency with flat band transmission because the whole waveguide structure is considered a single magnetic body. The results presented here provide guidelines for complex networks in frequency-division multiplexing operation. [ABSTRACT FROM AUTHOR]

In this paper, we consider the system of residual classes for data representation in communication channels. In this case, it is possible to perform parallel data processing without taking into account digit transfers. However, the most of communication channels are serial; therefore, we also consider the possibility of serial transmission of parallel data. We show that the use of the system of residual classes of Mersenne numbers as a base does not lead to decreasing the data transmission rate. We estimate the amount of information contained in the positions of the numbers of a positional code. [ABSTRACT FROM AUTHOR]

Hive and Impala queries are used to process a big amount of data. The overwriting amount of information requires an efficient data processing system. When we deal with a long-term batch query and analysis Hive will be more suitable for this query. Impala is the most powerful system suitable for real- time interactive Structured Query Language (SQL) query which are added a massive parallel processing to Hadoop distributed cluster. The data growth makes a problem with SQL Cluster because the execution processing time is increased. In this paper, a comparison is demonstrated between the performance time of Hive, Impala and SQL on two different data models with different queries chosen to test the performance. The results demonstrate that Impala outperforms Hive and SQL cluster when it comes to analyze data and processing tasks. Using two benchmark datasets, TPC-H and statistical computing, we compare the performance of Hive, Impala, and SQL clusters 2009 Statistical Graphics Data Expo. [ABSTRACT FROM AUTHOR]

The traditional distributed database storage architecture has the problems of low efficiency and storage capacity in managing data resources of seafood products. We reviewed various storage and retrieval technologies for the big data resources. A block storage layout optimization method based on the Hadoop platform and a parallel data processing and analysis method based on the MapReduce model are proposed. A multireplica consistent hashing algorithm based on data correlation and spatial and temporal properties is used in the parallel data processing and analysis method. The data distribution strategy and block size adjustment are studied based on the Hadoop platform. A multidata source parallel join query algorithm and a multi-channel data fusion feature extraction algorithm based on data-optimized storage are designed for the big data resources of seafood products according to the MapReduce parallel frame work. Practical verification shows that the storage optimization and data-retrieval methods provide supports for constructing a big data resource-management platform for seafood products and realize efficient organization and management of the big data resources of seafood products. The execution time of multidata source parallel retrieval is only 32% of the time of the standard Hadoop scheme, and the execution time of the multichannel data fusion feature extraction algorithm is only 35% of the time of the standard Hadoop scheme. [ABSTRACT FROM AUTHOR]

In order to improve the effect of agricultural disaster management, this paper uses environmental big data technology to analyse historical agricultural disaster data and analyse the role and impact of this technology in agricultural disaster management. How to efficiently use the effective resources in the meteorological big data to optimise the mining effect is an important issue in the assessment of agrometeorological disasters. Parallel information entropy attribute reduction algorithm can effectively reduce the scale of research data and increase the rate of data preprocessing. The classification assessment and early warning of agrometeorological disasters involve data mining classification prediction technology. Unlike most classification prediction algorithms, KNN classification is particularly suitable for application to meteorological data sets with cross-class domains and can accelerate the classification and prediction speed of big data in the parallel processing architecture. Finally, this paper designs experiments to verify the method proposed in this paper. The results of data analysis show that the model constructed in this paper has a certain effect. [ABSTRACT FROM AUTHOR]