Search

Obesity is a major public health problem worldwide, and the prevalence of childhood obesity is of particular concern. Effective interventions for preventing and treating childhood obesity aim to change behaviour and exposure at the individual, community, and societal levels. However, monitoring and evaluating such changes is very challenging. The EU Horizon 2020 project "Big Data against Childhood Obesity (BigO)" aims at gathering large-scale data from a large number of children using different sensor technologies to create comprehensive obesity prevalence models for data-driven predictions about specific policies on a community. It further provides real-time monitoring of the population responses, supported by meaningful real-time data analysis and visualisations. Since BigO involves monitoring and storing of personal data related to the behaviours of a potentially vulnerable population, the data representation, security, and access control are crucial. In this paper, we briefly present the BigO system architecture and focus on the necessary components of the system that deals with data access control, storage, anonymisation, and the corresponding interfaces with the rest of the system. We propose a three-layered data warehouse architecture: The back-end layer consists of a database management system for data collection, de-identification, and anonymisation of the original datasets. The role-based permissions and secured views are implemented in the access control layer. Lastly, the controller layer regulates the data access protocols for any data access and data analysis. We further present the data representation methods and the storage models considering the privacy and security mechanisms. The data privacy and security plans are devised based on the types of collected personal, the types of users, data storage, data transmission, and data analysis. We discuss in detail the challenges of privacy protection in this large distributed data-driven application and implement novel privacy-aware data analysis protocols to ensure that the proposed models guarantee the privacy and security of datasets. Finally, we present the BigO system architecture and its implementation that integrates privacy-aware protocols.

The BIG Data Center at Beijing Institute of Genomics (BIG) of the Chinese Academy of Sciences provides a suite of database resources in support of worldwide research activities in both academia and industry. With the vast amounts of multi-omics data generated at unprecedented scales and rates, the BIG Data Center is continually expanding, updating and enriching its core database resources through big data integration and value-added curation. Resources with significant updates in the past year include BioProject (a biological project library), BioSample (a biological sample library), Genome Sequence Archive (GSA, a data repository for archiving raw sequence reads), Genome Warehouse (GWH, a centralized resource housing genome-scale data), Genome Variation Map (GVM, a public repository of genome variations), Science Wikis (a catalog of biological knowledge wikis for community annotations) and IC4R (Information Commons for Rice). Newly released resources include EWAS Atlas (a knowledgebase of epigenome-wide association studies), iDog (an integrated omics data resource for dog) and RNA editing resources (for editome-disease associations and plant RNA editosome, respectively). To promote biodiversity and health big data sharing around the world, the Open Biodiversity and Health Big Data (BHBD) initiative is introduced. All of these resources are publicly accessible at http://bigd.big.ac.cn.

AbstractWith the rapid development of data acquisition and storage space, massive datasets exhibited with large sample size emerge increasingly and make more advanced statistical tools urgently need. To accommodate such big volume in the analysis, a variety of methods have been proposed in the circumstances of complete or right censored survival data. However, existing development of big data methodology has not attended to interval-censored outcomes, which are ubiquitous in cross-sectional or periodical follow-up studies. In this work, we propose an easily implemented divide-and-combine approach for analyzing massive interval-censored survival data under the additive hazards model. We establish the asymptotic properties of the proposed estimator, including the consistency and asymptotic normality. In addition, the divide-and-combine estimator is shown to be asymptotically equivalent to the full-data-based estimator obtained from analyzing all data together. Simulation studies suggest that, relative to the full-data-based approach, the proposed divide-and-combine approach has desirable advantage in terms of computation time, making it more applicable to large-scale data analysis. An application to a set of interval-censored data also demonstrates the practical utility of the proposed method. [ABSTRACT FROM AUTHOR]

Big data has attracted extensive attention from industries and universities during the past few years. Big data is crucial in many fields, such as business analytics, healthcare, the Internet of Things (IoT), smart home, supply chain, transportation, and fraud detection. Nevertheless, some challenges must be addressed, such as decentralization, integration, data sharing, privacy, and security. On the other hand, blockchain has numerous potential for improving big data services and applications because of its decentralized, verifiable, and anti‐tamper features. We apply the Systematic Literature Review (SLR) approach in this study to investigate blockchain integrity in big data and understand the different topics and significant areas already presented. This paper aims to analyze studies on blockchain integration in big data published between 2017 and 2022. A technical taxonomy is presented for blockchain integration into big data, including data storage, security, and applications based on the field of articles selected by the SLR method. Finally, the achievements and shortcomings of each study are discussed, and future research challenges and open issues related to blockchain integration in big data are highlighted. [ABSTRACT FROM AUTHOR]

Parallel power loads anomalies are processed by a fast-density peak clustering technique that capitalizes on the hybrid strengths of Canopy and K-means algorithms all within Apache Mahout's distributed machine-learning environment. The study taps into Apache Hadoop's robust tools for data storage and processing, including HDFS and MapReduce, to effectively manage and analyze big data challenges. The preprocessing phase utilizes Canopy clustering to expedite the initial partitioning of data points, which are subsequently refined by K-means to enhance clustering performance. Experimental results confirm that incorporating the Canopy as an initial step markedly reduces the computational effort to process the vast quantity of parallel power load abnormalities. The Canopy clustering approach, enabled by distributed machine learning through Apache Mahout, is utilized as a preprocessing step within the K-means clustering technique. The hybrid algorithm was implemented to minimise the length of time needed to address the massive scale of the detected parallel power load abnormalities. Data vectors are generated based on the time needed, sequential and parallel candidate feature data are obtained, and the data rate is combined. After classifying the time set using the canopy with the K-means algorithm and the vector representation weighted by factors, the clustering impact is assessed using purity, precision, recall, and F value. The results showed that using canopy as a preprocessing step cut the time it proceeds to deal with the significant number of power load abnormalities found in parallel using a fast density peak dataset and the time it proceeds for the k-means algorithm to run. Additionally, tests demonstrate that combining canopy and the K-means algorithm to analyze data performs consistently and dependably on the Hadoop platform and has a clustering result that offers a scalable and effective solution for power system monitoring. [ABSTRACT FROM AUTHOR]

Metasurface holographic encryption, as an emerging and highly promising technique, presents the advantages of multi‐dimension and high security and consequently provides a flexible and convenient platform for information encryption. Recent years have witnessed rapid progress in meta‐holographic encryption through the utilization of different dimensions of electromagnetic (EM) waves, yet enhancing the capacity and security of encrypted information remains an endless goal. In this context, the concept of three physical dimensions encrypted metasurface holography is proposed in which orbital angular momentum (OAM) is fully synergized with frequency and polarization state. The mechanism relies on the simultaneous control of these dimensions and arbitrarily overlaying them in each operation channel, thereby acting as a multi‐dimensional key to realize information encryption and decryption. As a proof‐of‐concept experiment, a three‐physical dimension encrypted meta‐hologram is designed that conceals four real images as encrypted information in 32 information channels. Remarkably, a specific OAM order, frequency, and polarization state is required to decrypt the designed meta‐hologram. This work opens a new avenue for multi‐dimensional encrypted holography, which has broad application prospects for information security and data storage in the era of big data. [ABSTRACT FROM AUTHOR]

Quantile regression offers a powerful means of understanding the comprehensive relationship between response variables and predictors. By formulating prior domain knowledge and assumptions as constraints on parameters, the estimation efficiency can be enhanced. This paper studies some methods based on multi-block ADMM (Alternating Direction Method of Multipliers) to fit general penalized quantile regression models with linear constraints on regression coefficients. Different formulations for handling linear constraints and general penalties are explored and compared. Among these formulations, the most efficient one is identified, which provides an explicit expression for each parameter during iterations and eliminates the nested-loop in existing algorithms. Furthermore, this work addresses the challenges posed by big data by developing a parallel ADMM algorithm suitable for distributed data storage. The algorithm's convergence and a robust stopping criterion are established. To demonstrate the excellent performance of the proposed algorithms, extensive numerical experiments and a real data example are presented. These empirical validations showcase the effectiveness of the methods in handling complex datasets. The details of theoretical proofs and different algorithm variations are provided in the Appendix. [ABSTRACT FROM AUTHOR]

The data warehousing process requires an architectural revolution to settle big-data challenges and address new data sources, such as social networks, recommendation systems, smart cities and the web to extract value from shared data. In this respect, the pipeline-modeling community for the acquisition, storage and processing of data for analysis purposes is enacting a wide range of technological solutions that present significant challenges and difficulties. More specifically, the choice of the most appropriate tool for the user’s specific business needs and the interoperability between the different tools have become primary challenges. From this perspective, we propose in this paper a new interactive framework based on machine learning (ML) techniques to assist experts in the process of modeling a customized pipeline for data warehousing. More precisely, we elaborate first (i) an analysis of the experts’ requirements and the characteristics of the data to be processed, then (ii) we propose the most appropriate architecture to their requirements from a multitude of specific architectures instantiated from a generic one, by using (iii) several ML methods to predict the most suitable tool for each phase and task within the architecture. Additionally, our framework is validated through two real-world use cases and user feedback. [ABSTRACT FROM AUTHOR]

At present, the aging of the population and the increase of health problems caused by lifestyle, sports rehabilitation has become an important means to improve the quality of life and restore physical function. However, traditional rehabilitation methods often rely on rehabilitation centers or hospitals, which are not only costly, but also greatly reduce the efficiency and accessibility of rehabilitation due to time and space limitations. The rapid development of wearable technology and big data analysis technology provides the possibility for real-time, accurate and personalized rehabilitation monitoring. Therefore, this paper focuses on the architecture and design of wearable sports rehabilitation monitoring system based on big data technology, and explains in detail sensor data acquisition and processing analysis, cloud computing and data storage, as well as user interface and interaction design. Finally, the results of two groups of simulation experiments are as follows: the accuracy rate of sports rehabilitation data increases by 11.9%, and the rehabilitation effect increases by about 20.8% on average. Through long-term data collection, the system helps researchers and doctors better understand the complex dynamics of the rehabilitation process, providing a scientific basis for future rehabilitation methods. [ABSTRACT FROM AUTHOR]

As mobile internet and Internet of Things technologies rapidly advance, the amount of spatio-temporal big data have surged, and efficient and secure management solutions are urgently needed. Although cloud storage provides convenience, it also brings significant data security challenges. Blockchain technology is an ideal choice for processing large-scale spatio-temporal big data due to its unique security features, but its storage scalability is limited because the data need to be replicated throughout the network. To solve this problem, a common approach is to combine blockchain with off-chain storage to form a hybrid storage blockchain. However, these solutions cannot guarantee the authenticity, integrity, and consistency of on-chain and off-chain data storage, and preprocessing is required in the setup phase to generate public parameters proportional to the data length, which increases the computational burden and reduces transmission efficiency. Therefore, this paper proposes a collaborative storage mechanism for spatio-temporal big data based on incremental aggregation sub-vector commitments, which uses vector commitment binding technology to ensure the secure storage of on-chain and off-chain data. By generating public parameters of fixed length, the computational complexity is reduced and the communication efficiency is improved while improving the security of the system. In addition, we design an aggregation proof protocol that integrates aggregation algorithms and smart contracts to improve the efficiency of data query and verification and ensure the consistency and integrity of spatio-temporal big data storage. Finally, simulation experiments verify the correctness and security of the proposed protocol, providing a solid foundation for the blockchain-based spatio-temporal big data storage system. [ABSTRACT FROM AUTHOR]

In an era where AI-driven decision-making is becoming increasingly important, the surge in data generation across different sectors poses significant scalability challenges for big data processing. This study delves into these challenges, aiming to enhance our understanding and management of large data volumes. It begins by stressing the critical importance of scalability in big data processing, highlighting its necessity for the functionality of data-centric applications today. The main challenges to scalability are then examined in the article, including problems with data storage, processing speed, and resource allocation. In order to assess how well distributed computing frameworks like MapReduce, Apache Spark, and Apache Hadoop can handle the growing needs of data processing, it offers a comprehensive examination of their performance. The study also broadens its scope to address how containerization and cloud computing technologies can help mitigate scaling issues. This research aims to provide an extensive overview of current technologies, frameworks, and techniques in order to tackle the scalability issues in big data processing in a complete manner. It seeks to ensure more efficient data processing techniques in the era of abundant information by advancing AI-driven decision-making capabilities in the face of expanding data quantities. [ABSTRACT FROM AUTHOR]

Hospitals around the world are deploying increasingly advanced systems to collect and store high-resolution physiological patient data for quality improvement and research. However, data accuracy, completeness, consistency, and contextual validity remain issues. This report highlights a data artifact known as waveform clipping in our hospital's physiological data capture system that went unnoticed for years, limiting data analysis and delaying several research projects. We aim to raise awareness in the medical informatics community about the importance of careful system setup, ongoing data validation, and close cooperation between clinicians and data scientists. [ABSTRACT FROM AUTHOR]

The transition to smart manufacturing introduces heightened complexity in regard to the machinery and equipment used within modern collaborative manufacturing landscapes, presenting significant risks associated with equipment failures. The core ambition of smart manufacturing is to elevate automation through the integration of state-of-the-art technologies, including artificial intelligence (AI), the Internet of Things (IoT), machine-to-machine (M2M) communication, cloud technology, and expansive big data analytics. This technological evolution underscores the necessity for advanced predictive maintenance strategies that proactively detect equipment anomalies before they escalate into costly downtime. Addressing this need, our research presents an end-to-end platform that merges the organizational capabilities of data warehousing with the computational efficiency of Apache Spark. This system adeptly manages voluminous time-series sensor data, leverages big data analytics for the seamless creation of machine learning models, and utilizes an Apache Spark-powered engine for the instantaneous processing of streaming data for fault detection. This comprehensive platform exemplifies a significant leap forward in smart manufacturing, offering a proactive maintenance model that enhances operational reliability and sustainability in the digital manufacturing era. [ABSTRACT FROM AUTHOR]

Streaming big data presents unique security challenges due to its real-time generation and distributed transmission methods, making it vulnerable to security threats such as data leakage and tampering. To address these challenges, we propose a secure transmission and storage scheme for streaming big data based on a data lake architecture. Our approach leverages an ECC lightweight encryption algorithm in a streaming data encryption interceptor to filter and encrypt key information at the data source. We also introduce the SSL secure transmission protocol to ensure secure data transmission over a multi-source streaming data transmission channel constructed using Flume and Kafka. Furthermore, we design a data partition scheme based on LZO compression and implement a data lake storage system using Hadoop. Our proposed scheme can efficiently and securely transfer streaming data from multiple data sources to the data lake, while providing high data query efficiency. Experimental results show that our stream data encryption interceptor reduces memory load by 18% and filters and encrypts key information at a faster speed compared to similar schemes. In addition, our data lake storage scheme demonstrates lower data write latency and space occupancy, making it well-suited for large-scale streaming data applications. [ABSTRACT FROM AUTHOR]

Copyright of Baghdad Science Journal is the property of Republic of Iraq Ministry of Higher Education & Scientific Research (MOHESR) and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Efficiency is a crucial factor when handling the retrieval and storage of data from vast amounts of records in a Big Data repository. These systems require a subset of data that can be accommodated within the combined physical memory of a cluster of servers. It becomes impractical to analyze all of the data if its size exceeds the available memory capacity. Retrieving data from virtual storage, primarily hard disks, is significantly slower compared to accessing data from main memory, resulting in increased access time and diminished performance. To address this, a proposed model aims to enhance performance by identifying the most suitable data locality structure within a big data set and reorganizing the data schema accordingly; by locality, it has been referred to as a particular access pattern. This allows transactions to be executed on data residing in the fastest memory layer, such as cache, main memory, or disk cache. [ABSTRACT FROM AUTHOR]

This paper presents a comprehensive literature review on the evolution of data-lake technology, with a particular focus on data-lake architectures. By systematically examining the existing body of research, we identify and classify the major types of data-lake architectures that have been proposed and implemented over time. The review highlights key trends in the development of data-lake architectures, identifies the primary challenges faced in their implementation, and discusses future directions for research and practice in this rapidly evolving field. We have developed diagrammatic representations to highlight the evolution of various architectures. These diagrams use consistent notations across all architectures to further enhance the comparative analysis of the different architectural components. We also explore the differences between data warehouses and data lakes. Our findings provide valuable insights for researchers and practitioners seeking to understand the current state of data-lake technology and its potential future trajectory. [ABSTRACT FROM AUTHOR]

Exploratory data analysis (EDA), data visualization, and visual analytics are essential for understanding and analyzing complex datasets. In this project, we explored these techniques and their applications in data analytics. The case discusses Tableau, a powerful data visualization tool, and Google BigQuery, a cloud-based data warehouse that enables users to store, query, and analyze large datasets. It also explored the benefits and applications of both tools and their integration with other platforms and services. The project offers an introductory insight into Tableau's functionalities, employing a data file from the US Census Bureau via Google BigQuery. [ABSTRACT FROM AUTHOR]

Copyright of China Sciencepaper is the property of China Sciencepaper and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Big Data and Cloud Computing are pivotal technologies in today's landscape, addressing the exponential growth of data and providing flexible, cost-effective platforms for users. Integrating these technologies offers significant efficiency and knowledge advantages across industries like Finance, Management, and supply chain. This paper explores the fusion of Big Data and Cloud Computing, leveraging the 4 V's (Volume, Velocity, Variety, Veracity) of Big Data to enhance performance, insights, detail, and security while reducing costs. It delves into analytics, technologies, challenges, trends, and security considerations, showcasing the potential and benefits of this integration. [ABSTRACT FROM AUTHOR]

Big data is growing fast following Big data has the characteristics of high volume, speed, and variety (3v) and continues to grow exponentially following the development of the world's use of information and communication technology. The main problem with big data is the data deluge. The need for technology and large data storage and processing methods to keep pace with the exponential data growth rate is potentially limitless, giving rise to the problem of exponentially increasing technology requirements as well. The weakness of previous big data analysis approaches (batch and online real time processing) is that it requires high technology (large storage, memory and processing). This paper proposes a new approach in big-data analysis by separating the basic knowledge (BK) construction process from original data with a much smaller volume (volume reduction), then analyzing it into final knowledge, thus requiring smaller/simpler analysis technology. The proposals include formulating the definition and representation of BK, developing methods for constructing BK from source data, and analyzing BK into final knowledge. We propose a BK construction method based on a knowledge extraction technique using BIRCH clustering algorithm for instance reduction, and handling high-dimensional problems by parallelizing the dimension calculation process to calculate the distance between instances. We use the Adjusted Rand Index (ARI) to measure the similarity of the final knowledge of the baseline and proposed methods. First, the BIRCH baseline was modified by parallelizing the calculations has succeeded in increasing speed from 17% to 25%. Next, the parallel BIRCH (PBIRCH) baseline was broken into BK construction and BK analysis, has succeeded in reducing volume by 96% or more and increasing speed by 43.50%, with similar final knowledge results (ARI=1). Based on these results, we conclude that the BK construction method and analysis from BK into final knowledge for highdimensional big data have significantly reduced volume and speed up the analytical process without reducing the quality of the final knowledge. [ABSTRACT FROM AUTHOR]

Enormous information is an accumulation of vast measures of information that contains copied things gathered from different sources. Distinguishing copied information from enormous information is a difficult issue. In this paper secure deduplication scheme has been presented for huge information stockpiling. Customers may store and transmit data effectively and efficiently due to cloud service providers. Providers of cloud services are drawn to the application of data de-duplication tools to lower storage costs and conserve bandwidth. To secure the data they share on the cloud, users of the cloud are interested in having a secure and private connection. As a result, the priory is transferred to the cloud, and the data is encrypted. Because the encryption objective conflicts with the purpose of data deduplication, this feature presents a difficult problem. The author then suggested brand-new hybrid chunking methods to boost deduplication performance in cloud storage. Content-based hybrid chunking algorithm of Two Threshold Two Divisor (TTTD) with a Dynamic Prime Coding (DPC) algorithm is suggested for reducing the deduplication data. To increase throughput, which means reducing execution time and CPU resource utilisation, an index method must be incorporated into the deduplication matching process. For chunks with higher security levels, it offers greater security protection, while for chunks with lower security levels, it trades off security for greater deduplication effectiveness. When moving data to open, semi-trusted systems like public clouds, identity management, secure data exchange, and privacy preservation become more challenging than they are in closed, trusted systems where security and privacy can be controlled more simply. Accordingly, the article proposed a cross-domain authentication technique for the verification process. Evaluations of the real-world datasets show the efficiency of the design. With other methods, the proposed hybrid Content-based TTTD-DPC analyses many performance parameters, including chunk distribution, processing time, chunking time, throughput, average chunk size, and deduplication ratio. The deduplication system's total throughput and processing speed are both enhanced by the proposed approach. In addition, the recommended methodology reduces computational cost (772 ms) in terms of hash function compared to other existing methods. [ABSTRACT FROM AUTHOR]

The exponential growth of digital data has created new challenges and opportunities to process, store, and analyze large datasets. Cloud computing has transformed the way businesses manage their IT infrastructure by providing a scalable and cost-effective platform for data storage and processing. Because of this, businesses now have access to new opportunities to exploit big data and thus acquire insights into the behavior of their customers, increase the efficiency of their operations, and drive innovation. Furthermore, big data analysis enables the exploration of links between a set of independent data, revealing many aspects that enable the prediction of correct decisions that can aid the achievement of desired goals. With this in mind, the purpose of this article is to investigate the ways in which big data analysis can be put to use in fields such as healthcare, presenting specific instances of effective use along the way. The purpose of this study is to provide a comprehensive overview of the opportunities and application areas offered by big data and cloud computing, as well as to highlight the need for businesses and organizations to adopt these technologies to maintain their competitive edge in the digital age. Furthermore, this paper emphasizes the importance of big data analysis in facilitating decision-making and goal achievement, and it encourages businesses and organizations to adopt these technologies to stay competitive in an increasingly data-driven world. It also discusses the ethical, legal, and security issues that arise when dealing with large amounts of data, as well as ways to address these challenges. [ABSTRACT FROM AUTHOR]

Biological big data are a massive amount of data generated from multi-omics experiments, such as genomics, transcriptomics, proteomics, metabolomics, phenomics, glycomics, epigenomics, and other omics. These data are used to study biological processes and to gain insights into how living systems work. It can also be used to develop new treatments for diseases and understand the causes of certain conditions. The storage and analysis of these data present several challenges owing to their sheer size and complexity. Storing these data efficiently requires a large amount of storage space and processing power. Furthermore, there are certain limitations in terms of the kind of insights that can be gained from multi-omics data because of their complexity. Despite these challenges, biological big data offers great potential for advancing our understanding of biology and developing new treatments for diseases. Big-data research is a rapidly growing field, with numerous applications. As the amount of data continues to increase, it is important to understand its storage, utility, limitations, and challenges. In this review article, various sources of big-data research and their storage capacities, limitations, and challenges are discussed. Factors affecting the data quality and accuracy have been reported. It will be helpful for researchers to understand the available big data in biology for their further utilization and integration into novel discovery. [ABSTRACT FROM AUTHOR]

The growing importance of data analytics is leading to a shift in data management strategy at many companies, moving away from simple data storage towards adopting Online Analytical Processing (OLAP) query analysis. Concurrently, NoSQL databases are gaining ground as the preferred choice for storing and querying analytical data. This article presents a comprehensive, systematic mapping, aiming to consolidate research efforts related to the integration of OLAP with NoSQL databases in Big Data environments. After identifying 1646 initial research studies from scientific digital repositories, a thorough examination of their content resulted in the acceptance of 22 studies. Utilizing the snowballing technique, an additional three studies were selected, culminating in a final corpus of twenty-five relevant articles. This review addresses the growing importance of leveraging NoSQL databases for OLAP query analysis in response to increasing data analytics demands. By identifying the most commonly used NoSQL databases with OLAP, such as column-oriented and document-oriented, prevalent OLAP modeling methods, such as Relational Online Analytical Processing (ROLAP) and Multidimensional Online Analytical Processing (MOLAP), and suggested models for batch and real-time processing, among other results, this research provides a roadmap for organizations navigating the integration of OLAP with NoSQL. Additionally, exploring computational resource requirements and performance benchmarks facilitates informed decision making and promotes advancements in Big Data analytics. The main findings of this review provide valuable insights and updated information regarding the integration of OLAP cubes with NoSQL databases to benefit future research, industry practitioners, and academia alike. This consolidation of research efforts not only promotes innovative solutions but also promises reduced operational costs compared to traditional database systems. [ABSTRACT FROM AUTHOR]

Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems. [ABSTRACT FROM AUTHOR]

For solving the storage and indexing problem of big data, this paper proposes a big data storage algorithm and an indexing algorithm, respectively. Simultaneously, the data classification and segmentation algorithm is given, and then, the classified data are segmented and sliced based on features of data. Moreover, the calculation of the service queue length on data slicing is given to achieve fast storage of slices in the data server. A data reconstruction method is given to index each slice of the original data and implement the data reconstruction according to the given reconstruction method. At the same time, an evaluation method for data reconstruction is also given to achieve the minimum difference between the reconstructed data and the original data. Simulation results show that the accurate rate of the data reconstruction method proposed in this paper is higher than 93%. In addition, compared with the existing data reconstruction methods, it is found that it not only has higher reconstruction accurate rate, faster reconstruction process, stronger anti-interference ability, but also lower information loss. These researches in this paper provide the powerful methods for protecting important documents, reducing losses, reducing waste of resources, and improving the convenience, speed, and safety in work, life, and so on. [ABSTRACT FROM AUTHOR]

The supply chain functions as a complex web, interconnecting various stakeholders such as suppliers, manufacturers, wholesalers, retailers, and ultimately, end consumers. Central to effective supply chain management is the meticulous handling of inventory, a critical factor influencing both cost efficiency and service excellence throughout the entire network. The management of inventory within this context extends beyond the confines of individual enterprises, bearing significance across the entirety of the supply chain. Consequently, achieving optimal performance necessitates a cohesive, holistic approach to management, aligning with the overarching objectives of the system. Through dynamic data analysis of multiple types of power materials, a dynamic inventory control model for power materials is constructed to achieve optimal adjustment of inventory management. Ultimately, a multi-granularity inventory control method based on big data analysis of power warehousing is constructed, which effectively improves inventory management efficiency and reduces logistics management costs for enterprises. Through big data analysis of power material warehousing, the characteristics of power material demand are excavated, and a classification method for power material demand is constructed to achieve an overall inventory control strategy for power materials. The implementation results show that the controlled inventory can better meet the changing demand, thereby improving inventory management efficiency. The multi-granularity inventory control method based on big data mining of warehousing combines inventory and multiobjective optimization theories, proves the applicability and feasibility of the proposed method, effectively improves inventory management efficiency, reduces logistics management costs for enterprises, and provides practical guidance and decision-making reference for improving the intensive management level of power production and maintenance materials. [ABSTRACT FROM AUTHOR]

Purpose: Although the challenges associated with big data are increasing, the question of the most suitable big data analytics (BDA) platform in libraries is always significant. The purpose of this study is to propose a solution to this problem. Design/methodology/approach: The current study identifies relevant literature and provides a review of big data adoption in libraries. It also presents a step-by-step guide for the development of a BDA platform using the Apache Hadoop Ecosystem. To test the system, an analysis of library big data using Apache Pig, which is a tool from the Apache Hadoop Ecosystem, was performed. It establishes the effectiveness of Apache Hadoop Ecosystem as a powerful BDA solution in libraries. Findings: It can be inferred from the literature that libraries and librarians have not taken the possibility of big data services in libraries very seriously. Also, the literature suggests that there is no significant effort made to establish any BDA architecture in libraries. This study establishes the Apache Hadoop Ecosystem as a possible solution for delivering BDA services in libraries. Research limitations/implications: The present work suggests adapting the idea of providing various big data services in a library by developing a BDA platform, for instance, providing assistance to the researchers in understanding the big data, cleaning and curation of big data by skilled and experienced data managers and providing the infrastructural support to store, process, manage, analyze and visualize the big data. Practical implications: The study concludes that Apache Hadoops' Hadoop Distributed File System and MapReduce components significantly reduce the complexities of big data storage and processing, respectively, and Apache Pig, using Pig Latin scripting language, is very efficient in processing big data and responding to queries with a quick response time. Originality/value: According to the study, there are significantly fewer efforts made to analyze big data from libraries. Furthermore, it has been discovered that acceptance of the Apache Hadoop Ecosystem as a solution to big data problems in libraries are not widely discussed in the literature, although Apache Hadoop is regarded as one of the best frameworks for big data handling. [ABSTRACT FROM AUTHOR]

Rapid processing with low-latency and high-throughput is a critical requirement for the applications of big data streams. However, the interferences among stream processing tasks in a data center decrease the utilization of the computational resources and prolong the latency of the tasks. Thus, we study an optimal scheduling method for processing a big data stream on heterogeneous servers with multicores in a data center. We model the big data stream processing and the scheduling problem with four objects or factors which are streaming data items, processing tasks, computational nodes and the cores inside each computational node. An interference model based on regression analysis and a prediction model based on the Autoregressive Integrated Moving Average are presented. Then, we propose a two-stage scheduling method including the fine-grained core scheduling and the coarse-grained node scheduling. In the core scheduling stage, we design a core scheduling algorithm named CS_TDF. In the node scheduling stage, we design a node scheduling algorithm named NS_ITF for a single time window and a continuous scheduling algorithm named PS_UIM for the entire data stream in all time windows. The experimental results show that our scheduling method achieves low interference and high computational resource utilization. [ABSTRACT FROM AUTHOR]

Analytical data platforms have been used for decades to improve organizational performance. Starting from the data warehouses used primarily for structured data processing, through the data lakes oriented for raw data storage and post-hoc data analyses, to the data lakehouses—a combination of raw storage and business intelligence pre-processing for improving the platform's efficacy. But in recent years, a new architecture called Data Mesh has emerged. The main promise of this architecture is to remove the barriers between operational and analytical teams in order to boost the overall value extraction from the big data. A number of attempts have been made to formalize and implement it in existing projects. Although being defined as a socio-technical paradigm, data mesh still lacks the technology support to enable its widespread adoption. To overcome this limitation, we propose a new view of the platform requirements alongside the formal governance definition that we believe can help in the successful adoption of the data mesh. It is based on fundamental aspects such as decentralized data domains and federated computational governance. In addition, we also present a blockchain-based implementation of a mesh platform as a practical validation of our theoretical proposal. Overall, this article demonstrates a novel research direction for information system decentralization technologies. [ABSTRACT FROM AUTHOR]

Big data technologies have proliferated since the dawn of the cloud-computing era. Traditional data storage, extraction, transformation, and analysis technologies have thus become unsuitable for the large volume, diversity, high processing speed, and low value density of big data in medical strategies, which require the development of novel big data application technologies. In this regard, we investigated the most recent big data platform breakthroughs in anesthesiology and designed an anesthesia decision model based on a cloud system for storing and analyzing massive amounts of data from anesthetic records. The presented Anesthesia Decision Analysis Platform performs distributed computing on medical records via several programming tools, and provides services such as keyword search, data filtering, and basic statistics to reduce inaccurate and subjective judgments by decision-makers. Importantly, it can potentially to improve anesthetic strategy and create individualized anesthesia decisions, lowering the likelihood of perioperative complications. [ABSTRACT FROM AUTHOR]

The proliferation of social networks, the Internet of Things, and economic mobility has led to an exponential increase in data. New data having high volume, high velocity, high variety, and high value are called big data. Big data present additional requirements in terms of storage and computation resources. Various enterprises aim to outsource their big data services to the cloud because of its cost efficiency, less management, resource pooling, and resilient computing. However, outsourcing the storage of sensitive data can expose them to potential security risks. Encryption presents a straightforward solution for data privacy preserving. In traditional encryption mechanisms, such as advanced encryption standard, the data owner and users must share an exact key for both data encryption and decryption. Currently, these mechanisms do not provide a scalable and secure solution for big data storage and analysis. Furthermore, they need to be more efficient to support big data velocity. Unfortunately, securing outsourced big data storage to a public cloud environment to later maintain efficient and secure processing over encrypted data by cloud servers cannot be ensured using traditional encryption mechanisms. In this paper, we propose a security approach for this issue by which honest but curious users or cloud service providers cannot reach complete information from the stored data. From the analysis, the proposed approach can provide secure cloud-assisted big data. Meanwhile, the performance evaluation shows the efficiency of the proposed approach. [ABSTRACT FROM AUTHOR]

In the current landscape, staying abreast of the latest technological advancements is a formidable challenge, especially given the deluge of data inundating the internet. The realization has dawned that effectively managing the surge in emerging data necessitates the integration of multiple technologies. In pursuit of this objective, the Internet of Things (IoT), renowned for its sensor‐based data capture capabilities, is frequently coupled with blockchain technology to ensure secure data storage and access. This amalgamation, in turn, leverages the cloud environment when data volume surpasses a machine's processing capacity, thus mitigating infrastructure and maintenance costs. This study endeavors to optimize data storage within the blocks of the blockchain (BCT) by storing an index that points to the actual data. This innovative approach not only conserves storage space but also enhances operational efficiency. Furthermore, it simplifies the task of identifying malicious or faulty nodes deployed at different locations for data capture within the prescribed time frame. To exemplify this implementation, a case study is presented, focusing on securing user votes through the creation of contracts. The results showcased underscore the preference for a permissioned blockchain, such as Fabric, over a permissionless one, like Ethereum, particularly in the context of security considerations. The findings reveal that as the number of operations (in this case, votes cast) increases, Ethereum's performance deteriorates, while Fabric exhibits exceptional robustness. Additionally, the study analyzes sensor data simulated via IoT nodes before and after the application of security algorithms to underscore the significance of the proposed Secure Cloud‐Based Blockchain (SCB2) model. The analysis encompasses various facets, including the creation, validation, and computation times of transactions and blocks within a node, and positions the model favorably in comparison to existing literature. [ABSTRACT FROM AUTHOR]

With the development of Industry 4.0 and big data technology, the Industrial Internet of Things (IIoT) is hampered by inherent issues such as privacy, security, and fault tolerance, which pose certain challenges to the rapid development of IIoT. Blockchain technology has immutability, decentralization, and autonomy, which can greatly improve the inherent defects of the IIoT. In the traditional blockchain, data is stored in a Merkle tree. As data continues to grow, the scale of proofs used to validate it grows, threatening the efficiency, security, and reliability of blockchain-based IIoT. Accordingly, this paper first analyzes the inefficiency of the traditional blockchain structure in verifying the integrity and correctness of data. To solve this problem, a new Vector Commitment (VC) structure, Partition Vector Commitment (PVC), is proposed by improving the traditional VC structure. Secondly, this paper uses PVC instead of the Merkle tree to store big data generated by IIoT. PVC can improve the efficiency of traditional VC in the process of commitment and opening. Finally, this paper uses PVC to build a blockchain-based IIoT data security storage mechanism and carries out a comparative analysis of experiments. This mechanism can greatly reduce communication loss and maximize the rational use of storage space, which is of great significance for maintaining the security and stability of blockchain-based IIoT. [ABSTRACT FROM AUTHOR]

The diversification of business forms leads to functional and spatial complexity in cities. The efficient determination of the complexity of an urban system is the basis for the scientific monitoring of the multi-functional aggregation within cities. Previous studies on the urban spatial structure were limited by the difficulty of collecting micro-data and the high time cost, and they focused on the macro-spatial structure, lacking fine-grained investigations of the micro-spatial structure. Additionally, high-resolution remote sensing images, which mainly rely on the textural characteristics of the spectrum of ground objects, cannot detect the social and economic functions of ground objects. Thus, it is difficult to meet the actual needs of urban planning and management. The purpose of this paper is to automatically identify the spatial heterogeneity and temporal variation of urban land use functions in the context of complex urban systems. The TF-IDF (term frequency–inverse document frequency) algorithm, a machine learning classification algorithm, and other methods are applied to identify the urban functions and distribution characteristics of the main urban area based on the POI (point of interest) data and urban form data. The results show the following: (1) From 2012 to 2022, all types of land use in Yiwu city grew at different rates, with logistics and warehousing space growing the fastest, which is in line with Yiwu's goal of building a national logistics center for trade and services. (2) The residential area has a spatial structure with a dense central circle and a scattered periphery extending from northeast to southwest and from east to west. (3) The commercial service sector shows clear spatial differentiation between the core and the periphery. The commercial functional areas of Niansanli, Houzhai, and Chengxi, where the number of commercial POIs is relatively small, are located at the intersection of the administrative subdistricts near the city center, indicating that the commercial economic activities of the downtown subdistrict have a certain spillover effect on adjacent subdistricts. (4) The public facilities of each subdistrict are generally located in the core of each subdistrict, which ensures better convenience and accessibility. (5) Industrial land with a large total area that is scattered and mixed with urban residential land gradually tends to be centralized, forming an industrial belt around the city. This study comprehensively considers the aggregation relationship between urban buildings and land use and improves the accuracy of land identification and functional zoning. [ABSTRACT FROM AUTHOR]

Modern organizations are currently wrestling with strenuous challenges relating to the management of heterogeneous big data, which combines data from various sources and varies in type, format, and content. The heterogeneity of the data makes it difficult to analyze and integrate. This paper presents big data warehousing and federation as viable approaches for handling big data complexity. It discusses their respective advantages and disadvantages as strategies for integrating, managing, and analyzing heterogeneous big data. Data integration is crucial for organizations to manipulate organizational data. Organizations have to weigh the benefits and drawbacks of both data integration approaches to identify the one that responds to their organizational needs and objectives. This paper aw well presents an adequate analysis of these two data integration approaches and identifies challenges associated with the selection of either approach. Thorough understanding and awareness of the merits and demits of these two approaches are crucial for practitioners, researchers, and decision-makers to select the approach that enables them to handle complex data, boost their decision-making process, and best align with their needs and expectations. [ABSTRACT FROM AUTHOR]

Master–slave blockchain is a novel information processing technology that is domain-oriented and uses efficient cryptography principles for trustworthy communication and storage of big data. Existing indexing methods primarily target the creation of a single-structured blockchain, resulting in extensive time and memory requirements. As the scale of domain data continues to grow exponentially, master–slave blockchain systems face increasingly severe challenges with regards to low query efficiency and extended traceback times. To address these issues, this paper propose a multi-level index construction method for the master–slave blockchain (MLI). Firstly, MLI introduces a weight matrix and partitions the entire master–slave blockchain based on the master chain structure, the weight of each partition is assigned. Secondly, for the master blockchain in each partition, a master chain index construction method based on jump consistent hash (JHMI) is proposed, which takes the key value of the nodes and the number of index slots as input and outputs the master chain index. Finally, a bloom filter is introduced to improve the column-based selection function and build a secondary composite index on the subordinate blockchain corresponding to each master block. Experimental results on three constraint conditions and two types of datasets demonstrate that the proposed method reduce the index construction time by an average of 9.28%, improve the query efficiency by 12.07%, and reduce the memory overhead by 24.4%. [ABSTRACT FROM AUTHOR]

One of the problems in the blockchain is the formation of increasingly large data (big data) because each block must store all the transactions it makes. With the problem of the appearance of extensive data (big data), many studies aim to maintain the data in small amounts. This research combines a sorting data technique and a proper compression technique to obtain efficient data storage on the blockchain. The result of this research is a blockchain platform called Adaptive Shrink and Shard Blockchain (AS2BC), which conceptually and computationally can minimize the use of storage space in the blockchain up to 22 times smaller. [ABSTRACT FROM AUTHOR]

The big data era demands an efficient and permanent data storage technology with the capacity of PB to EB scale. Optical data storage (ODS) offers a good candidate for long-lifetime storage, as the developing far-field super-resolution nanoscale writing technology improves its capacity to the PB scale. However, methods to efficiently read out this intensive ODS data are still lacking. In this paper, we demonstrate a sub-diffraction readout method based on polarization modulation, which experimentally achieves the sub-diffraction readout on Disperse Red 13 thin film with a resolution of 500 nm, exceeding the diffraction limit by 1.2 times (NA = 0.5). Differing from conventional binary encoding, we propose a specific polarization encoding method that enhances the capacity of ODS by 1.5 times. In the simulation, our method provides an optical data storage readout resolution of 150 nm, potentially to 70 nm, equivalent to 1.1 PB in a DVD-sized disk. This sub-diffraction readout method has great potential as a powerful readout tool for next-generation optical data storage. [ABSTRACT FROM AUTHOR]

In today's digital landscape, the exponential growth of data heightens security risks associated with traditional centralized storage systems. Utilizing blockchain technology, a shift towards decentralized data storage provides a more secure and private alternative. Central to our work is the exploration of symmetry in data management, a concept woven into the fabric of our proposed solution to challenge the inherency in InterPlanetary File System (IPFS) technology. Through the strategic utilization of smart contract-invoked random functions, our blockchain-based solution fragments and securely stores data in order to ensure a symmetrical balance between confidentiality and integrity. Our research endeavors are to contribute a robust, ethically grounded data storage framework fostering advancements in secure data sharing. The implications of this paper are significant in addressing contemporary challenges of data management within the expansive realm of big data. [ABSTRACT FROM AUTHOR]

Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract–Clean–Load–Transform (ECLT), designed for processing unstructured big data, with specific emphasis on text. The model aims to reduce execution time through experimental procedures. ECLT is applied and tested using Spark, which is a framework employed in Python. Finally, this paper compares the execution time of ECLT with different models by applying two datasets. Experimental results showed that for a data size of 1 TB, the execution time of ECLT is 41.8 s. When the data size increases to 1 million articles, the execution time is 119.6 s. These findings demonstrate that ECLT outperforms ETL, ELT, DELT, ELTL, and ELTA in terms of execution time. [ABSTRACT FROM AUTHOR]

With the creation of big data and the increasing adoption of cloud computing, secure and efficient storage solutions for big data in a cloud environment are of paramount importance. This paper presents a novel approach for achieving both efficiency and security in the storage of big data by utilizing Triple Data Encryption Standard (3DES). The proposed solution leverages the strength of 3DES, which applies the Data Encryption Standard (DES) algorithm three times sequentially, to provide a higher level of security. Additionally, to ensure efficiency in the cloud environment. The existing privacy preserving approaches showed several disadvantages such as a lack of data privacy and accurate data analysis, a lack of efficiency of performance and reliance on a third party, which considers the susceptibility of security and reveals the secret data of the individual. The outcomes demonstrate storage and privacy achieved by the proposed solution, making it a promising system for the storage of big data in cloud environments as compare to existing approaches. [ABSTRACT FROM AUTHOR]

In recent years, cloud computing technologies have been developed rapidly in this computing world to provide suitable on-demand network access all over the world. A cloud service provider offers numerous types of cloud services to the user. But the most significant issue is how to attain optimal virtual machine (VM) allocation for the user and design an efficient big data storage platform thereby satisfying the requirement of both the cloud service provider and the user. Therefore, this paper presents two novel strategies for optimizing VM resource allocation and cloud storage. An optimized cloud cluster storage service is introduced in this paper using a binarization based on modified fuzzy c-means clustering (BMFCM) algorithm to overcome the negative issues caused by the repetitive nature of the big data traffic. The BMFCM algorithm utilized can be implemented transparently and can also address problems associated with massive data storage. The VM selection is optimized in the proposed work using a hybrid COOT-reverse cognitive fruit fly (RCFF) optimization algorithm. The main aim of this algorithm is to improve the massive big data traffic and storage locality. The CPU utilization, VM power, memory dimension and network bandwidth are taken as the fitness function of the hybrid COOT-RCFF algorithm. When implemented in CloudSim and Hadoop, the proposed methodology offers improvements in terms of completion time, overall energy consumption, makespan, user provider satisfaction and load ratio. The results show that the proposed methodology improves the execution time and data retrieval efficiency by up to 32% and 6.3% more than the existing techniques. [ABSTRACT FROM AUTHOR]