Search

Encrypted deduplication combines encryption and deduplication in a seamless way to provide confidentiality guarantees for the physical data in deduplicated storage, yet it incurs substantial metadata storage overhead due to the additional storage of keys. We present a new encrypted deduplication storage system called Metadedup, which suppresses metadata storage by also applying deduplication to metadata. Its idea builds on indirection, which adds another level of metadata chunks that record metadata information. We find that metadata chunks are highly redundant in real-world workloads and hence can be effectively deduplicated. We further extend Metadedup to incorporate multiple servers via a distributed key management approach, so as to provide both fault-tolerant storage and security gaurantees. We extensively evaluate Metadedup from performance and storage efficiency perspectives. We show that Metadedup achieves high throughput in writing and restoring files, and saves the metadata storage by up to 93.94% for real-world backup workloads.

In-package DRAM-based Last-Level-Caches (LLCs) that cache data in small chunks (i.e., blocks) are promising for improving system performance due to their efficient main memory bandwidth utilization. However, in these high-capacity DRAM caches, managing metadata (i.e., tags) at low cost is challenging. Storing the tags in SRAM has the advantage of quick tag access but is impractical due to a large area overhead. Storing the tags in DRAM reduces the area overhead but incurs tag serialization latency for an associative LLC design, which is inevitable for achieving high cache hit rate. To address the area and latency overhead problem, we propose a block-based DRAM LLC design that decouples tag and data into two regions in DRAM. Our design stores the tags in a latency-optimized DRAM region as the tags are accessed more often than the data. In contrast, we optimize the data region for area efficiency and map spatially-adjacent cache blocks to the same DRAM row to exploit spatial locality. Our design mitigates the tag serialization latency of existing associative DRAM LLCs via selective in-DRAM tag comparison, which overlaps the latency of tag and data accesses. This efficiently enables LLC bypassing via a novel DRAM Absence Table (DAT) that not only provides fast LLC miss detection but also reduces in-package bandwidth requirements. Our evaluation using SPEC2006 benchmarks shows that our tag-data decoupled LLC improves system performance by 11.7 percent compared to a state-of-the-art direct-mapped LLC design and by 7.2 percent compared to an existing associative LLC design.

Metadata management constitutes a key prerequisite for enterprises as they engage in data analytics and governance. Today, however, the context of data is often only manually documented by subject matter experts, and lacks completeness and reliability due to the complex nature of data pipelines. Thus, collecting data lineage—describing the origin, structure, and dependencies of data—in an automated fashion increases quality of provided metadata and reduces manual effort, making it critical for the development and operation of data pipelines. In our practice report, we propose an end-to-end solution that digests lineage via (Py‑)Spark execution plans. We build upon the open-source component Spline, allowing us to reliably consume lineage metadata and identify interdependencies. We map the digested data into an expandable data model, enabling us to extract graph structures for both coarse- and fine-grained data lineage. Lastly, our solution visualizes the extracted data lineage via a modern web app, and integrates with BMW Group’s soon-to-be open-sourced Cloud Data Hub.

Metadata management is a crucial success factor for companies today, as for example, it enables exploiting data value fully or enables legal compliance. With the emergence of new concepts, such as the data lake, and new objectives, such as the enterprise-wide sharing of data, metadata management has evolved and now poses a renewed challenge for companies. In this context, we interviewed a globally active manufacturer to reveal how metadata management is implemented in practice today and what challenges companies are faced with and whether these constitute research gaps. As an outcome, we present the company’s metadata management goals and their corresponding solution approaches and challenges. An evaluation of the challenges through a literature and tool review yields three research gaps, which are concerned with the topics: (1) metadata management for data lakes, (2) categorizations and compositions of metadata management tools for comprehensive metadata management, and (3) the use of data marketplaces as metadata-driven exchange platforms within an enterprise. The gaps lay the groundwork for further research activities in the field of metadata management and the industry case represents a starting point for research to realign with real-world industry needs.

The author discusses the current experience of building the service of digital science publishing obtained by the libraries in the Western countries. She emphasizes that this process is incorporated into further informatization of libraries and their increasing role in managing science data. The digital publications integrate a number of interrelated programs comprising the whole cycle of scientific data management accomplished within the wider context of innovations. Digital publishing is a new type of library activities; it requires integrating competences of modern librarians (i.e. content selection, data supervising, metadata management, building digital collections, their preservation and archiving) and publishers (monitoring new trends in science and technology, selecting materials for publication, abstracting, scientific editing, developing marketing strategies).For the first time in the domestic library studies, the dynamics of this service in foreign countries is examined based on Library Publishing Directory for 2013– 2018. The author compares digital publishing services in four university libraries in different world regions and offers the findings of preliminary analysis of online publication services in foreign research libraries.

Background of this Study: This article examines how digital music handled on music streaming services, JOOX Indonesia in particular. Purposes: The aims was to bring insight that metadata management skill could help an enhancement over music streaming services through metadata and to improve the user experience when using music streaming services. Method:A single case study is chosen as the research method for this paper. The researcher did three months internship to see how the music file handled on the back end of JOOX. Semi-structured qualitative interviews and documentary analysis were used to collect and triangulate the qualitative data. Findings: The result shows JOOX using its operational self-possession procedures to handle its digital music file and using its own metadata standard with adaptation from music metadata standard. JOOX has a feature that utilizes music lyric. We found that lyric metadata embedded as a distinct entity on their backend system. Since lyric frequently used by the user as an access point when they do the retrieval, we propose to embed lyric as a field on music metadata to improve search result. Conclusion: These research shows are lyric as the essential part when users enjoy the music in music streaming services. By embed lyric on music metadata, lyric could be able as an access point for retrieval. Moreover, lyric as metadata could be part of music digital file handling.

NVMe SSDs provide extremely high performance and have been widely deployed in distributed object storage systems in data centers. However, we observe that there are still severe performance degradation and write amplification under the unaligned writes scenario with high-performance SSDs. In this article, we identify that the RMW sequence which is used to handle the unaligned writes incurs severe overhead in the data path. Besides, unaligned writes incur additional metadata management overhead in the block map table. To address these problems, we propose an object-based device system named NVStore to optimize the unaligned writes in cloud storage with NVMe SSDs. NVStore provides a Flexible Cache Management to reduce the RMW operations while supporting lazy page sync and ensuring data consistency. To optimize the metadata management, NVStore proposes a KV Affinity Metadata Management which co-designs the block map and key-value store to provides a flattened and decoupled metadata management. Evaluations show that NVStore provides at most 6.11× bandwidth of BlueStore in the cluster. Besides, NVStore can reduce at most 94.7 percent of the write traffic from metadata under unaligned writes compared to BlueStore and achieves smaller data write traffic which is about 50 percent of BlueStore and 65.7 percent of FileStore.

Purpose When faced with events, such as the global pandemic of coronavirus disease 2019 (COVID-19), libraries have a unique opportunity to develop a community facing response through born-digital collections. These collections provide challenges for metadata creation, collection development policies, workflows, and digital preservation. This paper aims to provide an overview of the Utah COVID-19 digital collection, with a discussion of impact and lessons learned. Design/methodology/approach This paper provides a case study of a born-digital collection initiative undertaken at the University of Utah in response to the COVID-19 pandemic. The project prompted engagement with the University of Utah communities and people across the state. Workflows, metadata management and partnerships are discussed, to provide a model for institutions developing similar projects during a time of crisis. Findings While the project was launched with open-ended and flexible goals, the response from the community has been both surprising and gratifying. Statistics and examples demonstrating reuse of collection materials are provided to highlight the impact and potential of community engagement. Originality/value Digital collecting projects during a historical event are not new, however the restrictions placed upon people worldwide during the COVID-19 pandemic created interesting circumstances for building this collection. Several lessons were learned throughout the project which will be useful for other institutions embarking upon related projects.

Apache Internet of Things Database (IoTDB) is a new open-source timeseries database management system. A distributed data management system not only needs to solve the problem of metadata synchronization between nodes caused by data partition and multiple replicas, but also needs to support efficient query request processing. To solve the problem of metadata synchronization among nodes, we propose a dual-layer granularity metadata management strategy. Based on the consistency hash partitioning method and Raft protocol, we designed a distributed framework that supports both strong consistency query and eventual consistency query. Based on the single-machine version of Apache IoTDB, we carried out the system implementation and experimental test. Compared with the single-level granularity management strategy, the test results showed that the two-level granularity metadata management strategy takes less memory resources and improves the write performance by $5%\sim10%$. Also, the results showed that the read and write performance of the distributed Apache IoTDB increases linearly with the extension of cluster size.

Most book and monograph publications still lack visibility within an electronic environment due to missing digitalized and persistent bibliographic data and/or fragmentary portability or interoperability based on the variety of formats, resulting in a more error-prone, non-standardized communication between various indexing systems and platforms. But books that cannot be discovered will not be read. OA metadata for books is essential for the transformation of the whole scholarly landscape, and one of its greatest advantages is full, immediate accessibility. 1 The point of origin for the OA META project was the realization that available book metadata (i.e. ONIX/MARC records) was not primarily created for the usage of indexing within an electronic research environment. Consequentially, both detail and focus of elements tagged in datasets are not necessarily selected to output solidified and persistent book records. The question was how to enrich or edit available metadata so that it can unfold its greatest impact in an e-context.

MotivationComputational systems biology analyses typically make use of multiple software and their dependencies, which often run across heterogeneous compute environments. This can introduce differences in performance and reproducibility. Capturing metadata (e.g. package versions, GPU model) currently requires repetitious code and is difficult to store centrally for analysis. Even where virtual environments and containers are used, updates over time mean that versioning metadata should still be captured within analysis pipelines to guarantee reproducibility.ResultsMicrobench is a simple and extensible Python package to automate metadata capture to a file or Redis database. Captured metadata can include execution time, software package versions, environment variables, hardware information, Python version, and more, with plugins. We present three case studies demonstrating Microbench usage to benchmark code execution and examine environment metadata for reproducibility purposes.AvailabilityInstall from the Python Package Index using pip install microbench. Source code is available from https://github.com/alubbock/microbench.Contactc.lopez@vanderbilt.edu

Computer-aided engineering (CAE) applications generate vast quantities of heterogeneous data. Domain experts often fail to explore and analyze these data, because they are not integrated across different applications. Existing data management solutions are rather tailored to scientific applications. In our approach, we tackle this issue by combining a data lake solution with graph-based metadata management. This provides a holistic view of all CAE data and of the data-generating applications in one interconnected structure. Based on a prototypical implementation, we discuss how this eases the task of domain experts to explore and extract data for further analyses.

In the parlance of Library and Information Sciences, metadata refers to information dimensions which are created, stored and shared in order to understand the value and contextual relevance of an information resource. It is inspired from the very idea of cataloguing library resources that has been the bread and butter of librarians for centuries. With the evolution of the World Wide Web and more recently the social semantic web, the importance of metadata has acquired a new dimension specific to web information management systems. The paper attempts to be a primer of various dimensions of metadata management for students, researchers and practitioners in the domain of Library and Information Sciences. It addresses wide ranging theoretical, technical and research issues in metadata management, and concludes with a fervent appeal to take up metadata practice and research.

Illegal use of memory pointers is a serious security vulnerability. A large number of malwares exploit the spatial and temporal nature of these vulnerabilities to subvert execution or glean sensitive data from an application. Recent countermeasures attach metadata to memory pointers, which define the pointer’s capabilities. The metadata is used by the hardware to validate pointer-based memory accesses. However, recent works have considerable overheads. Further, the pointer validation is decoupled from the actual memory access. We show that this could open up vulnerabilities in multithreaded applications and introduce new vulnerabilities due to speculation in out-of-order processors. In this article, we demonstrate that the overheads can be reduced considerably by efficient metadata management. We show that the hardware can be designed in a manner that would remain safe in multithreaded applications and immune to speculative vulnerabilities. We achieve these by ensuring that the pointer validations and the corresponding memory access is always done atomically and in order. To evaluate our scheme, which we call ALEXIA, we enhance an OpenRISC processor to perform the memory validation at runtime and also add compiler support. ALEXIA is the first hardware countermeasure scheme for memory protection that provides such an end-to-end solution. We evaluate the processor on an Altera FPGA and show that the runtime overhead, on average, is 14%, with negligible impact on the processor’s size and clock frequency. There is also a negligible impact on the program’s code and data sizes.

The ubiquity of data lakes has created fascinating new challenges for data management research. In this tutorial, we review the state-of-the-art in data management for data lakes. We consider how data lakes are introducing new problems including dataset discovery and how they are changing the requirements for classic problems including data extraction, data cleaning, data integration, data versioning, and metadata management.

Code Pointer Integrity (CPI) is an efficient control flow protection technique focusing on sensitive code pointers with a formal proof of security, but it relies on software lookup tables or Memory Management Unit (MMU) based address translation and instruction-level memory isolation which are impractical for resource-constrained embedded processors. This paper enables Architecture-assisted Run-time CPI on Embedded Processors (ARCE) with 2-level metadata to balance security, performance and resource overhead. The first level 2-bit property metadata colors data into different domains and the second level boundary metadata holds structure constraints for indirect code pointers only. With memory and instruction extensions, metadata shares the address space with program data and is propagated at runtime to maintain a precise set of sensitive code pointers. It lazily validates the content and boundary of sensitive pointers at dereference stage to eliminate false alarms. We implemented ARCE based on a shallow 3-stage pipeline processor Z-scale and validated its security effectiveness with code pointer attack vectors in RIPE. It introduces less than 1 percent performance overhead for benchmarks in C with 7.33 percent logic and 6.25 percent memory overhead. ARCE eliminates address space waste and dependency on advanced hardware which makes CPI practical even for systems with bare metal applications.

High-performance computing (HPC) systems generate huge amounts of metadata about different entities such as jobs, users, and files. Existing systems can efficiently record and manage part of these metadata, mainly the POSIX metadata of data files (e.g., file size, name, and permissions mode). But another important set of metadata, referred to as “rich” metadata in this study, which record not only wider range of entities (e.g., running processes and jobs) but also more complex relationships between them, are mostly missing in current HPC systems. Yet such rich metadata are critical for supporting many advanced data management functions such as identifying data sources and parameters behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. To uniformly and efficiently manage the rich metadata generated in HPC systems, We propose to utilize a graph model in this study. We identify the key challenges of implementing such a graph-based HPC rich metadata management system and present GraphMeta, a graph-based rich metadata management system designed and optimized for HPC platforms, to tackle these challenges. Extensive evaluations on both synthetic and real HPC metadata workloads show its advantages in both performance and scalability compared with existing solutions.

In an era of rapid advancement of algorithms that extract knowledge from data, data and metadata management are increasingly critical to research success. In materials science, there are few examples of experimental databases that contain many different types of information, and compared with other disciplines, the database sizes are relatively small. Underlying these issues are the challenges in managing and linking data across disparate synthesis and characterization experiments, which we address with the development of a lightweight data management framework that is generally applicable for experimental science and beyond. Five years of managing experiments with this system has yielded the Materials Experiment and Analysis Database (MEAD) that contains raw data and metadata from millions of materials synthesis and characterization experiments, as well as the analysis and distillation of that data into property and performance metrics via software in an accompanying open source repository. The unprecedented quantity and diversity of experimental data are searchable by experiment and analysis attributes generated by both researchers and data processing software. The search web interface allows users to visualize their search results and download zipped packages of data with full annotations of their lineage. The enormity of the data provides substantial challenges and opportunities for incorporating data science in the physical sciences, and MEAD’s data and algorithm management framework will foster increased incorporation of automation and autonomous discovery in materials and chemistry research.

Next-generation Storage Class Memory (SCM) offers low-latency, high-density, byte-addressable access and persistency. The potent combination of these attractive characteristics makes it possible for SCM to unify the main memory and storage to reduce the storage hierarchy. Aiming for this, several persistent memory systems were designed. However, the heavy metadata and transaction cost degrade the system performance. Moreover, neither of them pays attention to wear-leveling strategy. In this article, we present a lightweight persistent memory system, LiwePMS, which allows a fast access to persistent data stored in SCM with wear-aware memory management. LiwePMS makes performance improvement by simplifying the metadata management and the consistency method. LiwePMS abstracts SCM as heap space with container-based dynamic address mapping. Also, LiwePMS implements efficient wear-aware dynamic memory allocator and lightweight transaction mechanism for data consistency in user-space library. The experiments showed that LiwePMS persists key-value records 1.5× faster than Redis RDB mechanism. LiwePMS improves the performance of persistent region operation by more than 45%, 63%, and 1.1× comparing with HEAPO, Mnemosyne, and NVML, respectively. Also, the wear-leveling policy of memory allocator outperforms that of NVMalloc from 35% to 30%, and the transaction method promotes the transaction performance to 1.8× compared to NVML.

With new emerging technologies, such as satellites and drones, archaeologists collect data over large areas. However, it becomes difficult to process such data in time. Archaeological data also have many different formats (images, texts, sensor data) and can be structured, semi-structured and unstructured. Such variety makes data difficult to collect, store, manage, search and analyze effectively. A few approaches have been proposed, but none of them covers the full data lifecycle nor provides an efficient data management system. Hence, we propose the use of a data lake to provide centralized data stores to host heterogeneous data, as well as tools for data quality checking, cleaning, transformation, and analysis. In this paper, we propose a generic, flexible and complete data lake architecture. Our metadata management system exploits goldMEDAL, which is the most complete metadata model currently available. Finally, we detail the concrete implementation of this architecture dedicated to an archaeological project., 25th International Database Engineering & Applications Symposium (IDEAS 2021), Jul 2021, Montr\'eal, Canada

The Atmospheric Radiation Measurement (ARM) user facility is a US Department of Energy Office of Science user facility that is managed and operated through a collaborative effort led by nine US Department of Energy national laboratories. The ARM Data Center, located at Oak Ridge National Laboratory, is responsible for the timely collection, processing, and delivery of data products to the scientific community. The ARM Data Center holds more than 11,000 data products, including metadata collected from field campaigns, instruments, value-added products, and principal investigator–contributed data. These data sets are checked for successful transfer (for most data, this transfer is carried out automatically via the network; however, some of the largest data sets and some of the most remote sites require manual shipping of hard disks) and both the data and metadata are processed to a standard format, which is an ARM-standardized structure, via the Network Common Data Form. The Network Common Data Form is a self-describing binary format with many compatible software tools. Once processed, the data are cataloged, stored in the ARM Data Archive, and made discoverable through association with an array of metadata-characterizing information, such as location and measurement classification. These metadata enable powerful search capabilities through the ARM Data Center Data Discovery interface. This paper discusses the workflow of how the new discovery system has been redesigned from user requirements and how the data are distributed to the scientific community.

In times of continuously growing data sizes, performing insightful analysis is increasingly difficult. I/O libraries such as NetCDF and ADIOS2 offer options to manage additional metadata to make the data retrieval more efficient. However, queries on this metadata are difficult as it is currently stored inside the corresponding self-describing data formats. By replacing the file system underneath with the storage framework JULEA, we can use dedicated backends for key-value and object stores, as well as databases. Splitting the BP file content into file metadata and file data enables novel and highly efficient data management techniques without creating redundancy. We have kept our approach transparent to the application layer by implementing a custom ADIOS2 engine. Moreover, our data analysis interface allows speeding up metadata queries by a factor of up to 60,000 in comparison to the ADIOS2 API and data formats.

The recent advent of the hardware trusted execution environment (TEE), e.g., Intel SGX, enables encrypted and integrity-verified in-memory key-value (KV) stores. However, due to the architectural limitations of the hardware, it is non-trivial to build a secure in-memory KV store with SGX without compromising the performance. The reason comes from (i) the limited memory capacity the SGX TEE provides, and (ii) being unaware of the access patterns of skewed workloads, which are commonly seen in the real world.In this paper, we present Aria, a secure in-memory KV store based on SGX. Our goal is to utilize the limited resource while still achieving high performance. Aria places KV pairs and index structures directly in the untrusted memory and introduces the security metadata in the TEE to conduct protection. The core component of Aria is Secure Cache, a software-based cache layer, which uses the limited memory resource to guarantee the confidentiality and integrity (including freshness) of Aria. Secure Cache keeps the frequently accessed security metadata in the TEE memory at fine-granularity and evicts rarely-used ones to the untrusted memory. With Secure Cache, we have the opportunities to explore strategies that are impossible in SGX implementation. By decoupling the security metadata management from the index structure, Aria supports various index schemes. We implement Aria with the indexes of both a hash table and a B-tree. Experiments show that Aria improves throughput by up to 104% compared to the state-of-the-art system.

Many data storage systems will cause problems in the process of sharing data between systems. It will make it difficult for users to search for data, understand data, and transfer data between systems. Metadata management will help understand data, exchange data between different systems, find and share data between systems. This research focuses on guidelines design of metadata management process based on DAMA-DMBOK, which is safe to apply in companies or organizations. Within the DAMA-DMBOK framework, defined the metadata management process in general. It is necessary to have a guideline containing steps that can explain the metadata management process’s detail based on the DAMA-DMBOK framework. Design process guidelines require an assessment to measure how well the process can be relied on to implement in a company. This research is essential to make it easier for companies to increase data governance and data assets’ usability to be more efficient and reliable.

47th International Conference on Current Trends in Theory and Practice of Computer Science, SOFSEM 2021, Bolzano-Bozen, Italy, January 25–29, 2021, Proceedings; International audience; Data Lake (DL) is known as a Big Data analysis solution. A data lake stores not only data but also the processes that were carried out on these data. It is commonly agreed that data preparation/transformation takes most of the data analyst's time. To improve the efficiency of data processing in a DL, we propose a framework which includes a metadata model and algebraic transformation operations. The metadata model ensures the findability, accessibility, interoperability and reusability of data processes as well as data lineage of processes. Moreover, each process is described through a set of coarse-grained data transforming operations which can be applied to different types of datasets. We illustrate and validate our proposal with a real medical use case implementation.

In the era of big and omics data, good organization, management, and description of experimental data are crucial for achieving high-quality datasets. This, in turn, is essential for the export of robust results, to publish reliable papers, make data more easily available, and unlock the huge potential of data reuse. Lately, more and more journals now require authors to share data and metadata according to the FAIR (Findable, Accessible, Interoperable, Reusable) principles. This work aims to provide a step-by-step guideline for the FAIR data and metadata management specific to grapevine and wine science. In detail, the guidelines include recommendations for the organization of data and metadata regarding (i) meaningful information on experimental design and phenotyping, (ii) sample collection, (iii) sample preparation, (iv) chemotype analysis, (v) data analysis (vi) metabolite annotation, and (vii) basic ontologies. We hope that these guidelines will be helpful for the grapevine and wine metabolomics community and that it will benefit from the true potential of data usage in creating new knowledge being revealed.

The case study relates to the work undertaken by the Cardiff Metropolitan University (Cardiff Met) Library Service as part of the Wales Higher Education Libraries Forum (WHELF) Shared Library Management System (LMS) project. The study provides contextual information about the service and an overview of the WHELF project delivery, before discussing some key impacts for Cardiff Met in terms of service developments, and for library staff. The focus of those impacts relates to the collaborative nature of the work. It discusses how this has benefitted services such as user discovery, metadata management, and the acquisition process and collection management and then addresses the positive impact of being part of the collaborative environment for library staff. The case study completes with a brief look at some options on where WHELF and Cardiff Met could take the project in future.

Metadata management is an essential condition to follow the FAIR principles. Therefore, metadata management was one asset of an accompanying project within a funding scheme for registries in health services research. The metadata of the funded projects were acquired, combined in a database compatible with the metamodel of ISO/IEC 11179 “Information technology – Metadata registries” third edition (ISO/IEC 11179-3), and analyzed in order to support the development and the operation of the registries. In the second phase of the funding scheme, six registries delivered a complete update of their metadata. The mean number of data elements increased from 245.7 to 473.5 and the mean number of values from 569.5 to 1,306.0. The conceptual core of the database had to be extended by one third to cover the new elements. The reason for this increase remained unclear. Constraints from the grant might be causal, a deviation from an evidence-based development process as well. It is questionable, whether the revealed quality of the metadata is sufficient to fulfill the FAIR principles. The extension of the metamodel of ISO/IEC 11179-3 is in agreement with the literature. However, further research is needed to find workable solutions for metadata management.

International audience; Over the past two decades, we have witnessed an exponential increase of data production in the world. So-called big data generally come from transactional systems, and even more so from the Internet of Things and social media. They are mainly characterized by volume, velocity, variety and veracity issues. Big data-related issues strongly challenge traditional data management and analysis systems. The concept of data lake was introduced to address them. A data lake is a large, raw data repository that stores and manages all company data bearing any format. However, the data lake concept remains ambiguous or fuzzy for many researchers and practitioners, who often confuse it with the Hadoop technology. Thus, we provide in this paper a comprehensive state of the art of the different approaches to data lake design. We particularly focus on data lake architectures and metadata management, which are key issues in successful data lakes. We also discuss the pros and cons of data lakes and their design alternatives.

Computer-aided engineering (CAE) applications support the digital transformation of the manufacturing industry. They facilitate virtual product development and product testing via computer simulations. CAE applications generate vast quantities of heterogeneous data. Domain experts struggle to access and analyze them, because such engineering data are not sufficiently described with metadata. In this paper, we characterize the CAE domain and identify unsolved challenges for a tailored data and metadata management. For instance, work activities in product development projects and their relationships to data are not represented explicitly in current metadata models. We propose a metadata model that addresses all challenges and provides a connected view on all CAE data, metadata, and work activities of development projects. We validate the feasibility of our metadata model through a prototypical implementation and its application to a real-world use case. This verifies that our metadata model addresses the CAE-specific challenges and this way eases the task of domain experts to exploit relevant data.

TC 2: Software: Theory and Practice; International audience; Highly-heterogeneous and fast-arriving large amounts of data, otherwise said Big Data, induced the development of novel Data Management technologies. In this paper, the members of the IFIP Working Group 2.6 share their expertise in some of these technologies, focusing on: recent advancements in data integration, metadata management, data quality, graph management, as well as data stream and fog computing are discussed.

We present PacketMill, a system for optimizing software packet processing, which (i) introduces a new model to efficiently manage packet metadata and (ii) employs code-optimization techniques to better utilize commodity hardware. PacketMill grinds the whole packet processing stack, from the high-level network function configuration file to the low-level userspace network (specifically DPDK) drivers, to mitigate inefficiencies and produce a customized binary for a given network function. Our evaluation results show that PacketMill increases throughput (up to 36.4 Gbps -- 70%) & reduces latency (up to 101 us -- 28%) and enables nontrivial packet processing (e.g., router) at ~100 Gbps, when new packets arrive >10× faster than main memory access times, while using only one processing core. Part of proceedings: ISBN 978-1-4503-8317-2QC 20210210 Time-Critical Clouds ULTRA WASP

Their hierarchical organization and metadata management limit the traditional file storage systems’ scalability. The industry widely uses Distributed Object Storage Systems because they keep the advantages of traditional file storage systems, e.g., data sharing, and alleviate the scalability issue through benefiting from the flat namespaces and integrating the meta-data in the object. However, evaluating and comparing distributed object storage systems remains a significant challenge. The existing DOSS benchmarks provide simple read/write operations without considering the complex workload characteristics in request arrival patterns and request size distribution. This paper presents an Object Storage Benchmark suite, named OStoreBench, which characterizes critical paths of three real-world application scenarios, including online service, big data analysis, and file backup. We evaluate three state-of-the-practice object storage systems using OStoreBench, including Ceph, Openstack Swift, and Haystack. The benchmark suite is publicly available from https://github.com/EVERYGO111/OStoreBench.

Semantic technologies have recently gained considerable influence and attention in the field of technical communication and information management. While metadata management is already a well-known field of content management technologies, its semantic extension addresses more recently, for example, problems of model-based product development and related content engineering processes. On the other hand, dynamic search technology and content delivery can benefit from semantic modelling by enhancing search functionalities or by integrating various data sources utilizing semantic mapping. In this evolving environment, we propose a logical layer of content correlations as so-called semantic correlation rules (SCR). This layer can be understood as an interface between content management systems, semantic modelling systems and content delivery portals. Semantic correlation rules serve as a light-weight ontology consisting primarily of untyped semantic relations between metadata classes. In doing so, class-to-class linking mechanisms can be implemented in content delivery and search environments while serving as a basis for the previously introduced microDoc concept

Extract-transform-load (ETL) processes play a crucial role in data analysis in real-time data warehouse environments which demand low latency and high availability features for functionality. In essence, ETL- processes are becoming bottlenecks in such environments due to complexity growth, number of steps in data transformations, number of machines used for data processing and finally, increasing impact of human factors on development of new ETL-processes. In order to mitigate this impact and provide resilience of the ETL process, a special Metadata Framework is needed that can manage the design of new data pipelines and processes. In this work, we focus on ETL metadata and its use in driving process execution and present a proprietary approach to the design of the metadata-based process control that can reduce complexity, enhance resilience of ETL processes and allow their adaptive self-reorganization. We present a metadata framework implementation which is based on open-source Big Data technologies, describing its architecture and interconnections with external systems, data model, functions, quality metrics, and templates. A test execution of an experimental Airflow Directed Acyclic Graph (DAG) with randomly selected data is performed to evaluate the proposed framework.

Digital transformation of a railway system based on big data technologies relies on integrating large volumes of streaming data into digitally enabled enterprise systems to form a comprehensive and efficient intelligent transportation system. Data requirements of the smart railway transportation involve a large number of unstructured data and semi-structured data including railway KPI data. Traditional ETL technology cannot cope with fast growing demands of processing large volumes of real-time data collected from heterogeneous sources both inside the system and in the environment. According to the characteristics of the railway KPI data, this paper proposes the designs of an automated ETL system with higher versatility and efficiency of data processing. To reach the goals, we optimize the workflow of the ETL using a proprietary designed metadata management framework. Making ETL suitable for big data-driven railway transportation environment, requires redesigning the ETL processing rules by using metadata model and then optimizing the extracting, transforming and loading processes of the ETL system. Our experimental results with actual railway KPI data show that the proposed metadata supported automated ETL system can effectively serve the railway KPI data processing using open source distributed big data technologies. The proposed metadata framework proved to be efficient in processing complex data structures and large data capacity of big data.

Traditional I/O stacks feature a strict separation of layers, which provides portability benefits but makes it impossible for storage systems to understand the structure of data. Coupling storage systems with self-describing data formats can offer benefits by making the storage system responsible for managing file metadata and allowing it to use structural information for selecting appropriate storage technologies.Our proposed storage architecture enables novel data management approaches and has the potential to provide significant performance improvements in the long term. By making use of established self-describing data formats, no modifications are necessary to run existing applications, which helps preserve past investments in software development.Specifically, we have designed and implemented an HDF5 VOL plugin to map file data and metadata to object and key-value stores, respectively. Evaluations show that our coupled storage system offers competitive performance when compared with the native HDF5 data format. In some cases, performance could even be improved by up to a factor of 100.

Digital learning resources are becoming increasingly important, especially in times of COVID-19 home schooling. To align resources with educational objectives defined in school curricula, we need ways to describe and relate them. In this work, we present a new data model, a data management workflow and a prototypical information system enabling annotators to describe learning resources using the terminology and structure, while maintaining applicability in other contexts through transcription. Preliminary evaluation results with metadata curators show that our model is well-suited to efficiently model secondary education curricula. Provided that future evaluations with content creators, teachers and learners yield positive results, our model may enable curriculum specific learning analytics, search and recommendation without the need to annotate resources for each curriculum.

This paper presents an approach to manage metadata (target class labels) for the recorded primary Doppler radar data. This information is necessary for further research and development of target classification algorithms. For this purpose, a labelling methodology and an application radar data analysis and target labelling was developed. The application includes radar records file processing, Doppler filtering, tag creation, data visualizationand tag database. For the better context of analysed data, an interface to Geographic Information Software (GIS) program is included as well. GIS program allows overlaying radar Plan Position Indicator (PPI) data with map data, airplane transponder tracks, drone flight path log and other support data. Finally, the application notes and observations are presented.

Granting agencies invest millions of dollars on the generation and analysis of data, making these products extremely valuable. However, without sufficient annotation of the methods used to collect and analyze the data, the ability to reproduce and reuse those products suffers. This lack of assurance of the quality and credibility of the data at the different stages in the research process essentially wastes much of the investment of time and funding and fails to drive research forward to the level of potential possible if everything was effectively annotated and disseminated to the wider research community. In order to address this issue for the Hawai’i Established Program to Stimulate Competitive Research (EPSCoR) project, a water science gateway was developed at the University of Hawai‘i (UH), called the ‘Ike Wai Gateway. In Hawaiian, ‘Ike means knowledge and Wai means water. The gateway supports research in hydrology and water management by providing tools to address questions of water sustainability in Hawai‘i. The gateway provides a framework for data acquisition, analysis, model integration, and display of data products. The gateway is intended to complement and integrate with the capabilities of the Consortium of Universities for the Advancement of Hydrologic Science’s (CUAHSI) Hydroshare by providing sound data and metadata management capabilities for multi-domain field observations, analytical lab actions, and modeling outputs. Functionality provided by the gateway is supported by a subset of the CUAHSI’s Observations Data Model (ODM) delivered as centralized web based user interfaces and APIs supporting multi-domain data management, computation, analysis, and visualization tools to support reproducible science, modeling, data discovery, and decision support for the Hawai’i EPSCoR ‘Ike Wai research team and wider Hawai‘i hydrology community. By leveraging the Tapis platform, UH has constructed a gateway that ties data and advanced computing resources together to support diverse research domains including microbiology, geochemistry, geophysics, economics, and humanities, coupled with computational and modeling workflows delivered in a user friendly web interface with workflows for effectively annotating the project data and products. Disseminating results for the ‘Ike Wai project through the ‘Ike Wai data gateway and Hydroshare makes the research products accessible and reusable.

There has been a rapid development in genome sequencing, including high-throughput next generation sequencing (NGS) technologies, automation in biological experiments, new bioinformatics tools and utilization of high-performance computing and cloud computing. ChIP-based NGS technologies, e.g. ChIP-seq and ChIP-exo, are widely used to detect the binding sites of DNA-interacting proteins in the genome and help us to have a deeper mechanistic understanding of genomic regulation. As sequencing data is generated at an unprecedented pace from the ChIP-based NGS pipelines, there is an urgent need for a metadata management system. To meet this need, we developed the Platform for Eukaryotic Genomic Regulation (PEGR), a web service platform that logs metadata for samples and sequencing experiments, manages the data processing workflows, and provides reporting and visualization. PEGR links together people, samples, protocols, DNA sequencers and bioinformatics computation. With the help of PEGR, scientists can have a more integrated understanding of the sequencing data and better understand the scientific mechanisms of genomic regulation. In this paper, we present the architecture and the major functionalities of PEGR. We also share our experience in developing this application and discuss the future directions.

In recent years, increased interest in cloud-based data warehousing technologies has emerged with many enterprises moving away from on-premise data warehousing solutions. The incentives for adopting cloud data warehousing technologies are many: cost-cutting, on-demand pricing, offloading data centers, unlimited hardware resources, built-in disaster recovery, to name a few. There is inherent difference in the language surface and feature sets of on-premise and cloud data warehousing solutions. This could range from subtle syntactic and semantic differences, with potentially big impact on result correctness, to complete features that exist in one system but are missing in other systems. While there have been some efforts to help automate the migration of on-premise applications to new cloud environments, a major challenge that slows down the migration pace is the handling of features not yet supported, or partially supported, by the cloud technologies. In this paper we build on our earlier work in adaptive data virtualization and present novel techniques that allow running applications utilizing sophisticated database features within foreign query engines lacking the native support of such features. In particular, we introduce a framework to manage discrepancy of metadata across heterogeneous query engines, and various mechanisms to emulate database applications code in cloud environments without any need to rewrite or change the application code.

Outsourced data, as significant service offered by the cloud service provider (CSP), can effectively facilitate the data owner (DO) overcoming the storage limitations on massive data. To ensure the availability of data, DO usually outsources the data replications to multiple CSPs (multi-CSPs) and utilizes a third party metadata management (TPMM) to dominate the metadata of the corresponding replications. However, during the outsourced procedures, DO can hardly confirm the confidence of the TPMM who may take some malicious behaviors to affect the reliability of data. Thus, DO inevitably faces data security issues caused by the over-reliance on the semi-trusted TPMM to manage the metadata of replications. In this paper, we focus on the problem of reliable outsourced data service among multi-CSPs in untrusted environment, that is, how to reliably store and verify the metadata of the data replications in untrusted multi-CSPs environment. To address the problem, we use the novel blockchain technology as a medium to build a trusted outsourced service platform. Moreover, we fully consider the innovative characteristics of blockchain including decentralized architecture, redundancy storage, collective maintenance, and tamper resistant to ensure the data cannot be changed maliciously. We first design a blockchain-based outsourced service framework for storing data replications in untrusted environment, which contains three key layers, that is, storage layer , verification layer , and blockchain layer . Then, we devise a novel concept of verification peer (VP) for maintaining metadata stored by a form of blockchain, and each of which holds the entire blockchain locally to prevent metadata from being maliciously tampered with. Finally, based on the proposed model, we introduce a collaborative algorithm invoked by VPs to store and verify the metadata of replications. We present a completed analysis and conduct extensive experiments on multi-CSPs scenario. The evaluation results demonstrate that our proposed approach achieves superior performance.

The technology space of the University of Denver Libraries to manage digital collections and institutional resources isn&rsquo, t relegated to one department on campus &ndash, rather, it distributed across a network of collaborators with the skills and expertise to provide that support. The infrastructure, which is comprised of an archival metadata management system (Archivespace), a digital repository (Node.js + ElasticSearch), preservation storage (ArchivesDirect), and a streaming server (Kaltura) is independently but cooperatively managed across IT, library departments and vendors. The coordinated effort of digital curation activities still allows each group to focus on the service they have the most vested interest in providing. This paper will talk about the different management and development practices involved in developing our integrated infrastructure to provide digital collections as a service.

Atmospheric Radiation Measurement (ARM), a U.S. Department of Energy (DOE) scientific user facility, is a key geophysical data source for national and international climate research. Utilizing a standardized schema that has evolved since ARM inception in 1989, the ARM Data Center (ADC) processes over 1.8 petabytes of stored data across over 10,000 data products. Data sources include ARM-owned instruments, as well as field campaign datasets, Value Added Products, evaluation data to test new instrumentation or models, Principal Investigator data products, and external data products (e.g., NASA satellite data). In line with FAIR principles, a team of metadata experts classifies instruments and defines spatial and temporal metadata to ensure accessibility through the ARM Data Discovery. To enhance geophysical metadata collaboration across American and European organizations, this work will summarize processes and tools which enable the management of ARM data and metadata. For example, this presentation will highlight recent enhancements in-field campaign metadata workflows to handle the ongoing Multidisciplinary Drifting Observatory for the Study of Arctic Climate (MOSAiC) data. Other key elements of ARM data center include: the architecture of ARM data transfer and storage processes, evaluation of data quality, ARM consolidated databases. We will also discuss tools developed for identifying and recommending datastreams and enhanced DOI assignments for all data types to assist an interdisciplinary user base in selecting, obtaining, and using data as well as citing the appropriate data source for reproducible atmospheric and climate research.

The consistent management of research data is crucial for the success of long-term and large-scale collaborative research. Research data management is the basis for efficiency, continuity, and quality of the research, as well as for maximum impact and outreach, including the long-term publication of data and their accessibility. Both funding agencies and publishers increasingly require this long term and open access to research data. Joint environmental studies typically take place in a fragmented research landscape of diverse disciplines; researchers involved typically show a variety of attitudes towards and previous experiences with common data policies, and the extensive variety of data types in interdisciplinary research poses particular challenges for collaborative data management. In this paper, we present organizational measures, data and metadata management concepts, and technical solutions to form a flexible research data management framework that allows for efficiently sharing the full range of data and metadata among all researchers of the project, and smooth publishing of selected data and data streams to publicly accessible sites. The concept is built upon data type-specific and hierarchical metadata using a common taxonomy agreed upon by all researchers of the project. The framework’s concept has been developed along the needs and demands of the scientists involved, and aims to minimize their effort in data management, which we illustrate from the researchers’ perspective describing their typical workflow from the generation and preparation of data and metadata to the long-term preservation of data including their metadata., Deutsche Forschungsgemeinschaft http://dx.doi.org/10.13039/501100001659

The provisioning of an efficient ultra-large scalable distributed storage system for expanding cloud applications has been a challenging job for researchers in academia and industry. In such an ultra-large-scale storage system, data are distributed on multiple storage nodes for performance, scalability, and availability. The access to this distributed data is through its metadata, maintained by multiple metadata servers. The metadata carries information about the physical address of data and access privileges. The efficiency of a storage system highly depends on effective metadata management. This research presents an extensive systematic literature analysis of metadata management techniques in storage systems. This research work will help researchers to find the significance of metadata management and important parameters of metadata management techniques for storage systems. Methodical examination of metadata management techniques developed by various industry and research groups is described. The different metadata distribution techniques lead to various taxonomies. Furthermore, the article investigates techniques based on distribution structures and key parameters of metadata management. It also presents strengths and weaknesses of individual existing techniques that will help researchers to select the most appropriate technique for specific applications. Finally, it discusses existing challenges and significant research directions in metadata management for researchers.

As fusion experiment goes to steady state and high-performance data acquisition system been developed, traditional storage solutions cannot cope with the needs of fusion experiment data storage. Distributed file systems such as Lustre are being adopted by more and more facilities. The scaling-out of performance and capacity features addressed the main issue of traditional SAN (Storage Area Network) based storage. However, traditional file system based software solutions are still being used with the distributed file system. They lack modern data manage functions and limiting the performance of parallel storage. J-TEXT Cloud Database (JCDB) is a software stack that uses distributed database to provide fusion experiment data storage and management services. It ships with a storage engine powered by Cassandra database. This storage engine is designed for fusion experiment data and provides great performance. Data is divided into chunks when written and stored in a specially designed distributed database across the cluster. It has a MongoDB powered metadata management system which works seamlessly with the storage engine. JCDB is fully modular, handling different data type and metadata management functions are integrated as plugins. Even the storage engine can be changed. Though J-TEXT not supports long pulse experiments, the design of JCDB is aiming to meet long pulse discharge requirements, bring distributed database technology for future fusion devices such as China fusion engineering test reactor (CFETR). JCDB has the benefit of distributed file systems, provides complex metadata manage functions and comes with great performance.