Search

The main function of Hadoop is the storage and processing of big data, especially the processing of large datasets. However, in practice, there are numerous small files, and Hadoop has many flaws when dealing with these small files. A storage-optimization method for numerous agricultural resource small files based on Hadoop is proposed, using the precursor and subsequent relationship between different small files of agricultural resources to merge small files. By accessing small files and performing metadata caching through an index mechanism, as well as the prefetching mechanism of associated small files, the storage-optimization method improves the reading efficiency. Experimental results show that this method reduces the memory consumption of the Hadoop name node and improves the performance of the system. [ABSTRACT FROM AUTHOR]

Big-Data is the storing and retrieving is the current and first challenge in the field of IT. The large amount of data generated by different information system .Those generated data in the forms of structured, unstructured or semi-structured .now a day it become bigchallenge to store the Big-Data. This paper introduce study and the comparison of two distributed file system managing and handling different parameters with respect to frameworks of both distributed file system which is used to handle and maintain storage and access of the large amount of generated big data in the Andrew distributed file system and Hadoop distributed file system. This study and analyses will more useful for understanding the insight into framework of Andrew and HDFS and highlight the features those are common and difference between both distributed file systems. [ABSTRACT FROM AUTHOR]

The explosion in volumes and types of data has led to substantial challenges in data management. These challenges are often faced by front-line researchers who are already dealing with rapidly changing technologies and have limited time to devote to data management. There are good high-level guidelines for managing and processing scientific data. However, there is a lack of simple, practical tools to implement these guidelines. This is particularly problematic in a highly distributed research environment where needs differ substantially from group to group and centralised solutions are difficult to implement and storage technologies change rapidly. To meet these challenges we have developed dtool, a command line tool for managing data. The tool packages data and metadata into a unified whole, which we call a dataset. The dataset provides consistency checking and the ability to access metadata for both the whole dataset and individual files. The tool can store these datasets on several different storage systems, including a traditional file system, object store (S3 and Azure) and iRODS. It includes an application programming interface that can be used to incorporate it into existing pipelines and workflows. The tool has provided substantial process, cost, and peace-of-mind benefits to our data management practices and we want to share these benefits. The tool is open source and available freely online at http://dtool.readthedocs.io. [ABSTRACT FROM AUTHOR]

Abstract Persistent Identifier (PID) systems have evolved in the last decade to mitigate the problem of link rot and provide unique and resolvable identifiers for digital objects, becoming a key element in archival services. However, they still depend on centralized services which may either become unavailable or cease to behave or be managed as expected. Decentralized technologies may serve as the infrastructure for the design of a trustless PID system over which other incentive mechanisms may be devised. Here we discuss a basic model for such a novel PID system, based on the capacity of content addressing of current decentralized file systems and the concept and definition of verifiable claims as metadata statements about the existence of an immutable digital resource. An example basic design is described using the Interplanetary File System (IPFS) as supporting infrastructure with a minimal set of conventions, that could be used as an alternate mechanism for storage of byte streams in digital repositories as DSpace or CKAN. [ABSTRACT FROM AUTHOR]

Key-Value stores provide scalable metadata service for distributed file systems. However, the metadata's organization itself, which is organized using a directory tree structure, does not fit the key-value access pattern, thereby limiting the performance. To address this issues, we propose a distributed file system with a flattened and fine-grained division metadata service, LocoMeta, to bridge the performance gap between file system metadata and key-value stores. LocoMeta is designed to bridge the gap between file metadata to key-value store with two techniques. First, LocoMeta flattens the directory content and structure, which organizes file and directory index nodes in a flat space while reversely indexing the directory entries. Second, it exploits a fine-grained division method to improve the key-value access performance. Evaluations show that LocoMeta with eight nodes boosts the metadata throughput by five times, which approaches 93 percent throughput of a single-node key-value store, compared to 18 percent in the state-of-the-art IndexFS. [ABSTRACT FROM AUTHOR]

We propose topic models for unsupervised cluster matching, which is the task of finding matching between clusters in different domains without correspondence information. For example, the proposed model finds correspondence between document clusters in English and German without alignment information, such as dictionaries and parallel sentences/documents. The proposed model assumes that documents in all languages have a common latent topic structure, and there are potentially infinite number of topic proportion vectors in a latent topic space that is shared by all languages. Each document is generated using one of the topic proportion vectors and language-specific word distributions. By inferring a topic proportion vector used for each document, we can allocate documents in different languages into common clusters, where each cluster is associated with a topic proportion vector. Documents assigned into the same cluster are considered to be matched. We develop an efficient inference procedure for the proposed model based on collapsed Gibbs sampling. The effectiveness of the proposed model is demonstrated with real data sets including multilingual corpora of Wikipedia and product reviews. [ABSTRACT FROM AUTHOR]

Focus of this research work is optimizing the deduplication system by adjusting the pertinent factors in content defined chunking (CDC) to identify as the key ingredients by declaring chunk cut-points and efficient fingerprint lookup using bucket based index partitioning. For efficient chunking, proposed Differential Evolution (DE) algorithm based approach is optimized Two Thresholds Two Divisors (TTTD-P) CDC algorithm where significantly it reduces the number of computing operations by using single dynamic optimal parameter divisor D with optimal threshold value exploiting the multi-operations nature of TTTD. Therefore, proposed DE based TTTD-P optimize chunking to maximize chunking throughput with increased deduplication ratio (DR); and bucket indexing approach reduces hash values judgment time to identify and declare redundant chunk about 16 times faster than Rabin CDC, 5 times than Asymmetric Extremum (AE) CDC, 1.6 times than FAST CDC. Experimental results comparative analysis reveal that TTTD-P using fast BUZ rolling hash function with bucket indexing on Hadoop Distributed File System (HDFS) provide a comparatively maximum redundancy detection with higher throughput, higher deduplication ratio, lesser computation time and very low hash values comparison time as being best distributed deduplication for big data storage systems. [ABSTRACT FROM AUTHOR]

Increasing the software design quality is a key research challenge in object-oriented software development system. Cohesion is one of the key spect that helps to evaluate the quality and modularity of a software system at the design level. It helps to create software components that are directly reusable to the industry because of their less dependence on other components. In this paper, a new cohesion metric for object-oriented software, named as Usage Pattern Based Cohesion (UPBC), is proposed which is computed at the module level. This paper considers class as a module initially and subsequently group of classes (i.e. a package) is considered as a module with an aim of improving overall cohesion. This metric utilizes the Frequent Usage Patterns (FUP) extracted from different member functions interactions to capture the cohesiveness of the module. Further, the measured cohesion value is used to perform clustering of modules in order to increase cohesion and decrease coupling among modules simultaneously. The clustering is performed by using a newly proposed clustering algorithm called FUPClust (Frequent Usage Pattern based Clustering) based on FUP interactions among modules. The proposed approach is applied to two Java software systems and the results obtained show a significant improvement in the cohesiveness of the software system. [ABSTRACT FROM AUTHOR]

In this article, we design and implement a cooperative shingle-aware file system, called CosaFS, on heterogeneous storage devices that mix solid-state drives (SSDs) and shingled magnetic recording (SMR) technology to improve the overall performance of storage systems. The basic idea of CosaFS is to classify objects as hot or cold objects based on a proposed Lookahead with Recency Weight scheme. If an object is identified as a hot (small) object, then it will be served by SSD. Otherwise, cold (large) objects are stored on SMR. For an SMR, large objects can be accessed in large sequential blocks, rendering the performance of their accesses comparable with that of accessing the same large sequential blocks as if they were stored on a hard drive. Small objects, such as inodes and directories, are stored on the SSD where "seeks" for such objects are nearly free. With thorough empirical studies, we demonstrate that CosaFS, as a cooperative shingle-aware file system, with metadata separation and cache-assistance, is a very effective way to handle the disk-based data demanded by the shingled writes and outperforms the device- and host-side shingle-aware file systems in terms of throughput, IOPS, and access latency as well. [ABSTRACT FROM AUTHOR]

Repair performance in hierarchical data centers is often bottlenecked by cross-rack network transfer. Recent theoretical results show that the cross-rack repair traffic can be minimized through repair layering, whose idea is to partition a repair operation into inner-rack and cross-rack layers. However, how repair layering should be implemented and deployed in practice remains an open issue. In this article, we address this issue by proposing a practical repair layering framework called DoubleR.We design two families of practical double regenerating codes (DRC), which not only minimize the cross-rack repair traffic but also have several practical properties that improve state-of-the-art regenerating codes. We implement and deploy DoubleR atop the Hadoop Distributed File System (HDFS) and show that DoubleR maintains the theoretical guarantees of DRC and improves the repair performance of regenerating codes in both node recovery and degraded read operations. [ABSTRACT FROM AUTHOR]

In the domain of business process modeling, a case denotes a situation that requires a customized treatment, and this may take place if the case workers are entitled to decide the tasks to perform as well as their ordering. In the recent CMMN (Case Management Model and Notation) standard, a case involves both an information structure and a process. The former is patterned on the file system structure and the latter is made up of stages, which are groupings of interrelated tasks. The standard leaves some open issues, such as the determination of the performers of the tasks, and the definition of the inputs and outputs of the tasks in terms of the information items affected. To address these issues, this paper proposes an extension to CMMN in which stages represent states of information items whose types along with their attributes and relationships are defined in an information model. The benefits of the extension are illustrated by means of an example that concerns the handling of papers submitted to conferences. [ABSTRACT FROM AUTHOR]

Model Based Testing is one of the most critical area to be addressed efficiently to ensure effective testing of the given project. The system implemented combines UML with FSM to cover all scenarios with all possible paths. As Finite Machine also works on the trigger where conditions are framed and if these conditions are satisfied then next action is executed; this phenomenon motivates to build a framework for generating the test cases automatically covering all paths (activity diagram in UML helps to cover all paths) and conditions ( Finite State Machine helps to frame set of conditions). Model Based Testing designed considering all paths and conditions to check all scenarios to generate detailed test cases for given project or application. [ABSTRACT FROM AUTHOR]

Primary storage dedup is difficult to be accomplished because of challenges to achieve low IO latency and high throughput while eliminating data redundancy effectively in the critical IO Path. In this paper, we design and implement the PDFS, a partially dedupped file system for primary workloads, which is built on a generalized framework using partial data lookup for efficient searching of redundant data in quickly chosen data subsets instead of the whole data. PDFS improves IO latency and throughput systematically by techniques including write path optimization, data dedup parallelization and write order preserving. Such design choices bring dedup to the masses for general primary workloads. Experimental results show that PDFS achieves 74-99 percent of the theoretical maximum dedup ratio with very small or even negative performance degradations compared with main stream file systems without dedup support. Discussions about varied configuring experiences of PDFS are also carried out. [ABSTRACT FROM PUBLISHER]

Most big-data analysis systems require users to adopt restricted abstractions to achieve scaling and system stability. While highly effective at establishing data locality and eliminating interdependencies, this approach is not easily incorporated into scientific workflows that are often complex and irregular graphs of sequential programs with multiple dependencies. To address this, we have developed an active storage cluster file system named Confuga which harnesses the file information already available in the workflow to enable efficient and controlled distribution of dependencies across active storage nodes. Confuga is built upon the idea of leveraging a job's namespace to eliminate unknown transfers and to plan the replication of all job dependencies. Replication is carried out through two opposing transfer methodologies: centrally managed push transfers and distributed pulls. We evaluate the effectiveness of the two transfer mechanisms using workflows that stress the ability of the cluster to replicate dependencies. Ultimately, we show that a balance of the two approaches achieves optimal file distribution. This is shown in two bioinformatics workflows where a careful balance of the two mechanisms leads to 48% and 77% improvements over only push or pull. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

This paper acquaints with a created application for generating Rainbow Tables and the results of testing Rainbow Tables, according to the length of the chosen chain. The paper presents a specialized application containing its own algorithms for reduction functions, changing the length of chain, generating Rainbow Tables and measuring the effectivity of the password search in detail. Within the executed tests, the dependence of Rainbow Tables size on the password length, the affection of the hash search by the size of the chosen chain and their links to collisions, which arise from the principle of using the reduction function, were observed. The results objectively describe the pros and cons of using Rainbow Tables and show the possibilities and restrictions for their effective usage. [ABSTRACT FROM AUTHOR]

The notion of defining a cluster as a component in a mixture model was put forth by Tiedeman in 1955; since then, the use of mixture models for clustering has grown into an important subfield of classification. Considering the volume of work within this field over the past decade, which seems equal to all of that which went before, a review of work to date is timely. First, the definition of a cluster is discussed and some historical context for model-based clustering is provided. Then, starting with Gaussian mixtures, the evolution of model-based clustering is traced, from the famous paper by Wolfe in 1965 to work that is currently available only in preprint form. This review ends with a look ahead to the next decade or so. [ABSTRACT FROM AUTHOR]

The state-of-the-art storage architecture of high-performance computing systems was designed decades ago, and with today's scale and level of concurrency, it is showing significant limitations. Our recent work proposed a new architecture to address the I/O bottleneck of the conventional wisdom, and the system prototype (FusionFS) demonstrated its effectiveness on up to 16 K nodes—the scale on par with today's largest supercomputers. The main objective of this paper is to investigate FusionFS's scalability towards exascale. Exascale computers are predicted to emerge by 2018, comprising millions of cores and billions of threads. We built an event-driven simulator (FusionSim) according to the FusionFS architecture, and validated it with FusionFS's traces. FusionSim introduced less than 4 percent error between its simulation results and FusionFS traces. With FusionSim we simulated workloads on up to two million nodes and find out almost linear scalability of I/O performance; results justified FusionFS's viability for exascale systems. In addition to the simulation work, this paper extends the FusionFS system prototype in the following perspectives: (1) the fault tolerance of file metadata is supported, (2) the limitations of the current system design is discussed, and (3) a more thorough performance evaluation is conducted, such as N-to-1 metadata write, system efficiency, and more platforms such as Amazon Cloud. [ABSTRACT FROM PUBLISHER]

With simple access interfaces and flexible billing models, cloud storage has become an attractive solution to simplify the storage management for both enterprises and individual users. However, traditional file systems with extensive optimizations for local disk-based storage backend can not fully exploit the inherent features of the cloud to obtain desirable performance. In this paper, we present the design, implementation, and evaluation of Coral, a cloud based file system that strikes a balance between performance and monetary cost. Unlike previous studies that treat cloud storage as just a normal backend of existing networked file systems, Coral is designed to address several key issues in optimizing cloud-based file systems such as the data layout, block management, and billing model. With carefully designed data structures and algorithms, such as identifying semantically correlated data blocks, kd-tree based caching policy with self-adaptive thrashing prevention, effective data layout, and optimal garbage collection, Coral achieves good performance and cost savings under various workloads as demonstrated by extensive evaluations. [ABSTRACT FROM PUBLISHER]

A triple system is a collection of b blocks, each of size three, on a set of v points. It is j-balanced when every two j -sets of points appear in numbers of blocks that are as nearly equal as possible, and well balanced when it is j-balanced for each j ∈ {1, 2, 3}.Well-balanced systems arise in the minimization of variance in file availability in distributed file systems. It is shown that when a triple system that is 2-balanced and 3-balanced exists, so does one that is well balanced. Using known and new results on variants of group divisible designs, constructions for well-balanced triple systems are developed. Using these, the spectrum of pairs (v, b) for which such a well-balanced triple system exists is determined completely. [ABSTRACT FROM AUTHOR]

We introduce the Forward Observer system, which is designed to provide data assurance in field data acquisition while receiving significant amounts (several terabytes per flight) of Synthetic Aperture Radar data during flights over the polar regions, which provide unique requirements for developing data collection and processing systems. Under polar conditions in the field and given the difficulty and expense of collecting data, data retention is absolutely critical. Our system provides a storage and analysis cluster with software that connects to field instruments via standard protocols, replicates data to multiple stores automatically as soon as it is written, and provides pre-processing of data so that initial visualizations are available immediately after collection, where they can provide feedback to researchers in the aircraft during the flight. [ABSTRACT FROM AUTHOR]

In the early days of computing, files where just a natural way of storing information - which reflected the way one would file their punch cards in a cabinet drawer. Unfortunately, the requirement to fragment information into such chunks, is a huge bottleneck for the evolution of global information space that the Internet has become. The concept of file causes several problems including unnatural clustering of information, unnecessary replication of data and very expensive information discovery in distributed computing environments. The overall goal of this work is to design an architecture enabling new era in computing and networking - a computing infrastructure without the concept of file. Files are seen by many specialists as one of the main bottlenecks of modern IT systems evolution. This is mostly due to a very unnatural fragmentation of information into chunks which are easier to manage by operating systems but much more difficult for information processing tools and eventually by humans themselves. [ABSTRACT FROM AUTHOR]

Existing provenance systems operate at a single layer of abstraction (workflow/process/OS) at which they record and store provenance. However, the provenance captured from different layers provides the highest benefit when integrated through a unified provenance framework. To build such a framework, a comprehensive provenance model able to represent the provenance of data objects with various semantics and granularity is the first step. In this paper, the authors propose a provenance model able to represent the provenance of any data object captured at any abstraction layer and present an abstract schema of the model. The expressive nature of the model enables a wide range of provenance queries. The authors also illustrate the utility of their model in real world data processing systems. In the paper, they also introduce a data provenance distributed middleware system composed of several different components and services that capture provenance according to their model and securely stores it in a central repository. As part of our middleware, the authors present a thin stackable file system, called FiPS, for capturing local provenance in a portable manner. FiPS is able to capture provenance at various degrees of granularity, transform provenance records into secure information, and direct the resulting provenance data to various persistent storage systems. [ABSTRACT FROM AUTHOR]

The MapReduce programming model, in which the data nodes perform both the data storing and the computation, was introduced for big-data processing. Thus, we need to understand the different resource requirements of data storing and computation tasks and schedule these efficiently over multi-core processors. In particular, the provision of high-performance data storing has become more critical because of the continuously increasing volume of data uploaded to distributed file systems and database servers. However, the analysis of the performance characteristics of the processes that store upstream data is very intricate, because both network and disk inputs/outputs (I/O) are heavily involved in their operations. In this paper, we analyze the impact of core affinity on both network and disk I/O performance and propose a novel approach for dynamic core affinity for high-throughput file upload. We consider the dynamic changes in the processor load and the intensiveness of the file upload at run-time, and accordingly decide the core affinity for service threads, with the objective of maximizing the parallelism, data locality, and resource efficiency. We apply the dynamic core affinity to Hadoop Distributed File System (HDFS). Measurement results show that our implementation can improve the file upload throughput of end applications by more than 30% as compared with the default HDFS, and provide better scalability. [ABSTRACT FROM AUTHOR]

The backup taken of a file system must be consistent, preserving data integrity across files in the file system. With file system sizes getting very large, and with demand for continuous access to data, backup has to be taken when the file system is active (is online). Arbitrarily taken online backup may result in an inconsistent backup copy. We propose a scheme referred to as mutual serializability to take a consistent backup of an active file system assuming that the file system supports transactions. The scheme extends the set of conflicting operations to include read-read conflicts, and it is shown that if the backup transaction is mutually serializable with every other transaction individually, a consistent backup copy is obtained. The user transactions continue to serialize within themselves using some standard concurrency control protocol such as Strict 2PL. We put our scheme into a formal framework to prove its correctness, and the formalization as well as the correctness proof are independent of the concurrency control protocol used to serialize user transactions. The scheme has been implemented and experiments show that consistent online backup is possible with reasonable overhead. [ABSTRACT FROM AUTHOR]

Majority of R&D outcomes consists of data, information and documents integrated into a category of Grey Literature. From portfolio of these outcomes especially publications, patents, products and innovations are interesting for scientific community and wide range of R&D stakeholders indoors and abroad. Current Research Information Systems (CRIS) are one of key software tools for data collection and access, dissemination of information about scientific activities and their outcomes. The EU standards for these systems, primarily the CERIF data format, are covered by the international association euroCRIS. The aim of providing the Information System about Science, Research and Innovation SK CRIS is to integrate all available research information at the national level. The system was designed on a data structure compatible with the CERIF data format and it became a member of the CRIS systems family. The SK CRIS contains data about research projects funded by public resources, a registry of researchers, a registry of research organisations and research and development results. Data acquisition and refilling is possible by using integration interface importing data from external systems. It concerns mainly project data imported from systems of grant agencies. Direct data entry made by research subjects by online forms is the second way how to receive data. The SK CRIS integrates also other available external data sources: registries of researchers from universities and Slovak Academy of Sciences and also publications from universities registered in the Central Registry of Publication Activity. The SK CRIS makes contextual data about science and research accessible during their life cycle. It means that a user can see whether some document was created as a result of the project. The information about its authors and originators, persons and organisations is also available, as well as usage of a document and its citations. The SK CRIS is not limited to work only with metadata. It contains also the functionality allowing the integration of collected data with fulltext documents. The benefit of the SK CRIS is mostly its ability to offer aggregated information from entered data and to present their relationship. Main characteristic of the SK CRIS is the CERIF based concept of objects (entities) with attributes, with a data model guaranteeing interoperability and full language variability. The relations and semantics allow to record the objects with relationship and characterise properties and time attributes by roles and time details. However, the system will be used mainly by scientific community but also by research management, decision makers and public. Enlargement of the SK CRIS functionality and integration of the other external data sources is our vision for near future. In the same time we consider the interconnection with other European CRIS systems. The SK CRIS information system was developed within the Activity No 4 of the NISPEZ national project implemented by the Slovak Centre of Scientific and Technical Information: Enlargement of Central Information Portal for Research, Development and Innovation (CIP RDI) with new functionalities complying with EU standards. [ABSTRACT FROM AUTHOR]

The explosive growth in data volume and complexity imposes great challenges for file systems. To address these challenges, an innovative namespace management scheme is in desperate need to provide both the ease and efficiency of data access. In almost all today’s file systems, the namespace management is based on hierarchical directory trees. This tree-based namespace scheme is prone to severe performance bottlenecks and often fails to provide real-time response to complex data lookups. This paper proposes a Semantic-Aware Namespace scheme, called SANE, which provides dynamic and adaptive namespace management for ultra-large storage systems with billions of files. SANE introduces a new naming methodology based on the notion of semantic-aware per-file namespace, which exploits semantic correlations among files, to dynamically aggregate correlated files into small, flat but readily manageable groups to achieve fast and accurate lookups. SANE is implemented as a middleware in conventional file systems and works orthogonally with hierarchical directory trees. The semantic correlations and file groups identified in SANE can also be used to facilitate file prefetching and data de-duplication, among other system-level optimizations. Extensive trace-driven experiments on our prototype implementation validate the efficacy and efficiency of SANE. [ABSTRACT FROM AUTHOR]

Data are generated and updated tremendously fast by users through any devices in anytime and anywhere in big data. Coping with these multiform data in real time is a heavy challenge. Hadoop distributed file system (HDFS) is designed to deal with data for building a distributed data center. HDFS uses the data duplicates to increase data reliability. However, data duplicates need a lot of extra storage space and funding in infrastructure. Using the deduplication technique can improve utilization of the storage space effectively. In this paper, we propose a dynamic deduplication decision to improve the storage utilization of a data center which uses HDFS as its file system. Our proposed system can formulate a proper deduplication strategy to sufficiently utilize the storage space under the limited storage devices. Our deduplication strategy deletes useless duplicates to increase the storage space. The experimental results show that our method can efficiently improve the storage utilization of a data center using the HDFS system. [ABSTRACT FROM AUTHOR]

Due to the explosive growth in the size of scientific data-sets, data-intensive computing and analysing are an emerging trend in computational science. In these applications, data pre-processing is widely adopted because it can optimise the data layout or format beforehand to facilitate the future data access. On the other hand, current research shows an increasing popularity of MapReduce framework for large-scale data processing. However, the data access patterns which are generally applied to scientific data-set are not supported by current MapReduce framework directly. This gap motivates us to provide support for these scientific data access patterns in MapReduce framework. In our work, we study the data access patterns in matrix files and propose a new concentric data layout solution to facilitate matrix data access and analysis in MapReduce framework. Concentric data layout is a data layout which maintains the dimensional property in chunk level. Contrary to the continuous data layout adopted in the current Hadoop framework, concentric data layout stores the data from the same sub-matrix into one chunk. This layout can guarantee that the average performance of data access is optimal regardless of the various access patterns. The concentric data layout requires reorganising the data before it is being analysed or processed. Our experiments are launched on a real-world halo-finding application; the results indicate that the concentric data layout improves the overall performance by up to 38%. [ABSTRACT FROM AUTHOR]