Search

Parallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high-performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real-world applications, we have created h5bench, a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one-dimensional arrays, two-dimensional meshes, three-dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre-exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library). [ABSTRACT FROM AUTHOR]

As modern supercomputers evolve to exascale, their I/O subsystems are becoming increasingly complex, making optimization of I/O for scientific applications a daunting task. Although I/O profiling tools facilitate the process of optimizing application I/O performance, legacy profiling toolslack flexibility in their level of detail and ability to correlate traces with other sources of data. Additionally, a lack of robust trace analysis tools makes it difficult to derive actionable insights from large-scale I/O traces.Darshan is an HPC I/O characterization tool that records statistics in a lightweight manner that makes it appropriate for full-time production deployment. However, Darshan’s default characterization mechanism records information at a fixed granularity. We augment Darshan by proposing Darshan eXtended Tracing (DXT) for more detailed profiling of I/O software stacks. DXT enables users and administrators to vary the level of fidelity captured by Darshan at run time without modifying or recompiling applications. This capability facilitates systematic analysis on the I/O behavior of applications and can provide useful application kernel I/O traces to help advance parallel I/O research. We have demonstrated the power of DXT by obtaining a wide range of useful statistics for multiple case studies, and we further show that DXT isable to do so same with negligible overhead.

The Cori system at NERSC has two computepartitions with different CPU architectures: a 2,004 nodeHaswell partition and a 9,688 node KNL partition, whichranked as the 5th most powerful and fastest supercomputeron the November 2016 Top 500 list. The compute partitionsshare a common storage configuration, and understanding theIO performance gap between them is important, impactingnot only to NERSC/LBNL users and other national labs, butalso to the relevant hardware vendors and software developers.In this paper, we have analyzed performance of single coreand single node IO comprehensively on the Haswell and KNLpartitions, and have discovered the major bottlenecks, whichinclude CPU frequencies and memory copy performance. Wehave also extended our performance tests to multi-node IOand revealed the IO cost difference caused by network latency,buffer size, and communication cost. Overall, we have developeda strong understanding of the IO gap between Haswell and KNLnodes and the lessons learned from this exploration will guideus in designing optimal IO solutions in many-core era.

Containers enable high performance and easy deployment due to their lightweight architecture, thus facilitating resource utilization in edge computing nodes. Secure container runtimes have attracted significant attention because of the necessity for overcoming the security vulnerabilities of containers. As the runtimes adopt an additional layer such as virtual machines and user-space kernels to enforce isolation, the container performance can be degraded. Even though previous studies presented experimental results on performance evaluations of secure container runtimes, they lack a detailed analysis of the root causes that affect the performance of the runtimes. This paper explores the architecture of three secure container runtimes in detail: Kata containers, gVisor, and Firecracker. We focus on file I/O, which is one of the key aspects of container performance. In addition, we present the results of the user- and kernel-level profiling and reveal the major factors that impact the file I/O performance of the runtimes. As a result, we observe three key findings: (1) Firecracker shows the highest file I/O performance as it allows for utilizing the page cache inside VMs, and (2) Kata containers offer the lowest file I/O performance by consuming the largest amount of CPU resources. Also, we observe that gVisor scales well as the block size increases because the file I/O requests are mainly handled by the host OS similar to native applications. [ABSTRACT FROM AUTHOR]

As modern supercomputers evolve to exascale, their I/O subsystems are becoming increasingly complex, making optimization of I/O for scientific applications a daunting task. Although I/O profiling tools facilitate the process of optimizing application I/O performance, legacy profiling toolslack flexibility in their level of detail and ability to correlate traces with other sources of data. Additionally, a lack of robust trace analysis tools makes it difficult to derive actionable insights from large-scale I/O traces.Darshan is an HPC I/O characterization tool that records statistics in a lightweight manner that makes it appropriate for full-time production deployment. However, Darshan’s default characterization mechanism records information at a fixed granularity. We augment Darshan by proposing Darshan eXtended Tracing (DXT) for more detailed profiling of I/O software stacks. DXT enables users and administrators to vary the level of fidelity captured by Darshan at run time without modifying or recompiling applications. This capability facilitates systematic analysis on the I/O behavior of applications and can provide useful application kernel I/O traces to help advance parallel I/O research. We have demonstrated the power of DXT by obtaining a wide range of useful statistics for multiple case studies, and we further show that DXT isable to do so same with negligible overhead.

Optimized for parallel operations, Intel's second generation Xeon Phi processor, code-named Knights Landing (KNL), is actively utilized in high performance computing systems based on its highly integrated cores and high-bandwidth on-package memory, Multi-Channel DRAM (MCDRAM). Recently, the emergence of data-intensive applications and the utilization of many-core processors have further increased I/O performance requirements of high performance computing systems. Therefore, it is necessary to understand and analyze the I/O characteristics of many integrated core systems. In this paper, we experimentally analyze the I/O characteristics of KNL, focusing on single-thread, buffered-write operations. We determine that KNL has a bottleneck in its buffered write operation that utilizes page cache. To find this bottleneck point and identify its cause, we conduct the experiments in two different ways. First, we measure the execution time of kernel functions through kernel I/O path. Second, we measure the occurrence count of system events such as cache-misses and branch-misses. With results from these experiments, we discuss the characteristics on KNL's I/O performance involving the performance bottlenecks. [ABSTRACT FROM AUTHOR]

Parallel Input output is an essential component of modern high-performance computing (HPC). Obtaining good I/O performance for a broad range of applications on diverse HPC platforms is a major challenge, in part, because of complex inter dependencies between I/O middleware and hardware. The parallel file system and I/O middleware layers all offer optimization parameters that can, in theory, result in better I/O performance. Unfortunately, the right combination of parameters is highly dependent on the application, HPC platform, problem size, and concurrency. Scientific application developers do not have the time or expertise to take on the substantial burden of identifying good parameters for each problem configuration. They resort to using system defaults, a choice that frequently results in poor I/O performance. We expect this problem to be compounded on exascale-class machines, which will likely have a deeper software stack with hierarchically arranged hardware resources. We present as a solution to this problem an autotuning system for optimizing I/O performance, I/O performance modeling, I/O tuning, and I/O patterns. We demonstrate the value of this framework across several HPC platforms and applications at scale.

Containers enable high performance and easy deployment due to their lightweight architecture, thus facilitating resource utilization in edge computing nodes. Secure container runtimes have attracted significant attention because of the necessity for overcoming the security vulnerabilities of containers. As the runtimes adopt an additional layer such as virtual machines and user-space kernels to enforce isolation, the container performance can be degraded. Even though previous studies presented experimental results on performance evaluations of secure container runtimes, they lack a detailed analysis of the root causes that affect the performance of the runtimes. This paper explores the architecture of three secure container runtimes in detail: Kata containers, gVisor, and Firecracker. We focus on file I/O, which is one of the key aspects of container performance. In addition, we present the results of the user- and kernel-level profiling and reveal the major factors that impact the file I/O performance of the runtimes. As a result, we observe three key findings: (1) Firecracker shows the highest file I/O performance as it allows for utilizing the page cache inside VMs, and (2) Kata containers offer the lowest file I/O performance by consuming the largest amount of CPU resources. Also, we observe that gVisor scales well as the block size increases because the file I/O requests are mainly handled by the host OS similar to native applications.

Parallel Input output is an essential component of modern high-performance computing (HPC). Obtaining good I/O performance for a broad range of applications on diverse HPC platforms is a major challenge, in part, because of complex inter dependencies between I/O middleware and hardware. The parallel file system and I/O middleware layers all offer optimization parameters that can, in theory, result in better I/O performance. Unfortunately, the right combination of parameters is highly dependent on the application, HPC platform, problem size, and concurrency. Scientific application developers do not have the time or expertise to take on the substantial burden of identifying good parameters for each problem configuration. They resort to using system defaults, a choice that frequently results in poor I/O performance. We expect this problem to be compounded on exascale-class machines, which will likely have a deeper software stack with hierarchically arranged hardware resources. We present as a solution to this problem an autotuning system for optimizing I/O performance, I/O performance modeling, I/O tuning, and I/O patterns. We demonstrate the value of this framework across several HPC platforms and applications at scale.

Flash-based solid state drives (SSDs) are widely adopted in both industry and the academia since SSDs offer higher performance, lower latency, and lower power consumption compared with traditional hard disk drives (HDDs). Unfortunately, the performance of SSDs can be affected by I/O access patterns. For example, random I/O operation degrades the SSD performance compared with sequential I/O operation since the random I/O operation reduces the spatial locality and increases garbage collection (GC) overhead. To handle this issue, in this article, we propose an efficient block address transformation scheme in the block layer to improve both performance and portability. To do this, we first transform random access patterns to sequential access patterns by sequentializing the block addresses in the block layer. Second, we devise a mapping table for managing transformed block addresses. Third, we support correct read operations and transaction processing for updating the mapping table to avoid sacrificing the consistency. Finally, we provide a cleaning scheme to reclaim invalid data generated by the transformed sequential access. This scheme increases spatial locality, reduces GC overhead, and operates well on any file systems or devices. Experimental results show the proposed scheme improves performance by up to 2x, 1.86x, 2.15x, and 42.8% compared with existing scheme in the case of diverse file systems such as EXT4, XFS, BTRFS, and F2FS, respectively.

Recently, flash-based solid-state drives (SSDs) are widely used in industry and academia due to their higher bandwidth and lower latency compared with traditional hard disk drives (HDDs). Furthermore, SSDs with the Non-Volatile Memory Express (NVMe) interface can provide higher performance and ultra-low latency compared with the Serial AT Attachment (SATA) SSDs. Due to their high performance, NVMe SSDs are adopted in many systems as fast storage devices. However, the performance of NVMe SSDs can be negatively affected by I/O access patterns. For example, random write access patterns can have negative impacts on performance due to the unique characteristics of SSDs such as out-of-place update and garbage collection. In this paper, we propose an address remapping scheme to improve the I/O performance of NVMe SSDs. Our proposed scheme transforms random access patterns into sequential access patterns in the NVMe device driver. This allows our scheme to improve the I/O performance of NVMe SSDs while supporting widely used file systems such as EXT4, XFS, BTRFS, and F2FS without any modification to the device. Experimental results show that our proposed scheme can improve the performance of NVMe SSD by up to 64.1% compared with the existing scheme.

I/O efficiency is essential to productivity in scientific computing, especially as many scientific domains become more data-intensive. Many characterization tools have been used to elucidate specific aspects of parallel I/O performance, but analyzing components of complex I/O subsystems in isolation fails to provide insight into critical questions: how do the I/O components interact, what are reasonable expectations for application performance, and what are the underlying causes of I/O performance problems? To address these questions while capitalizing on existing component-level characterization tools, we propose an approach that combines on-demand, modular synthesis of I/O characterization data into a unified monitoring and metrics interface (UMAMI) to provide a normalized, holistic view of I/O behavior. We evaluate the feasibility of this approach by applying it to a month-long benchmarking study on two distinct largescale computing platforms. We present three case studies that highlight the importance of analyzing application I/O performance in context with both contemporaneous and historical component metrics, and we provide new insights into the factors affecting I/O performance. By demonstrating the generality of our approach, we lay the groundwork for a production-grade framework for holistic I/O analysis.

I/O efficiency is essential to productivity in scientific computing, especially as many scientific domains become more data-intensive. Many characterization tools have been used to elucidate specific aspects of parallel I/O performance, but analyzing components of complex I/O subsystems in isolation fails to provide insight into critical questions: how do the I/O components interact, what are reasonable expectations for application performance, and what are the underlying causes of I/O performance problems? To address these questions while capitalizing on existing component-level characterization tools, we propose an approach that combines on-demand, modular synthesis of I/O characterization data into a unified monitoring and metrics interface (UMAMI) to provide a normalized, holistic view of I/O behavior. We evaluate the feasibility of this approach by applying it to a month-long benchmarking study on two distinct largescale computing platforms. We present three case studies that highlight the importance of analyzing application I/O performance in context with both contemporaneous and historical component metrics, and we provide new insights into the factors affecting I/O performance. By demonstrating the generality of our approach, we lay the groundwork for a production-grade framework for holistic I/O analysis.

The Cori system at NERSC has two computepartitions with different CPU architectures: a 2,004 nodeHaswell partition and a 9,688 node KNL partition, whichranked as the 5th most powerful and fastest supercomputeron the November 2016 Top 500 list. The compute partitionsshare a common storage configuration, and understanding theIO performance gap between them is important, impactingnot only to NERSC/LBNL users and other national labs, butalso to the relevant hardware vendors and software developers.In this paper, we have analyzed performance of single coreand single node IO comprehensively on the Haswell and KNLpartitions, and have discovered the major bottlenecks, whichinclude CPU frequencies and memory copy performance. Wehave also extended our performance tests to multi-node IOand revealed the IO cost difference caused by network latency,buffer size, and communication cost. Overall, we have developeda strong understanding of the IO gap between Haswell and KNLnodes and the lessons learned from this exploration will guideus in designing optimal IO solutions in many-core era.

User‐level file systems are usually adopted to bridge the gap between efficacy and efficiency of file system developments for new applications' I/O demands. And the widely known user‐space file system framework, FUSE, is commonly utilized to deployed user‐level file systems. This article first uses a popular stack‐able file system as a case study to exam how FUSE affects I/O performance. Based on the testing and analytical results, this article then presents SHC, an implementation method to implement a user‐level file system without FUSE intervention. Experimental results indicate that SHC improves write bandwidth by up to 5.6x compared with that of FUSE and present leading superiority on read cases. [ABSTRACT FROM AUTHOR]

Summary: An effective way to mitigate the lifetime and performance problems of NAND flash SSDs induced by garbage collection is separating the cold data and hot data. However, all existing solutions do this separation from the viewpoint of the hot data and thus suffer from expensive cost on monitoring the update behaviors of workloads. Different from the conventional wisdom, we propose to do the separation from the viewpoint of the cold data, which is rooted on the following two dedicated observations. First, the data migrated by garbage collection demonstrate a cold property compared with the other data. Second, it is inexpensive to separate the former data from the latter data. In the proposed method, named Isolation, we inexpensively separate the cold data via garbage collection. More specifically, through writing the data migrated by garbage collection (cold data) to dedicated blocks and writing the other data (hot data) to other blocks, the separation is done naturally. To evaluate Isolation, we conduct extensive trace‐driven simulations under seven typical workloads. The simulation results demonstrate the effectiveness and efficiency of Isolation. In most cases, Isolation can reduce the number of erasures and average I/O response time up to 47.3% and 80.1%, respectively. Existing approaches integrated with Isolation can further reduce the number of erasures and average I/O response time up to 7.8%‐18.5% and 10.7%‐41.4%, respectively. [ABSTRACT FROM AUTHOR]

The use of flash based Solid State Drives (SSDs) has expanded rapidly into the cloud computing environment. In cloud computing, ensuring the service level objective (SLO) of each server is the major criterion in designing a system. In particular, eliminating performance interference among virtual machines (VMs) on shared storage is a key challenge. However, studies on SSD performance to guarantee SLO in such environments are limited. In this paper, we present analysis of I/O behavior for a shared SSD as storage in terms of proportionality and stability. We show that performance SLOs of SSD based storage systems being shared by VMs or tasks are not satisfactory. We present and analyze the reasons behind the unexpected behavior through examining the components of SSDs such as channels, DRAM buffer, and Native Command Queuing (NCQ). We introduce two novel SSD-aware host level I/O schedulers on Linux, called A+CFQ and H+BFQ, based on our analysis and findings. Through experiments on Linux, we analyze I/O proportionality and stability in multi-tenant environments. In addition, through experiments using real workloads, we analyze the performance interference between workloads on a shared SSD. We then show that the proposed I/O schedulers almost eliminate the interference effect seen in CFQ and BFQ, while still providing I/O proportionality and stability for various I/O weighted scenarios.

While a dual-copy redundant array of independent disk (RAID) buffering can remove single point of failure in the buffer of RAID-enabled solid-state drives (SSDs), it loses a half of buffer space efficiency as a result. This article introduces a patch-based data management scheme for dual-copy buffers in RAID-enabled SSDs, for better improving buffer use efficiency. Different from conventional methods that caches data/parity chunks in the RAID buffer, it caches the modified portions of data chunks (called data patches) and then employs a cost-evaluation model to direct patch data replacement. Besides, in order to correctly respond to applications’ read/write requests, we design the patch-based read/write mode by taking the original dirty data chunks and relevant up-to-date patches into account. Through a series of simulation tests on several realistic disk traces, we illustrate that patch-based data management can work for RAID-enabled SSDs to enhance buffer use efficiency, and then yield better I/O performance. Especially, our proposal can noticeably reduce the I/O latency by 39.1% and the number of block erases by 3.9% on average, in contrast to state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This paper proposes a new data placement policy to allocate data blocks across storage servers of the distributed/parallel file systems, for yielding even block access workload distribution. To this end, we first analyze the history of block access sequence of a specific application and then introduce a k-partition algorithm to divide data blocks into multiple groups, by referring their access frequency. After that, each group has almost the same access workloads, and we can thus distribute these block groups onto storage servers of the distributed file system, to achieve the goal of uniformly assigning data blocks when running the application. In summary, this newly proposed data placement policy can yield not only an even data distribution but also the block data access balance. The experimental results show that the proposed scheme can greatly reduce I/O time and better improve utilization of storage servers when running the database-relevant applications, compared with the commonly used block data placement strategy, i.e., the round-robin placement policy.

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. Parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new applications, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performing parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries. [ABSTRACT FROM AUTHOR]

Mobile devices, such as smartphones, have become a necessity in our daily life. However, users may notice that after being used for a long time, mobile devices begin to exhibit a sluggish response. Based on an empirical study on a collection of aged smartphones, this work identified that file fragmentation is among the key factors that contribute to the progressive degradation of response time. This study takes a three-step approach: First, this study designed a set of reproducible file-system aging processes based on User-Interface (UI) script replay. Through the aging processes, it confirmed that file fragmentation quickly emerged, and SQLite files were among the most severely fragmented files. Second, based on the workloads of a selection of popular mobile applications, this study observed that file fragmentation did have an impact on user-perceived latencies. Specifically, the launching time of Chrome on an aged file system was 79 percent slower than it was on a pristine file system. Third, this study evaluated existing treatments of file fragmentation, including space preallocation, persistent journal, and file defragmentation to understand their efficacies and limitations. This study also evaluated a state-of-the-art copyless defragmenter, $\mathsf{janusd}$janusd, to show its advantage over the existing methods. [ABSTRACT FROM AUTHOR]

Energy consumption is one of the most significant aspects of large-scale storage systems where multilevel caches are widely used. In a typical hierarchical storage structure, upper-level storage serves as a cache for the lower level, forming a distributed multilevel cache system. In the past two decades, several classic LRU-based multilevel cache policies have been proposed to improve the overall I/O performance of storage systems. However, few power-aware multilevel cache policies focus on the storage devices in the bottom level, which consume more than 27% of the energy of the whole system [1]. To address this problem, we propose a novel power-aware multilevel cache (PAM) policy that can reduce the energy consumption of high-performance and I/O bandwidth storage devices. In our PAM policy, an appropriate number of cold dirty blocks in the upper level cache are identified and selected to flush directly to the storage devices, providing high probability extension of the lifetime of disks in standby mode. To demonstrate the effectiveness of our proposed policy, we conduct several simulations with real-world traces. Compared to existing popular cache schemes such as PA-LRU, PB-LRU, and Demote, PAM reduces power consumption by up to 15% under different I/O workloads, and improves energy efficiency by up to 50.5%. [ABSTRACT FROM AUTHOR]

Distributed file systems have been commonly used as the back end to provide high-performance I/O services for hosting complicated data processing, such as database workloads. By buffering the frequently used data in the local file cache, which is called the client caching mechanism, can reduce the I/O latency and improve the performance of file access. However, the overhead and complexity for ensuring data consistency may offset the performance benefits caused by data caching. This paper proposes an adaptive, per-block cache update mechanism, which can ensure data consistency of block data in a distributed file system, with low synchronization overhead. Specifically, based on the recent history of read/write requests, the proposed scheme can adaptively select the best-suited consistency update policy for each data block at run time. The experiments on database applications show that our newly proposed mechanism can greatly reduce the overhead resulted by maintaining cache consistency, especially for the workloads having fluctuating read/write ratios. As a consequence, the system performance can be also enhanced. [ABSTRACT FROM AUTHOR]

Many problems in science and engineering are usually emulated as a set of mutually interacting models, resulting in a coupled or multiphysics application. These component models show challenges originating from their interdisciplinary nature and from their computational and algorithmic complexities. In general, these models are independently developed and maintained, so that they commonly employ the global file system for exchanging their data in the coupled application. To effectively use the local file cache on the compute node for exchanging the data among the processes of such applications, and consequently boosting I/O performance, this article presents a novel mechanism to migrate a process from one compute node to another node on the basis of block I/O dependency. In this newly proposed mechanism, the block I/O dependency between two involved processes running on the different nodes is profiled as block access similarity by taking advantage of the Cohen's kappa statistic. Then, the process is supposed to be dynamically migrated from its source node to the destination node, on which there is another process having heavy block I/O dependency. As a result, both processes can exchange their data by utilizing the local file cache instead of the global file system to reduce I/O time. The experimental results demonstrate that the I/O performance can be significantly improved, and the time required for executing the application can be resultantly decreased, as expected. [ABSTRACT FROM AUTHOR]

Key–value stores are being widely used as the storage system for large-scale internet services and cloud storage systems. However, they are rarely used in HPC systems, where parallel file systems are the dominant storage solution. In this study, we examine the architecture differences and performance characteristics of parallel file systems and key–value stores. We propose using key–value stores to optimize overall Input/Output (I/O) performance, especially for workloads that parallel file systems cannot handle well, such as the cases with intense data synchronization or heavy metadata operations. We conducted experiments with several synthetic benchmarks, an I/O benchmark, and a real application. We modeled the performance of these two systems using collected data from our experiments, and we provide a predictive method to identify which system offers better I/O performance given a specific workload. The results show that we can optimize the I/O performance in HPC systems by utilizing key–value stores. [ABSTRACT FROM AUTHOR]

With the enormous and increasing user demand, I/O performance is one of the primary considerations to build a data center. Several new technologies in data centers, such as tiered storage, prompt the widespread usage of multilevel cache techniques. In these storage systems, the upper level storage typically serves as a cache for the lower level, which forms a distributed multilevel cache system. However, although many excellent multilevel cache algorithms have been proposed to improve the I/O performance, they still have potential to be enhanced by investigating the history information of hints. To address this challenge, in this paper, we propose a novel hint frequency based approach (HFA), to improve the overall multilevel cache performance of storage systems. The main idea of HFA is using hint frequencies (the total number of demotions/promotions by employing demote/promote hints) to efficiently explore the valuable history information of data blocks among multiple levels. HFA can be applied with several popular multilevel cache algorithms, such as Demote, Promote and Hint-K. Simulation results show that, compared with original multilevel cache algorithms such as Demote, Promote and Hint-K, HFA can improve the I/O performance by up to 20% under different I/O workloads. [ABSTRACT FROM AUTHOR]

Cloud computing increasingly leverages SMP virtual machines (VMs) to host multi-threaded applications. Interrupt balancing as a problem becomes more challenging because VMs are subject to the hypervisor's scheduling. Since the scheduling delays are typically tens of milliseconds, when they are added to one VM's interrupt delivery, they can seriously degrade the VM's I/O performance. Traditional balancing techniques are designed for dedicated environments, which cannot work well in virtualized environments because VMs are disallowed to directly control the hardware in many cases. In this paper, we present hBalance, a very simple approach to offload interrupt load balancing from SMP-VMs to the hypervisor. To accelerate the interrupt processing, our approach does not require shortening the hypervisor's scheduling time slice, but dynamically redirects interrupts from preempted virtual CPUs to running ones in a balanced manner. hBalance supports both Fully Virtualiized (FV) guests and Para-Virtualized (PV) guests, and exhibits high portability among various hypervisors. With our prototype implementation in Xen, the experimental results with both micro-level and application-level benchmarks show that hBalance significantly improves SMP-VMs’ I/O performance while introduces moderate overhead. [ABSTRACT FROM AUTHOR]

This paper presents two mechanisms that can significantly improve the I/O performance of both hard and solid-state drives for read operations: KDSim and REDCAP. KDSim is an in-kernel disk simulator that provides a framework for simultaneously simulating the performance obtained by different I/O system mechanisms and algorithms, and for dynamically turning them on and off, or selecting between different options or policies, to improve the overall system performance.REDCAPis aRAMbased disk cache that effectively enlarges the built-in cache present in disk drives. By using KDSim, this cache is dynamically activated/deactivated according to the throughput achieved. Results show that, by using KDSim and REDCAP together, a system can improve its I/O performance up to 88% for workloads with some spatial locality on both hard and solid-state drives, while it achieves the same performance as a 'regular system' for workloads with random or sequential access patterns. [ABSTRACT FROM AUTHOR]

With the explosive growth in data volume, the I/O bottleneck has become an increasingly daunting challenge for big data analytics in the Cloud. Recent studies have shown that moderate to high data redundancy clearly exists in primary storage systems in the Cloud. Our experimental studies reveal that data redundancy exhibits a much higher level of intensity on the I/O path than that on disks due to relatively high temporal access locality associated with small I/O requests to redundant data. Moreover, directly applying data deduplication to primary storage systems in the Cloud will likely cause space contention in memory and data fragmentation on disks. Based on these observations, we propose a performance-oriented I/O deduplication, called POD, rather than a capacity-oriented I/O deduplication, exemplified by iDedup, to improve the I/O performance of primary storage systems in the Cloud without sacrificing capacity savings of the latter. POD takes a two-pronged approach to improving the performance of primary storage systems and minimizing performance overhead of deduplication, namely, a request-based selective deduplication technique, called Select-Dedupe, to alleviate the data fragmentation and an adaptive memory management scheme, called iCache, to ease the memory contention between the bursty read traffic and the bursty write traffic. We have implemented a prototype of POD as a module in the Linux operating system. The experiments conducted on our lightweight prototype implementation of POD show that POD significantly outperforms iDedup in the I/O performance measure by up to 87.9 percent with an average of 58.8 percent. Moreover, our evaluation results also show that POD achieves comparable or better capacity savings than iDedup. [ABSTRACT FROM PUBLISHER]

SUMMARY We propose a new method for tiered storage and evaluate its characteristics. In this method, the fast tier is divided into two areas and the data in each area are managed on the basis of two I/O (input/output) measurement periods. When I/Os to these areas increase, the proposed method allocates the frequently accessed areas to a fast tier in advance, thereby improving storage I/O performance. We evaluate the proposed method by simulating storage I/Os and data migration between tiers. When I/O locality is high or frequently accessed areas rarely move, the proposed method is more effective and improves storage I/O performance by up to 44%. [ABSTRACT FROM AUTHOR]

I/O performance has been critical for large-scale distributed systems. Many approaches, including hint-based multilevel cache, have been proposed to smooth the gap between different levels. These solutions demote or promote cache blocks based on the latest history information, which is insufficient for applications where frequent demote and promote operations occur. In this paper, we propose a novel multilevel buffer cache using K-step hints (Hint-K) to improve the I/O performance of distributed systems. The basic idea is to promote a block from the lower level cache to the higher level(s) or demote a block vice versa based on the block's previous K-step promote or demote operations, which are referred to as K-step hints. If we make an analogy between Hint-K and LRU-K, then LRU-K keeps track of the times of last K references for blocks within a single cache level, while our Hint-K keeps track of the information of the last K movements (either demote or promote) of blocks among different cache levels. We develop our Hint-K algorithms and design a mathematical model that can efficiently describe the activeness of any block in any cache level. Simulation results show that Hint-K achieves better performance compared to the existing popular multilevel cache schemes such as PROMOTE, DEMOTE, and MQ under different I/O workloads. [ABSTRACT FROM AUTHOR]

Applications in the cloud computing era have the emergent requirement of fast I/O support. Compared with a hard drive disk, a solid state disk (SSD) has low delay, low energy consumption, high throughput and other advantages. However, the semantics of an SSD cannot be recognized by current virtual machine monitors. The trim instruction, which plays an important role in space management of an SSD, cannot be passed to the underlying SSD device in a virtualized environment. So, how to bridge the semantics gap between the application layer and virtualization layer for SSD devices should be an important problem. We propose Vtrim to solve the above problem in this paper. Vtrim monitors the operations in the virtual machine and sends the SSD semantics to the Domain 0 immediately. And then the semantics are translated into the operations of Domain 0, which can trigger the SSD's local instructions. To improve the write performance in multiple guest operating systems, we set a Vtrim cache to buffer all instructions from the guests and flush them into an SSD in a well-scheduled way. Experiment results in the para-virtualization environments with Vtrim show that the random write performance is improved by even up to 100% and the average response time is reduced by up to 40%. [ABSTRACT FROM AUTHOR]

Infrastructure as a Service clouds often use virtual machines to host different customers on the same physical hardware. This form of resource sharing can lead to unpredicatable performance degradations for the individual customer, especially with regard to data-intensive applications, which heavily depend on stable I/O characteristics. One traditional approach to cope with I/O fluctuations is adaptive online compression. In this paper we present a new scheme for adaptive online compression which has been explicitly designed to work in co-located virtual machine environments. In contrast to existing adaptive online compression schemes, the decision model of our approach does not rely on the system metrics CPU utilization and I/O bandwidth, which we demonstrate to be often displayed inaccurately inside XEN, KVM, and Amazon EC2-based virtual machines. Instead, it only considers the application data rate. Without requiring any calibration or training phase our adaptive compression scheme can improve the I/O throughput of virtual machines up to a factor of four as shown through extended experimental evaluations. [ABSTRACT FROM AUTHOR]

Abstract: Association rule mining, in which generating frequent patterns is a key step, is an effective way of identifying inherent and unknown interrelationships between characteristics of celestial spectra data and its physicochemical properties. In this study, we first make use of the first-order predicate logic to represent knowledge derived from celestial spectra data. Next, we propose a concept of constrained frequent pattern trees (CFP) along with an algorithm used to construct CFPs, aiming to improve the efficiency and pertinence of association rule mining. Finally, we quantitatively evaluate the CPU and I/O performance of our novel interrelation analysis method using a variety of real-world data sets. Our experimental results show that it is practical to study the laws of celestial bodies using our new interrelation analysis method to discover correlations between celestial spectra data characteristics and the physicochemical properties. [Copyright &y& Elsevier]