Search

Traditional LSM-tree-based key-value storage systems face significant read amplification issues due to the multi-level structure of LSM-tree, the unordered SSTable files in Level 0, and the lack of an in-memory index structure for key-value pairs. We observed that, under the influence of workloads with locality features, key-value pairs exhibit a range-specific access intensity. Addressing the three reasons for LSM-tree read amplification, we have utilized the range-specific access intensity of workload to propose a multi-tree structure consisting of a B+ tree, a single-level hot tree, and an LSM-tree with partition-based Level 0. This aims to enhance the read performance of LSM-tree-based key-value storage systems. We designed the prototype, MTDB, based on LevelDB. The experimental results show that MTDB's read performance is 1.62× to 2.02× that of LevelDB, and it approaches or exceeds the read performance of KVell and Bourbon while reducing memory overhead by 58.85%–86%. [ABSTRACT FROM AUTHOR]

Deploying LSM-tree-based key-value stores on distributed file systems is a common approach to building distributed key-value storage systems, such as RocksDB on HDFS (RoH). However, due to the inherent read amplification characteristics of the LSM-tree, RoH faces performance degradation issues in read operations, including gets and scans, caused by the transmission of a large amount of irrelevant data between data nodes and the master node. To address these challenges, we firstly propose a computation offloading strategy that shifts get and scan operations from the master node to data nodes to reduce unnecessary data transmission. To boost the read operations, we then design concurrent execution methods: Concurrent-Get and Concurrent-Scan. Finally, we implement our prototype system, CDNRocks, based on RoH, and conduct extensive experiments in a distributed multi-node environment to demonstrate the effectiveness of CDNRocks. The results indicate that CDNRocks outperforms RoH by exhibiting better read performance, less data transmission, and lower CPU utilization on the master node. Furthermore, compared to other distributed key-value storage systems like Cassandra and HBase, CDNRocks excels in read performance within small cluster environments(with node counts up to 9), while achieving a balanced trade-off in load performance and efficient CPU and memory utilization. [ABSTRACT FROM AUTHOR]

With the continuous development of information technology, distributed database has become a research hotspot. Due to the limit support for SQL and defects in transaction processing and consistency of distributed databases based on NoSQL architecture, NewSQL databases based on LSM-Tree become gradually the mainstream of applications, such as TiDB and OceanBase. The distributed LSM-Tree storage architecture divides the data into baseline data and incremental data. Through the compaction operation, the incremental data of different partitions and the baseline data are continuously merged and stored on the disk, thereby reducing memory pressure. However, compaction will occupy a large amount of system resources and seriously affect system availability. This paper proposes an asynchronous compaction mechanism based on LSM-Tree architecture. By subdividing the compaction process, the data merging is asynchronous, which effectively shortens the time for a single compaction operation. Experiments show that the asynchronous compaction mechanism proposed in this paper can significantly shorten the data merging time and improve the robustness and usability of the system in high-frequency writing scenarios.

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

LSM-based storage systems are widely used for superior write performance on block devices. However, they currently fail to efficiently support secondary indexing, since a secondary index query operation usually needs to retrieve multiple small values, which scatter in multiple LSM components. In this work, we revisit secondary indexing in LSM-based storage systems with byte-addressable persistent memory (PM). Existing PM-based indexes are not directly competent for efficient secondary indexing. We propose Perseid, an efficient PM-based secondary indexing mechanism for LSM-based storage systems, which takes into account both characteristics of PM and secondary indexing. Perseid consists of (1) a specifically designed secondary index structure that achieves high-performance insertion and query, (2) a lightweight hybrid PM-DRAM and hash-based validation approach to filter out obsolete values with subtle overhead, and (3) two adapted optimizations on primary table searching issued from secondary indexes to accelerate non-index-only queries. Our evaluation shows that Perseid outperforms existing PM-based indexes by 3–7× and achieves about two orders of magnitude performance of state-of-the-art LSM-based secondary indexing techniques even if on PM instead of disks. [ABSTRACT FROM AUTHOR]

Emerging Zoned Namespace (ZNS) is a new-style Solid State Drive (SSD) that manages data in a zoned manner, which can achieve higher performance by strictly obeying the sequential write mode in each zone and alleviating the redundant overhead of garbage collections. Unfortunately, flash memory usually has a serious problem with limited program/erase cycles. Meanwhile, inappropriate data placement strategy of storage systems can lead to imbalanced wear among zones, severely reducing the lifespan of ZNS SSDs. In this paper, we propose a Wear-Leveling-Aware Data Placement (WADP) to solve this problem with negligible performance cost. First, WADP employs a wear-aware empty zone allocation algorithm to quantify the resets of zones and choose the less-worn zone for each allocation. Second, to prevent long-term zone occupation of infrequently written data (namely cold data), we propose a wear-leveling cold zone monitoring mechanism to identify cold zones dynamically. Finally, WADP adopts a real-time I/O pressure-aware data migration mechanism to adaptively migrate cold data for achieving wear-leveling among zones. We implement the proposed WADP in ZenFS and evaluate it with widely used workloads. Compared with state-of-the-art solutions, i.e., LIZA and FAR, the experimental results show that WADP can significantly reduce the standard deviation of zone resets while maintaining decent performance.

Log-structured-merge tree or LSM-tree is a technological underpinning in key-value (KV) stores to support a wide range of performance-critical applications. By conducting data re-organization in the background by virtue of compaction operations, the KV stores have the potential to swiftly service write requests with sequential batched disk writes and read requests for KV items constantly sorted by the compaction. Compaction demands high I/O bandwidth and CPU speed to facilitate quality service to user read/write requests. With the emergence of high-speed SSDs, CPUs are increasingly becoming a performance bottleneck. To mitigate the bottleneck limiting the KV-store's performance and that of the applications supported by the store, we propose a system - gLSM - to leverage GPGPU to remarkably accelerate the compaction operations. gLSM fully utilizes the parallelism and computational capability inside GPGPUs to improve the compaction performance. We design a driver framework to parallelize compaction operations handled between a pair of CPU and GPGPU. We employ data independence and GPGPU-orient radix-sorting algorithm to concurrently conduct compaction. A key-value separation method is devised to slash the transfer of data volume from CPU-side memory to the GPGPU counterpart. The results reveal that gLSM improves the throughput and compaction bandwidth by up to a factor of 2.9 and 26.0, respectively, compared with the four state-of-the-art KV stores. gLSM also reduces the write latency by 73.3%. gLSM exhibits a performance improvement by up to 45% compared against its variant where there are no KV separation and collaboration sort modules. [ABSTRACT FROM AUTHOR]

Key–value (KV) stores based on the LSM-tree have become the mainstream of contemporary store engines, but there are problems with high write and read amplification. Moreover, the real-world workload has a high data skew, and the existing KV store lacks hot-awareness, leading to its unreliable and poor performance on the highly skewed real-world workload. In this paper, we propose HoaKV, which unifies the key design ideas of hot issues, KV separation, and hybrid indexing technology in a system. Specifically, HoaKV uses the heat differentiation in KV pairs to manage the hot data and the cold data and conducts real-time dynamic adjustment data classification management. It also uses partial KV separation technology to manage differential KV pairs for large and small KV pairs in the cold data. In addition, HoaKV uses hybrid indexing technology to index the hot data and the cold data, respectively, to improve the performance of reading, writing, and scanning at the same time. In the mixed read and write workloads experments show that HoaKV performs significantly better than several state-of-the-art KV store technologies such as LevelDB, RocksDB, PebblesDB, and WiscKey. [ABSTRACT FROM AUTHOR]

The Log-Structured Merge Tree (LSM-Tree) is widely used in key-value (KV) stores because of its excwrite performance. But LSM-Tree-based KV stores still have the overhead of write-ahead log and write stall caused by slow L0 flush and L0-L1 compaction. New byte-addressable, persistent memory (PM) devices bring an opportunity to improve the write performance of LSM-Tree. Previous studies on PM-based LSM-Tree have not fully exploited PM’s “dual role” of main memory and external storage. In this article, we analyze two strategies of memtables based on PM and the reasons write stall problems occur in the first place. Inspired by the analysis result, we propose FlatLSM, a specially designed flat LSM-Tree for non-volatile memory based KV stores. First, we propose PMTable with separated index and data. The PM Log utilizes the Buffer Log to store KVs of size less than 256B. Second, to solve the write stall problem, FlatLSM merges the volatile memtables and the persistent L0 into large PMTables, which can reduce the depth of LSM-Tree and concentrate I/O bandwidth on L0-L1 compaction. To mitigate write stall caused by flushing large PMTables to SSD, we propose a parallel flush/compaction algorithm based on KV separation. We implemented FlatLSM based on RocksDB and evaluated its performance on Intel’s latest PM device, the Intel Optane DC PMM with the state-of-the-art PM-based LSM-Tree KV stores, FlatLSM improves the throughput 5.2× on random write workload and 2.55× on YCSB-A. [ABSTRACT FROM AUTHOR]

Approximate nearest neighbor (ANN) search in high-dimensional spaces is fundamental in many applications. Locality-sensitive hashing (LSH) is a well-known methodology to solve the ANN problem. Existing LSH-based ANN solutions typically employ a large number of individual indexes optimized for searching efficiency. Updating such indexes might be impractical when processing high-dimensional streaming data. In this paper, we present a novel disk-based LSH index that offers efficient support for both searches and updates. The contributions of our work are threefold. First, we use the write-friendly LSM-trees to store the LSH projections to facilitate efficient updates. Second, we develop a novel estimation scheme to estimate the number of required LSH functions, with which the disk storage and access costs are effectively reduced. Third, we exploit both the collision number and the projection distance to improve the efficiency of candidate selection, improving the search performance with theoretical guarantees on the result quality. Experiments on four real-world datasets show that our proposal outperforms the state-of-the-art schemes. [ABSTRACT FROM AUTHOR]

Let there be a set of objects, where each object is characterized by Boolean features. Let logical query or Boolean query denote a query to find objects characterized by a Boolean function by features, for example, “documents with all words ...and any words ...and without words ...”. The terms “object” and “document” are used interchangeably hereinafter. The features may have different semantics, i. e. some features may correspond to the words of the document, some to labels or categories, and some to per-bit time quantum of a document’s date. Although indices executing Boolean queries are well researched in the literature, the common technique of maintaining posting lists is not always acceptable. If the data volume can reach to the order of petabytes, a compact index structure becomes vital. The aim of the research is to suggest a method to build an efficient bitmap index in secondary memory that allows us to update or append data being indexed with high write throughput. We propose an efficient LSM-based index for bitmaps and feature design for practical applications. We also discuss aspects of building of joined indices in order to achieve good scalability. The paper describes the architecture and algorithms of a suggested index and includes results of our experiments that show sustainable performance of our solution.

The log-structured merge tree (LSM-tree) has become an essential component in many key-value systems and expanded its scope to full-fledged database engines (e.g., MyRocks). In the database landscape, vendors face growing customer demands for real-time analytic solutions to handle hybrid transactional/analytical processing (HTAP) workloads that pose significant challenges. Among the challenges is IO amplification that drives system designers to rethink write-optimized engines to survive HTAP loads. This paper follows the same philosophy, reexamines LSM-trees used for database systems, and rethinks IO amplification under HTAP loads to shed some light on practical remedies for upcoming challenges. We propose two practical techniques to alleviate IO amplification: 1) aligned compaction for reducing write amplification, 2) snapshot filters for reducing read amplification. The two techniques are lightweight (i.e., near-zero resource consumption) and are compatible with state-of-the-art methods. We integrated our techniques into RocksDB and demonstrated that the modified RocksDB exhibits reduced IO amplification under HTAP workloads with negligible resource consumption.

LSM-tree is widely used in key-value stores for big data storage, but it suffers from write amplification brought by frequent compaction operations. An effective solution for this problem is key-value separation, which decouples values from the LSM-tree and stores them in a separate value log. However, existing key-value separation schemes achieve poor range query performance, especially for small key-value pairs, because they focus on mitigating write amplification but neglect access characteristics of the SSD. In this article, we propose FenceKV, which aims to achieve better range query performance while maintaining reasonable update performance for update-intensive workloads. FenceKV employs a new partition method to map values to the storage space based on the key-range to achieve efficient update and range query. Moreover, it adopts a key-range garbage collection policy to mitigate the garbage collection overhead and maintain sequential access for range queries. We compare FenceKV with modern key-value stores with various workloads, and results show that FenceKV can improve the range query performance significantly, while maintaining reasonable update performance compared to the existing designs of key-value separation. [ABSTRACT FROM AUTHOR]

Host-managed shingled magnetic recording drives (HM-SMR) are advantageous in capacity to harness the explosive growth of data. For key-value (KV) stores based on log-structured merge trees (LSM-trees), the HM-SMR drive is an ideal solution owning to its capacity, predictable performance, and economical cost. However, building an LSM-tree-based KV store on HM-SMR drives presents severe challenges in maintaining the performance and space utilization efficiency due to the redundant cleaning processes for applications and storage devices (i.e., compaction and garbage collection). To eliminate the overhead of on-disk garbage collection (GC) and improve compaction efficiency, this article presents GearDB, a GC-free KV store tailored for HM-SMR drives. GearDB improves the write performance and space efficiency through three new techniques: a new on-disk data layout, compaction windows, and a novel gear compaction algorithm. We further augment the read performance of GearDB with a new SSTable layout and read ahead mechanism. We implement GearDB with LevelDB, and use zonefs to access a real HM-SMR drive. Our extensive experiments confirm that GearDB achieves both high performance and space efficiency, i.e., on average 1.7× and 1.5× better than LevelDB in random write and read, respectively, with up to 86.9% space efficiency. [ABSTRACT FROM AUTHOR]

LSM-tree based Key-value (KV) stores greatly fit the needs of write-intensive applications with its efficient data store and retrieval operations to datasets. To accommodate ever-growing datasets, shingled magnetic recording (SMR) drives have become a popular option to provide large storage capacity for KV stores at low cost. SMR drives achieve high storage density via overlapping tracks on the disk surface. However, the overlapped track layout induces the sequential-write constraint and prevents KV stores from storing and rearranging KV pairs efficiently. In this paper, we present KVSTL, a KV store aware Shingled Translation Layer (STL), to preserve the merits of existing KV stores, while exploiting the high storage density of SMR drives. KVSTL is proposed as an application support to hide the management complexity of SMR drives and facilitates the management of SMR drives via passing only the “level” and “invalidation” information of LSM-tree based KV stores onto SMR drives. The proposed KVSTL achieves its performance enhancement via managing key-value pairs with level awareness and enabling efficient storage space management with the invalidation information. The results show that KVSTL can reduce the written data amount for 69.45 percent on average and the latency for up to 62.72 percent when compared with SMR-based LevelDB. [ABSTRACT FROM AUTHOR]

Emerging Zoned Namespace (ZNS) is a new-style Solid State Drive (SSD) that manages data in a zoned manner, which can achieve higher performance by strictly obeying the sequential write mode in each zone and alleviating the redundant overhead of garbage collections. Unfortunately, flash memory usually has a serious problem with limited program/erase cycles. Meanwhile, inappropriate data placement strategy of storage systems can lead to imbalanced wear among zones, severely reducing the lifespan of ZNS SSDs. In this paper, we propose a Wear-Leveling-Aware Data Placement (WADP) to solve this problem with negligible performance cost. First, WADP employs a wear-aware empty zone allocation algorithm to quantify the resets of zones and choose the less-worn zone for each allocation. Second, to prevent long-term zone occupation of infrequently written data (namely cold data), we propose a wear-leveling cold zone monitoring mechanism to identify cold zones dynamically. Finally, WADP adopts a real-time I/O pressure-aware data migration mechanism to adaptively migrate cold data for achieving wear-leveling among zones. We implement the proposed WADP in ZenFS and evaluate it with widely used workloads. Compared with state-of-the-art solutions, i.e., LIZA and FAR, the experimental results show that WADP can significantly reduce the standard deviation of zone resets while maintaining decent performance. [ABSTRACT FROM AUTHOR]

Log-structured merge (LSM) tree key-value (KV) stores have been widely deployed in many NoSQL and SQL systems, serving online big data applications such as social networking, graph processing, machine learning, etc. The batch processing of sorted data merging (i.e., compaction) in LSM-tree key-value stores improves the write efficiency, and some lazy compaction methods have been proposed to accumulate more data within a batch. However, these batched writing methods lead to significant tail latency, which is unacceptable for online processing. Aiming to optimize both latency and throughput, we propose a novel Lower-level Driven Compaction (LDC) method which breaks the limitations of the traditional upper-level driven compaction manner and triggers practical compaction actions bottom-up, with the benefits of both decreasing the compaction granularity for smaller latency and reducing write amplification for higher throughput. Furthermore, we extend LDC to Adaptive LDC (ALDC) by adding an adaptive policy to adjust the key compaction threshold to fit the changes of workloads’ features. The experimental results indicate that ALDC reduces the tail latency significantly and meanwhile achieves a much higher and stable throughput compared with existing approaches. [ABSTRACT FROM AUTHOR]

An LSM-tree (log-structured merge-tree) is a hierarchical, orderly and disk-oriented data storage structure which makes full use of the characteristics of disk sequential writing, which are much better than those of random writing. However, an LSM-tree can only be queried by a key and cannot meet the needs of a spatial query. To improve the query efficiency of spatial data stored in LSM-trees, the traditional method is to introduce stand-alone tree-like secondary indexes, the problem with which is the read amplification brought about by dual index queries. Moreover, when more spatial data are stored, the index tree becomes increasingly large, bringing the problems of a lower query efficiency and a higher index update cost. To address the above problems, this paper proposes an ER-tree(embedded R-tree) index structure based on the orderliness of LSM-tree data. By building an SER-tree(embedded R-tree on an SSTable) index structure for each storage component, we optimised dual index queries into single and organised SER-tree indexes into an ER-tree index with a binary linked list. The experiments showed that the query performance of the ER-tree index was effectively improved compared to that of stand-alone R-tree indexes. [ABSTRACT FROM AUTHOR]

Persistent key-value stores have been widely adopted as storage engines for modern IT infrastructures because they provide high performance with simple design principles. Moreover, many key-value stores commonly employ LSM-tree as their index structure due to its attractive features such as high write throughput and storage space efficiency. Unfortunately, LSM-tree has critical drawbacks in that it leads to write/read amplification problem. One of the prevalent solutions for remedying the write amplification problem is the tiering merge policy that reduces the number of rewrites by delaying merge operations. However, in spite of this advantage, the tiering merge policy may lead to a side-effect that induces high read amplification, increasing search/scan cost for upcoming read operations. In this paper, we concentrate on mitigating the high read amplification problem of the tiering merge policy, while maintaining its low write amplification. To achieve this, we propose a novel LSM-tree scheme, called Spatially Fragmented LSM-tree (SFM), which delays merge operations only for the non-read-intensive key-spaces. For this, SFM identifies the read intensity of each key-spaces by dynamically estimating their read/write hotness. We have implemented SFM based on PebblesDB and evaluated the performance benefits of our scheme under real-world workloads of Facebook. Experimental results clearly show that our scheme improves throughput by up to $1.67\times $ compared with the conventional schemes while maintaining low write amplification, and also indicate that its latency is lowered by up to 41.41% on average by mitigating the read amplification problem of the existing schemes, by up to 43.68%.

Key-value stores based on a log-structured merge (LSM) tree have emerged in big data systems because of their scalability and reliability. An LSM-tree offers a multilevel data structure with a simple interface. However, it performs file rewrites at the disk level, which causes write amplification. This study is concerned with this problem in relation to an embedded board environment, which can be used in edge computing. Addressing the major problems associated with an LSM-tree, we propose a new key-value store named CaseDB, which aggressively separates keys and bloom filters on the non-volatile memory express (NVMe) drive and stores the values on the SSD. Our solution reduces the I/O cost and enhances the overall performance in a cost-efficient manner. CaseDB employs a memory component, CBuffer, to avoid small write operations, and a delayed value compaction technique that guarantees the sorted order for both keys and values. CaseDB also utilizes deduction-based data deduplication to prevent space amplification in the values layer. The experiments show that CaseDB outperforms LevelDB and WiscKey 5.7 and 1.8 times, respectively, with respect to data writes, and additionally improves the read performance by 1.5 times. CaseDB also avoids the space amplification of WiscKey.

The log structured merge (LSM) tree has been widely adopted by database systems owing to its superior write performance. However, LSM-tree based databases face vulnerabilities when processing analytical queries due to the read amplification caused by its architecture and the limited use of storage devices with high bandwidth. To flexibly handle transactional and analytical workloads, we proposed and implemented OurRocks taking full advantage of NVMe SSD and GPU devices, which improves scan performance. Although the NVMe SSD serves multi GB/s I/O rates, it is necessary to solve the data transfer overhead which limits the benefits of the GPU processing. The primary idea is to offload the scan operation to the GPU with filtering predicate pushdown and resolve the bottleneck from the data transfer between devices with direct memory access (DMA). OurRocks benefits from all the features of write-optimized database systems, in addition to accelerating the analytic queries using the aforementioned idea. Experimental results indicate that OurRocks effectively leverages resources of the NVMe SSD and GPU and significantly improves the execution of queries in the YCSB and TPC-H benchmarks, compared to the conventional write-optimized database. Our research demonstrates that the proposed approach can speed up the handling of the data-intensive workloads. [ABSTRACT FROM AUTHOR]

Modern database systems desperate for the ability to support highly scalable transactions and efficient queries simultaneously for real-time applications. One solution is to utilize query optimization techniques on the on-line transaction processing (OLTP) systems. The materialized view is considered as a panacea to decrease query latency. However, it also involves the significant cost of maintenance which trades away transaction performance. In this paper, we examine the design space and conclude several design features for the implementation of a view on a distributed log-structured merge-tree (LSM-tree), which is a well-known structure for improving data write performance. As a result, we develop two incremental view maintenance (IVM) approaches on LSM-tree. One avoids join computation in view maintenance transactions. Another with two optimizations is proposed to decouple the view maintenance with the transaction process. Under the asynchronous update, we also provide consistency queries for views. Experiments on TPC-H benchmark show our methods achieve better performance than straightforward methods on different workloads. [ABSTRACT FROM AUTHOR]

Persistent key-value stores have emerged as a main component in the data access path of modern data processing systems. However, they exhibit high CPU and I/O overhead. Nowadays, due to power limitations, it is important to reduce CPU overheads for data processing. In this article, we propose Kreon, a key-value store that targets servers with flash-based storage, where CPU overhead and I/O amplification are more significant bottlenecks compared to I/O randomness. We first observe that two significant sources of overhead in key-value stores are: (a) The use of compaction in Log-Structured Merge-Trees (LSM-Tree) that constantly perform merging and sorting of large data segments and (b) the use of an I/O cache to access devices, which incurs overhead even for data that reside in memory. To avoid these, Kreon performs data movement from level to level by using partial reorganization instead of full data reorganization via the use of a full index per-level. Kreon uses memory-mapped I/O via a custom kernel path to avoid a user-space cache. For a large dataset, Kreon reduces CPU cycles/op by up to 5.8×, reduces I/O amplification for inserts by up to 4.61×, and increases insert ops/s by up to 5.3×, compared to RocksDB. [ABSTRACT FROM AUTHOR]

An LSM-tree (log-structured merge-tree) is a hierarchical, orderly and disk-oriented data storage structure which makes full use of the characteristics of disk sequential writing, which are much better than those of random writing. However, an LSM-tree can only be queried by a key and cannot meet the needs of a spatial query. To improve the query efficiency of spatial data stored in LSM-trees, the traditional method is to introduce stand-alone tree-like secondary indexes, the problem with which is the read amplification brought about by dual index queries. Moreover, when more spatial data are stored, the index tree becomes increasingly large, bringing the problems of a lower query efficiency and a higher index update cost. To address the above problems, this paper proposes an ER-tree(embedded R-tree) index structure based on the orderliness of LSM-tree data. By building an SER-tree(embedded R-tree on an SSTable) index structure for each storage component, we optimised dual index queries into single and organised SER-tree indexes into an ER-tree index with a binary linked list. The experiments showed that the query performance of the ER-tree index was effectively improved compared to that of stand-alone R-tree indexes.

Log-structured merge tree (LSM-tree)-based key-value stores are widely deployed in largescale storage systems. The underlying reason is that the traditional relational databases cannot reach the high performance required by big-data applications. As high-throughput alternatives to relational databases, LSM-tree-based key-value stores can support high-throughput write operations and provide high sequential bandwidth in storage systems. However, the compaction process triggers write amplification and is confronted with the degraded write performance, especially under update-intensive workloads. To address this issue, we design a holistic key-value store to explorer near-data processing (NDP) and on-demand scheduling for compaction optimization in an LSM-tree key-value store, named DStore. DStore makes full use of various computing capacities in the host-side and device-side subsystems. DStore dynamically divides the whole host-side compaction tasks into the above two-side subsystems according to two-side different computing capabilities. Meanwhile, the device must be featured with an NDP model. The divided compaction tasks are performed by the host and the device in parallel. In DStore, the NDP-based devices exhibit low-latency and high-bandwidth performance, thus facilitating key-value stores. DStore not only accomplishes compaction for key-value stores but also improves the system performance. We implement our DStore prototype in a real-world platform, and different kinds of testbeds are employed in our experiment. LevelDB and a static compaction optimization using the NDP model (called Co-KV) are used to compare with the DStore in our evaluation. Results show that DStore achieves about 3.7× performance improvement over LevelDB under the db_bench workload. In addition, DStore-enabled key-value stores outperform LevelDB by a factor of about 3.3× and 77% in terms of throughput and latency under YCSB benchmark, respectively.

Summary: LSM‐trie‐based key‐value (KV) store is often used to manage an ultralarge dataset in reality by introducing a number of sublevels at each level, its linear growth pattern can fairly reduce the write amplification in store operations. Although this design is effective for the write operation, the last level holds a large proportion of KV items, leading to the extreme imbalance of data distribution. Therefore, to support efficient read, we need to carefully consider this imbalance. On the other hand, to ensure that acquired data is latest, the LSM‐trie needs to search the dataset at different levels one by one, and this search method may take a lot of unnecessary time. When the number of items is ultralarge, the random lookup performance may be poor due to the imbalance data distribution. To address this issue, we improve the read performance of the LSM‐trie by changing its serial search to parallel search, using two threads to simultaneously search at the last level and other levels, respectively. Our experiment results show that the read performance of the LSM‐trie can be improved up to 98.35% and on average 71.55%. [ABSTRACT FROM AUTHOR]

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries: open-range queries, closed-range queries, and range counts. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the point and range query performance of state-of-the-art orderpreserving indexes, while consuming only 10 bits per trie node. The false-positive rates in SuRF for both point and range queries are tunable to satisfy different application needs. We evaluate SuRF in RocksDB as a replacement for its Bloom filters to reduce I/O by filtering requests before they access on-disk data structures. Our experiments on a 100-GB dataset show that replacing RocksDB's Bloom filters with SuRFs speeds up open-seek (without upper-bound) and closed-seek (with upper-bound) queries by up to 1.5× and 5× with a modest cost on the worst-case (all-missing) point query throughput due to slightly higher false-positive rate. [ABSTRACT FROM AUTHOR]

Recently, the log-structured merge-tree (LSM-tree) has been widely adopted for use in the storage layer of modern NoSQL systems. Because of this, there have been a large number of research efforts, from both the database community and the operating systems community, that try to improve various aspects of LSM-trees. In this paper, we provide a survey of recent research efforts on LSM-trees so that readers can learn the state of the art in LSM-based storage techniques. We provide a general taxonomy to classify the literature of LSM-trees, survey the efforts in detail, and discuss their strengths and trade-offs. We further survey several representative LSM-based open-source NoSQL systems and discuss some potential future research directions resulting from the survey. [ABSTRACT FROM AUTHOR]

Time-series DBMSs based on the LSM-tree have been widely applied in numerous scenarios ranging from daily life to industrial production. Compared to the traditional key–value data, the time-series data workload has significant features of writing and querying in chronological order. While simultaneously, such features bring new challenges to efficient queries, especially data compaction. Namely, an effective time-series data compaction algorithm is crucial for efficient storage and query of massive time-series data. However, the current data compaction method based on traditional LSM-tree cannot solve the problems such as high delay and inaccurate range of time sequence data query. Therefore, we propose a novel compaction algorithm, Time-Tiered Compaction , to customize for time-series scenarios. Time-Tiered Compaction leverages the characteristics of time-series workloads to estimate query loads to select optimum SSTables for merging during every compaction process, reducing unnecessary expenses. Time-Tiered Compaction is implemented on Apache IoTDB to evaluate algorithm performance using TPC-xIoT. Results show that the presented Time-Tiered Compaction significantly reduces (30%) the latency of the range queries with only a slight increase (5%) in point queries, compared with traditional strategies. [ABSTRACT FROM AUTHOR]

Key-value (KV) stores play an increasingly critical role in supporting diverse large-scale applications in modern data centers hosting terabytes of KV items which even might reside on a single server due to virtualization purposes. The combination of the ever-growing volume of KV items and storage/application consolidation is driving a trend of high storage density for KV stores. Shingled Magnetic Recording (SMR) represents a promising technology for increasing disk capacity, which however comes with the increased complexity of handling random writes. To take the best advantages of SMR drives, applications are expected to work in an SMR-friendly way. In this work, we present SEALDB, a Log-Structured Merge tree (LSM-tree) based key-value store that is specifically optimized for SMR drives via avoiding random writes and the corresponding write amplification on SMR drives. First, for LSM-trees, SEALDB collects and groups participating data of each compaction into sets. Using a set as the basic unit for compactions, SEALDB improves compaction efficiency by reducing random I/Os. Second, SEALDB creates variable sized bands on original HM-SMR drives, named dynamic bands. Dynamic bands store sets in an SMR-friendly way to eliminate the auxiliary write amplification from SMR drives. Third, SEALDB employs two light-weight garbage collection (GC) policies to further improve the space efficiency. We demonstrate the advantages of SEALDB via extensive experiments with various workloads. Overall, SEALDB delivers impressive performance compared with LevelDB, e.g., $3.42\times$3.42×/$2.65\times$2.65× faster for random writes (without or with GCs), and $3.96\times$3.96× faster for sequential reads. [ABSTRACT FROM AUTHOR]

SQLite has been deployed in millions of mobile devices from Web to smartphone applications on various mobile operating systems. However, due to the uncoordinated IO interactions with the underlying file system (e.g., ext4), SQLite is not efficient with low transactions per second. In this paper, we for the first time propose a new SQLite-like database engine, called SQLiteKV, which adopts the LSM-tree-based data structure but retains the SQLite operation interfaces. With its SQLite interface, SQLiteKV can be utilized by existing applications without any modification, while providing high performance with its LSM-tree-based data structure. We separate SQLiteKV into the front-end and back-end. In the front-end, we develop a light-weight SQLite-to-KV compiler to solve the semantic mismatch, so SQL statements can be efficiently translated into KV operations. We also design a novel coordination caching mechanism with memory defragmentation so query results can be effectively cached inside SQLiteKV by alleviating the discrepancy of data management between front-end SQLite statements and back-end data organization. In the back-end, we adopt an LSM-tree-based key-value database engine, and propose a lightweight metadata management scheme to mitigate the memory requirement. We have implemented and deployed SQLiteKV on a Google Nexus 6P smartphone. Experimental results with various workloads show that SQLiteKV outperforms SQLite up to six times. [ABSTRACT FROM AUTHOR]

Log-structured merge tree (LSM-tree) based key–value stores are widely employed in large-scale storage systems. In compaction, high-level sorted string table files (i.e. , SSTables) are merged with low-level overlapping key ranges and sorted for data queries. However, the compaction process incurs write amplification, which degrades system performance particularly under update-intensive workloads. Current optimizations mostly focus on reducing the overload of compaction in the host but rarely exploit parallelisms between the host and the device to fully utilize computing resources in the entire system for optimizing compaction performance. In this study, we propose Co-KV, a Co llaborative K ey- V alue store, to improve compaction performance in LSM-tree-based key–value stores. Co-KV is based on a near-data processing (i.e. , NDP) model-enabled storage device. Co-KV exhibits the following advantages: (1) it reduces write amplification and host-side CPU costs using a compaction offloading scheduling between a host computer and an NDP-enabled storage device; (2) it relieves the overload associated with data transfer between the host and the storage device; and (3) it improves the compaction of LSM-tree based key–value stores under update-intensive workloads. We employed an Open Ethernet Driver (OED), which is a real-world NDP platform as the testbed for our experiments. Extensive db_bench evaluations demonstrate that Co-KV achieves overall throughput improvements of approximately 1.75x, CPU cost reductions of approximately 68.1%, and write amplification reductions by up to 50.0% over the state-of-the-art LevelDB. Under YCSB workloads, Co-KV increases throughput by 1.7x ∼ 1.9x while decreasing the write amplification and average latency by up to 50.0% and 46.3%, respectively. • We propose an NDP model-based key-value store framework named as Co-KV which can make full use of the parallelism between the host and the device and leverage computation resources in the overall system. • We design and implement a compaction-SSTables-aware (i.e., CSA) scheduling in Co-KV. This scheduling offloads compaction tasks to the host and the device sides according to the SSTables-size region. This scheduling CSA yields an excellent improvement in the process of compaction. • We employ a real-world near-data processing-based platform and perform extensive experiments. With Co-KV in the platform, the experimental results validate that the overload of the host CPU and I/O resources is relieved and the overall performance is optimized in key-value stores. • Low write amplification. Write amplification induces extra data movement from the device to the host during compaction. It largely impacts overall performance; thus, it is essential to minimize write amplification for better performance of the key-value store. Co-KV reduces the burden of the CPU execution time and I/O resources by offloading a portion of compaction tasks into the NDP-enabled device to enhance the overall performance. • Optimized the host- and device-side computation. The compaction costs computation in the host, while the computation in devices is rarely exploited. Based on the NDP model, Co-KV executes compaction between the host and NDP-enabled device using the computation resources of both for high performance. [ABSTRACT FROM AUTHOR]

Key-value stores are increasingly adopting LSM-trees as their enabling data structure in the backend block storage, and persisting their clustered data through a block manager, usually a file system. In general, a file system is expected to not only provide file/directory abstraction to organize data but also retain the key benefits of LSM-trees, namely, sequential and aggregated I/O patterns on the physical device. Unfortunately, our in-depth experimental analysis reveals that some of these benefits of LSM-trees can be completely negated by the underlying file level indexes from the perspectives of both data layout and I/O processing. As a result, the write performance of LSM-trees is kept at a level far below that promised by the sequential bandwidth offered by the storage devices. In this paper, we address this problem and propose LDS, an LSM-tree Direct Storage system that manages the storage space based on the LSM-tree objects and provides simplified consistency control by leveraging the copy-on-write nature of the LSM-tree structure, to fully reap the benefits of LSM-trees. Running LevelDB, a popular LSM-tree based key-value store, on LDS as a baseline, comparing that to LevelDB running on three representative file systems (ext4, f2fs, btrfs) with HDDs and SSDs, respectively, we evaluate and study the performance potentials of LSM-trees. Evaluation results show that the write throughputs of LevelDB can be improved by from 1.8× to 3× on HDDs, and from 1.3× to 2.5× on SSDs, by employing the LSM-tree friendly data layout of LDS. [ABSTRACT FROM AUTHOR]

In this article, we show that key-value stores backed by a log-structured merge-tree (LSM-tree) exhibit an intrinsic tradeoff between lookup cost, update cost, and main memory footprint, yet all existing designs expose a suboptimal and difficult to tune tradeoff among these metrics. We pinpoint the problem to the fact that modern key-value stores suboptimally co-tune the merge policy, the buffer size, and the Bloom filters' false-positive rates across the LSM-tree's different levels. We present Monkey, an LSM-tree based key-value store that strikes the optimal balance between the costs of updates and lookups with any given main memory budget. The core insight is that worst-case lookup cost is proportional to the sum of the false-positive rates of the Bloom filters across all levels of the LSM-tree. Contrary to state-of-the-art key-value stores that assign a fixed number of bits-per-element to all Bloom filters, Monkey allocates memory to filters across different levels so as to minimize the sum of their false-positive rates. We show analytically that Monkey reduces the asymptotic complexity of the worst-case lookup I/O cost, and we verify empirically using an implementation on top of RocksDB that Monkey reduces lookup latency by an increasing margin as the data volume grows (50—80% for the data sizes we experimented with). Furthermore, we map the design space onto a closed-form model that enables adapting the merging frequency and memory allocation to strike the best tradeoff among lookup cost, update cost and main memory, depending on the workload (proportion of lookups and updates), the dataset (number and size of entries), and the underlying hardware (main memory available, disk vs. flash). We show how to use this model to answer what-if design questions about how changes in environmental parameters impact performance and how to adapt the design of the key-value store for optimal performance. [ABSTRACT FROM AUTHOR]

LSM-tree has been widely used in data management production systems for write-intensive workloads. However, as read and write workloads co-exist under LSM-tree, data accesses can experience long latency and low throughput due to the interferences to buffer caching from the compaction, a major and frequent operation in LSM-tree. After a compaction, the existing data blocks are reorganized and written to other locations on disks. As a result, the related data blocks that have been loaded in the buffer cache are invalidated since their referencing addresses are changed, causing serious performance degradations. To re-enable high-speed buffer caching during intensive writes,we propose Log-Structured buffered-Merge tree (simplified as LSbM-tree) by adding a compaction buffer on disks to minimize the cache invalidations on buffer cache caused by compactions. The compaction buffer efficiently and adaptively maintains the frequently visited datasets. In LSbM, strong locality objects can be effectively kept in the buffer cache with minimum or no harmful invalidations. With the help of a small on-disk compaction buffer, LSbM achieves a high query performance by enabling effective buffer caching, while retaining all the merits of LSM-tree for write-intensive data processing and providing high bandwidth of disks for range queries. We have implemented LSbM based on LevelDB. We show that with a standard buffer cache and a hard disk, LSbM can achieve 2x performance improvement over LevelDB. We have also compared LSbM with other existing solutions to show its strong cache effectiveness. [ABSTRACT FROM AUTHOR]

Key-value (KV) stores have become a backbone of large-scale applications in today's data centers. Write-optimized data structures like the Log-Structured Merge-tree (LSM-tree) and their variants are widely used in KV storage systems like BigTable and RocksDB. Conventional LSM-tree organizes KV items into multiple, successively larger components, and uses compaction to push KV items from one smaller component to another adjacent larger component until the KV items reach the largest component. Unfortunately, current compaction scheme incurs significant write amplification due to repeated KV item reads and writes, and then results in poor throughput. We propose a new compaction scheme, delayed compaction (dCompaction), that decreases write amplification. dCompaction postpones some compactions and gather them into the following compaction. In this way, it avoids KV item reads and writes during compaction, and consequently improves the throughput of LSM-tree based KV stores. We implement dCompaction on RocksDB, and conduct extensive experiments. Validation using YCSB framework shows that compared with RocksDB dCompaction has about 30% write performance improvements and also comparable read performance. [ABSTRACT FROM AUTHOR]