Search

3D QLC NAND has recently entered the SSD market offering capacity increase and cost reduction compared to 3D TLC NAND. However, the endurance of QLC NAND is limited. Moreover, due to reduction of the available margin between the programmed threshold voltage distributions, QLC NAND is more susceptible to bit errors. Read voltage calibration is a key element of modern NAND flash memory controllers to improve the overall bit-error rate and maintain enterprise level reliability. To reduce the calibration overhead associated with the increased number of pages and read voltages in QLC NAND, page grouping is an effective approach. This paper presents open block characterization and read voltage calibration results of state-of-the-art 3D QLC NAND. We present experimental measurements of the bit-error characteristics and threshold voltage distributions based on closed and open block test patterns. We discuss the reliability issues with open blocks in preserving uniform characteristics within a page group at the boundary programmed layer and analyze the performance of different calibration algorithms.

The Kafka streaming platform has at its heart a distributed commit log. This resembles the change log that exists in every relational database system. It has been suggested that Kafka be viewed not just as a messaging system, but as the core of a database. The database is in effect “turned inside out” as the normally hidden change log becomes the first class entity of the system, while what is normally considered primary, i.e. the table, view, indexes etc. are just derived from this log. This is appealing as a vision, but raises challenges when applied within an actual enterprise system. The challenges arise from the conflicting interests and requirements of analytics and transactional systems. Running everything on a single system leads to tradeoffs; our intentions here is to identify some of the practical problems with using Kafka as a single data store within an enterprise and to describe our initial approach to resolving them. In particular we present preliminary approaches to ensure consistency and coherence of data from multiple database tables when distributed over Kafka and how to address compliance by encrypting/decrypting data at the Kafka producers and consumers.

We explain our experiences in designing, building and running a large corporate Datalake. Our platform has been running for over two years and makes a wide variety of corporate data assets, such as sales, marketing, customer information, as well as data from less conventional sources such as weather, news and social media available for analytics purposes to many teams across the company. We focus on describing the management of data and in particular how it is transferred and ingested into the platform.

This paper discusses the reliability challenges of 3D NAND flash memory and their impact on flash management for enterprise storage applications. Emphasis is given to the read voltage calibration and its critical role in achieving low error-rates and low latency read performance, as well as in enabling accurate block health estimation. We present experimental results that demonstrate the improvements in endurance, retention and read-disturb from different read voltage calibration schemes, and we address their requirements from a system perspective, i.e., the accuracy vs. complexity trade-off. We discuss the above aspects for state-of-the-art 3D TLC and QLC NAND flash memory.

Over the last few years, NAND flash manufacturers have steadily increased the number of bits stored per cell to achieve significant cost reductions. However, the increased density does not come without drawbacks. All key flash performance metrics, including latency and endurance, significantly degrade as bit density increases. Particularly, sustained write throughput is the worst affected as writes are roughly one order of magnitude slower than reads and further require precursory block erases in the background. As a result, many recent flash controllers operate flash blocks both in single-bit (high endurance and performance) and in multi-bit (high density) mode. In theory, such hybrid controllers are a great way of hiding flash technology limitations. A controller can use a small percentage of the flash blocks in single-bit mode as a cache which allows orders of magnitude higher write bandwidth and endurance in environments where the access patterns of the workload are skewed and bursty. In practice, however, many devices fall short of expectations when write performance varies significantly and utilization increases. We argue that a principled approach is required to understand the design trade-offs of hybrid NAND flash controllers. To this end, we develop a modeling framework for estimating the performance and endurance of hybrid controllers. The modeling framework computes the internal data movement generated by a hybrid controller by relying on advanced analytical models that offer both accurate and fast predictions. The data flow is then translated into higher-level metrics that quantify upper bounds for the overall performance of an SSD such as write throughput, latency, and device endurance. Using our modeling framework, we compare different controller architectures, identify their strong and weak points, and show that there is room to improve the efficiency of the hybrid controllers used today.

3D NAND flash memory has entered dynamically into the space of enterprise server and storage systems, offering significantly higher capacity and better endurance than the latest 2D technology node. Moreover, the advancements in vertical stacking, cell design and program/read algorithms, have also enabled TLC 3D NAND flash with enterprise-level reliability, thus achieving further increase in capacity and cost-per-bit reduction. This paper presents an in-depth analysis of the bit-error characteristics of state-of-the-art 64-layer 3D TLC NAND flash with a focus on read-voltage calibration. We provide experimental measurements of the RBER and threshold voltage distributions using typical and mixed-mode test patterns of program/erase cycling, retention and read-disturb. Moreover, we quantify the RBER components attributed to threshold voltage level overlapping and on-chip 2-step program errors. Finally, we characterize how the optimal read voltages change under different device stress and we evaluate calibration schemes with different performance and complexity trade-offs.

In recent years, several approaches to hydrologic frequency analysis have been proposed that enable one to direct attention to that portion of an overall probability distribution that is of greatest interest. The majority of the studies have focused on the upper tail of a distribution for flood analyses, though the same ideas can be applied to low flows. This paper presents an evaluation of the performances of five different estimation methods that place an emphasis on fitting the lower tail of the lognormal distribution for estimation of the ten-year low-flow quantile. The methods compared include distributional truncation, MLE treatment of censored data, partial probability weighted moments, LL-moments, and expected moments. It is concluded that while there are some differences among the alternative methods in terms of their biases and root mean square errors, no one method consistently performs better than the others, particularly with recognition that the underlying population distribution is unknown. Therefore, it seems perfectly legitimate to make a selection of a method on the basis other criteria, such as ease of use. It is also shown in this paper that the five alternative methods can perform about as well as, if not better than, an estimation strategy involving fitting the complete lognormal distribution using L-moments.

A precipitation time series is often a necessary input for the analysis and design of hydrologic and hydraulic systems. The precipitation records employed for these purposes can be either measured observa- tions or generated by stochastic simulation. One common problem with recorded and generated precipitation data is that often it is not in small enough time increments for use in engineering applications (e.g., continuous hydrologic simulation). To solve this problem, rainfall amounts can be disaggregated into shorter time increments. This paper evaluates the use of artificial neural networks (ANNs) for the disaggregation of hourly rainfall data into subhourly time increments. Two different ANN models are introduced and evaluated in this paper. A back- propagation/steepest-descent algorithm trains one model and the other model uses the idea of self-organization in a competitive learning ANN. The results indicate that the performance of both ANN models are: (1) com- parable to other disaggregation schemes in terms of predicting the overall disaggregated rainfall hyetograph; and (2) improvements over other disaggregation models in the prediction of the maximum incremental rainfall intensity (depth) within a storm hyetograph. Based on these results, the use of ANN models can be recommended as a viable alternative for hourly rainfall disaggregation.

A real-time database system (RTDBS) must satisfy the timing constraints of its transactions as well as the temporal consistency constraints of its temporal data. The data is considered to be temporally consistent if it is absolutely and relatively consistent. A data object is absolutely consistent if its age is younger than an absolute threshold, and a set of data objects is considered to be relatively consistent if the dispersion of ages is smaller than a relative threshold. It is often diAcult to select appropriate values for the thresholds that satisfy both absolute and relative consistency, especially in the case of objects whose values are derived (derived objects). In this paper we propose a new measure of temporal consistency for derived objects that is based on a new definition for the age of the derived object. Our proposed age provides additional insight into the temporal consistency of derived data objects and is useful for real-time database design and management. ” 2000 Elsevier Science Inc. All rights reserved.

In addition to timing constraints, a real-time database has temporal consistency constraints for its temporal data. The temporal consistency constraints require the data to represent a state of the real-world that is up-to-date and also require data to represent past states of the real-world with values that are close in time. Factors, such as concurrency control, can cause transactions to miss their deadlines and data to become temporally inconsistent. Approximate query processing (AQP) has been proposed as a strategy to satisfy the timing constraints of real-time databases. Approximate answers are provided by AQP if it is not possible to produce an exact answer by a specified deadline. In this paper, we examine the temporal consistency of the data during traditional and AQP. Four metrics of temporal consistency are utilized to compare the age and dispersion of the data during traditional query processing (TQP) versus approximate query processing. Simulation results identify factors, such as the concurrency control algorithm, the number of transactions and the percentage of query transactions, that affect the temporal inconsistency of the data.

The unit hydrograph is a common tool in hydraulic design. Used correctly, it allows a design engineer to estimate a runoff hydrograph from a drainage basin given a rainfall event. The typical method for estimating a unit hydrograph for a gaged watershed is by deconvolution. However, distinct storms produce different unit hydrographs for a single watershed. Consequently, a design engineer usually develops a composite, or average, unit hydrograph based on several recorded storm events. Common methods for estimating this composite unit hydrograph include curve fitting, simple aggregation, and multistorm optimization techniques. This paper introduces a new method to perform aggregation of unit hydrographs. The method is an extension to the simple averaging technique, in which prior to averaging, the individual unit hydrograph time ordinates are normalized with respect to the average time to peak. The normalization method is compared to a simple averaging technique and two multistorm aggregation techniques at six rural watersheds in Alabama. The results indicate that on average the normalization method predicts runoff nearly as accurately as the multistorm techniques, and displays improvement for 60 percent of the storms tested when compared with the simple averaging technique.

Low-flow estimates, as determined by probabilistic modeling of observed data sequences, are commonly used to describe certain streamflow characteristics. Unfortunately, however, reliable low-flow estimates can be difficult to come by, particularly for gaging sites with short record lengths. The shortness of records leads to uncertainties not only in the selection of a distribution for modeling purposes but also in the estimates of the parameters of a chosen model. In flood frequency analysis, the common approach to mitigation of some of these problems is through the regionalization of frequency behavior. The same general approach is applied here to the case of low-flow estimation, with the general intent of not only improving low-flow estimates but also illustrating the gains that might be attained in so doing. Data used for this study is that which has been systematically observed at 128 streamflow gaging sites across the State of Alabama. Our conclusions are that the log Pearson Type 3 distribution is a suitable candidate for modeling of Alabama low-flows, and that the shape parameter of that distribution can be estimated on a regional basis. Low-flow estimates based on the regional estimator are compared with estimates based on the use of only at-site estimation techniques.

Architectural simulator platforms are particularly complex and error-prone programs that aim to simulate all hardware details of a given target architecture. The development of a stable cycle-accurate architectural simulator can easily take several man-years. Discovering and fixing all visible errors in the simulator often requires significant effort, much higher than for writing the simulator in the first place. In addition, there are no guarantees that all programming errors will be eliminated, no matter how much effort is put into it. This paper presents dynamic runtime testing, a methodology for rapid development and accurate error detection in architectural cycle-accurate simulators. In dynamic runtime testing, the simulator execution is dynamically compared with a simple and functionally equivalent emulator. A possible error is detected if any instruction produces different results in the simulator and the emulator. Dynamic testing can help the developers of architectural simulators to get a reliable and accurate verification of functional correctness. Based on our experience, dynamic testing reduced the simulator modification time from 12-18 person-months to 3-4 person-months, and it only modestly reduced the simulator performance (in our case under 20%).

The adoption of multi- and many-core architectures for mainstream computing undoubtedly brings profound changes in the way software is developed. In particular, the use of fine grained locking as the multi-core programmer’s coordination methodology is considered by more and more experts as a dead-end. The transactional memory (TM) programming paradigm is a strong contender to become the approach of choice for replacing locks and implementing atomic operations in concurrent programming. Combining sequences of concurrent operations into atomic transactions allows a great reduction in the complexity of both programming and verification, by making parts of the code appear to execute sequentially without the need to program using fine-grained locking. Transactions remove from the programmer the burden of figuring out the interaction among concurrent operations that happen to conflict when accessing the same locations in memory. The EU-funded FP7 VELOX project designs, implements and evaluates an integrated TM stack, spanning from programming language to the hardware support, and including runtime and libraries, compilers, and application environments. This paper presents an overview of the VELOX TM stack and its associated challenges and contributions.

Hardware-assisted watchpoint systems enhance the execution of numerous dynamic software techniques, such as memory protection, module isolation, deterministic execution, and data race detection. In this paper, we show that previous hardware proposals may introduce significant energy overheads, and propose WatchPoint Filtering (WPF), a novel filtering mechanism that eliminates unnecessary watchpoint checks. We evaluate WPF on two state-of-the-art proposals for hardware-assisted watchpoints using two common memory checkers. WPF eliminates 83% of the watchpoint checks (up to 99.7%) and reduces 57% of the dynamic energy overhead (up to 78%) on average, without introducing additional performance execution overhead.