Search

Software failures caused by data race bugs have always been major concerns in parallel and distributed systems, despite significant efforts spent in software testing. Due to their nondeterministic and hard-to-reproduce features, when evaluating systems’ operational reliability, a rather long period of experimental execution time is expected to be spent on observing failures caused by data race conditions. To address this problem, in this paper, we make two contributions. First, this paper proposes stress testing with influencing factors, in which the system runs under certain workloads for a long time with controlled stress conditions to accelerate the occurrence of data race failures. Second, it explores and formulates mathematical relationship models between data races’ statistical characteristics of time to failure (TTF) or mean TTF (MTTF) and the influencing factors. Such relationship models are used for TTF/MTTF extrapolation under different operational conditions and are essential to reduce systems’ reliability evaluation time. The proposed method is empirically evaluated on six applications suffering from failures caused by real-world data race bugs. Through analysis of the experimental results, we obtain several important findings: First, the reduction in the manifestation time to data race failures achieved by controlling the influencing factors is statistically significant. Second, Power model is the best-fitting model of the relationship between the MTTF and the influencing factors. Third, Power Weibull distribution is the best-fitting probability distribution between the TTF and the influencing factors. Finally, the TTF/MTTF can be accurately estimated with the approach proposed in this paper. [ABSTRACT FROM AUTHOR]

To ensure that an access control (AC) system is safe, there must be a reliable means to verify that the specified AC policy model conforms to the safety requirements and policy author's intentions. A general verification approach includes black-box and white-box testing, as well as the generation of sufficient test cases to check the correctness of model implementations. [ABSTRACT FROM AUTHOR]

The Linux kernel does not export a stable, well-defined kernel interface, complicating the development of kernel-level services, such as device drivers and file systems. While there does exist a set of functions that are exported to external modules, this set of functions frequently changes, and the functions have implicit, ill-documented preconditions. No specific debugging support is provided. We present Diagnosys, an approach to automatically constructing a debugging interface for the Linux kernel. First, a designated kernel maintainer uses Diagnosys to identify constraints on the use of the exported functions. Based on this information, developers of kernel services can then use Diagnosys to generate a debugging interface specialized to their code. When a service including this interface is tested, it records information about potential problems. This information is preserved following a kernel crash or hang. Our experiments show that the generated debugging interface provides useful log information and incurs a low performance penalty. [ABSTRACT FROM AUTHOR]

The growing dependence of society on software systems places a high premium on their reliable operation. Moreover, the stringent reliability expectations imposed on these systems must be achieved despite their increasing size and complexity, and decreasing resources available for their development and maintenance. To mitigate these dual challenges, a systematic approach to guide the allocation of resources to the components of a software system is necessary. This paper presents an optimization framework which considers the contribution of each component to system reliability to determine the amount of effort to be allocated to each component, towards the ultimate objective of achieving the specified system reliability target with minimal effort. We assume that the contribution of a component to system reliability is governed by two factors: the system architecture, and the effort-reliability relationship of the component. This characterization is referred to as “architecture-based optimization” because it considers the system architecture explicitly in the effort allocation process. It is demonstrated that the architecture-based optimization framework outperforms other effort allocation strategies, including equal component weighting, and expert opinion. [ABSTRACT FROM AUTHOR]

Abstract: Models are abstractions of reality that have predetermined limits (often not consciously thought through) on what problem domains the models can be used to explore. These limits are determined by the range of observed data used to construct and validate the model. However, it is important to remember that operating the model beyond these limits, one of the reasons for building the model in the first place, potentially brings unwanted behaviour and thus reduces the usefulness of the model. Our experience with the Agricultural Production Systems Simulator (APSIM), a farming systems model, has led us to adapt techniques from the disciplines of modelling and software development to create a model development process. This process is simple, easy to follow, and brings a much higher level of stability to the development effort, which then delivers a much more useful model. A major part of the process relies on having a range of detailed model tests (unit, simulation, sensibility, validation) that exercise a model at various levels (sub-model, model and simulation). To underline the usefulness of testing, we examine several case studies where simulated output can be compared with simple relationships. For example, output is compared with crop water use efficiency relationships gleaned from the literature to check that the model reproduces the expected function. Similarly, another case study attempts to reproduce generalised hydrological relationships found in the literature. This paper then describes a simple model development process (using version control, automated testing and differencing tools), that will enhance the reliability and usefulness of a model. [ABSTRACT FROM AUTHOR]

With the growing size and complexity of software applications, traditional software reliability methods are insufficient to analyze inter-component interactions of modular software systems. The number of test cases may be extremely large for this application; therefore, it is hard for us to extensively test each software component given resource limitations. In this paper, we propose an adaptive framework of incorporating path testing into reliability estimation for modular software systems. Three estimated methods based on common program structures, namely, sequence, branch, and loop structures, are proposed to calculate the path reliability. Consequently, the derived path reliabilities can be applied to the estimates of software reliability. Some experiments are performed based on two real systems. In addition, the accuracy and correlation with respect to the experiments are investigated by simulation and sensitivity analysis. Experimental results show that the path reliability has a high correlation to the actual software reliability. For software with loop structures, a smaller loop number can be assigned to derive an acceptable estimation of path reliability. Further, the sensitivity analysis can be used to identify critical modules and paths for resource allocation. It can be concluded that the proposed methods are useful and helpful for estimating software reliability and can be adaptively used in the early stages of software development. [ABSTRACT FROM AUTHOR]

In this paper, we propose two general frameworks for deriving several software reliability growth models based on a non-homogeneous Poisson process (NHPP) in the presence of imperfect debugging and error generation. The proposed models are initially formulated for the case when there is no differentiation between failure observation and fault removal testing processes, and then extended for the case when there is a clear differentiation between failure observation and fault removal testing processes. During the last three decades, many software reliability growth models (SRGM) have been developed to describe software failures as a random process, and can be used to evaluate development status during testing. With SRGM, software engineers can easily measure (or forecast) the software reliability (or quality), and plot software reliability growth charts. It is not easy to select the best model from a plethora of models available. There are few SRGM in the literature of software engineering that differentiates between failure observation and fault removal processes. In real software development environments, the number of failures observed need not be the same as the number of faults removed. Due to the complexity of software systems, and an incomplete understanding of software, the testing team may not be able to remove the fault perfectly on observation of a failure, and the original fault may remain, resulting in a phenomenon known as imperfect debugging, or get replaced by another fault causing error generation. In the case of imperfect debugging, the fault content of the software remains the same; while in the case of error generation, the fault content increases as the testing progresses. Removal of observed faults may result in the introduction of new faults. [ABSTRACT FROM AUTHOR]

Software testing is an important issue in software engineering. As software systems become increasingly large and complex, the problem of how to optimally allocate the limited testing resource during the testing phase has become more important, and difficult. Traditional Optimal Testing Resource Allocation Problems (OTRAPs) involve seeking an optimal allocation of a limited amount of testing resource to a number of activities with respect to some objectives (e.g., reliability, or cost). We suggest solving OTRAPs with Multi-Objective Evolutionary Algorithms (MOEAs). Specifically, we formulate OTRAPs as two types of multi-objective problems. First, we consider the reliability of the system and the testing cost as two objectives. Second, the total testing resource consumed is also taken into account as the third objective. The advantages of MOEAs over state-of-the-art single objective approaches to OTRAPs will be shown through empirical studies. Our study has revealed that a well-known MOEA, namely Nondominated Sorting Genetic Algorithm II (NSGA-II), performs well on the first problem formulation, but fails on the second one. Hence, a Harmonic Distance Based Multi-Objective Evolutionary Algorithm (HaD-MOEA) is proposed and evaluated in this paper. Comprehensive experimental studies on both parallel-series, and star-structure modular software systems have shown the superiority of HaD-MOEA over NSGA-II for OTRAPs. [ABSTRACT FROM AUTHOR]

This article discusses the drawbacks of using a single test to measure the reliability of computer programs and quality of computer software. A one-test policy contradicts common wisdom about testing and mathematical analyses of testing. A one-test policy has a legitimate use in two situations, namely at the early stages of debugging when large faults are predicted, and when a single test can be comprehensive and thorough. It is possible that this policy is a software development department's reaction to the uncertainty associated with figuring out how many tests is enough.

This paper presents a testing strategy designed to detect errors in the control flow of a computer program, and the conditions under which this strategy is reliable are given and characterized. The control flow statements in a computer program partition the input space into a set of mutually exclusive domains, each of which corresponds to a particular program path and consists of input data points which cause that path to be executed. The testing strategy generates test points to examine the boundaries of a domain to detect whether a domain error has occurred, as either one or more of these boundaries will have shifted or else the corresponding predicate relational operator has changed. If test points can be chosen within ∈ of each boundary, under the appropriate assumptions, the strategy is shown to be reliable in detecting domain errors of magnitude greater than ∈. Moreover, the number of test points required to test each domain grows only linearly with both the dimensionality of the input space and the number of predicates along the path being tested. [ABSTRACT FROM AUTHOR]

The theory of test data selection proposed by Goodenough and Gerhart is examined. In order to extend and refine this theory, the concepts of a revealing test criterion and a revealing subdomain are proposed. These notions are then used to provide a basis for constructing program tests. A subset of a program's input domain is revealing if the existence of one incorrectly processed input implies that all of the subset's elements axe processed incorrectly. The intent of this notion is to partition the program's domain in such a way that all elements of an equivalence class are either processed correctly or incorrectly. A test set is then formed by choosing one element from each class. This process represents perfect program testing. For a practical testing strategy, the domain is partitioned into subdomains which are revealing for errors considered likely to occur. Three programs which have previously appeared in the literature are discussed and tested using the notions developed in the paper. [ABSTRACT FROM AUTHOR]

Fault-based testing attempts to show that particular faults cannot exist in software by using test sets that differentiate between the original program (hypothesized to be correct) and faulty alternate programs. The success of this approach depends on a number of assumptions, notably that programmers are competent insofar as they only commit relatively trivial faults, and that faults only couple infrequently. Fault coupling occurs when test sets are able to differentiate between the original program and faulty alternate programs when faults occur in isolation, but not when they occur in combination; it is a complicating factor in fault-based testing. Fault coupling is studied here within the context of finite bijective functions. A complete mathematical solution of the problem is possible in this simplified case; the results indicate that fault coupling does indeed occur infrequently, and are thus in agreement with the empirical results obtained by others in the field. One surprising result is that certain kinds of test set are able to avoid fault coupling altogether. [ABSTRACT FROM AUTHOR]

A commonly used approach for detecting defects in a program is to generate a suite of test inputs which exercises the entire program under a given testing criterion. Current techniques to generate test cases automatically that satisfy a testing criterion choose test input data by considering the testing requirements in an arbitrary order. In this paper, a technique is presented that groups the requirements and orders the generation of the test cases to reduce the number of test cases generated. The grouping is based on the statically determined property of post-dominance. Test cases for each group are generated taking into account requirements for other groups. The use of this technique to generate test cases can be expected to produce fewer test cases that have to be retained with the software. A smaller number of test cases will require less effort and resources to test the software. The technique is especially useful in the maintenance environment, whether retesting of a changed program is done exhaustively or incrementally. The technique not only attempts to minimize the number of test cases generated but also reduces the effort to generate the test cases using the concept of a program slice. [ABSTRACT FROM AUTHOR]

The paper considers some of the reasons advanced by commercial and industrial institutions in order to avoid using analysis and test tools. These insights serve to explain why so few tools are currently in use in such organisations. There is clearly a challenge for the testing community to meet, and this paper is a faltering step towards formulating the goals. [ABSTRACT FROM AUTHOR]