Search

Software testing and debugging are standard practices of software quality assurance since they enable the identification and correction of failures. Benchmarks have been used in that context as a group of programs to support the comparison of different techniques according to pre-established parameters. However, the reasons that inspire researchers to propose novel benchmarks are not fully understood. This article reports the investigation, identification, classification, and externalization of the state of the art about the proposition of benchmarks on software testing and debugging domains. The study was carried out using systematic mapping procedures according to the guidelines widely followed by software engineering literature. The search identified 1674 studies, from which, 25 were selected for analysis. A list of benchmarks is provided and descriptively mapped according to their characteristics, motivations, and scope of use for their creation. The lack of data to support the comparison between available and novel software testing and debugging techniques is the main motivation for the proposition of benchmarks. Advancements in the standardization and prescription of benchmark structure and composition are still required. Establishing such a standard could foster benchmark reuse, thereby saving time and effort in the engineering of benchmarks for software testing and debugging. [ABSTRACT FROM AUTHOR]

Spectrum-based fault localization (SBFL) techniques can automatically localize software faults. They employ the program spectrum, such as code coverage profile with test verdicts, to rank the program entities based on their code suspiciousness. In the past decades, researchers have proposed many approaches to optimize these techniques; however, the program structure, which can influence their performance, is not taken into consideration in developing and improving these techniques. In this article, we identify and analyze the effect of the program structure on the application of SBFL techniques. We observe that some specific program structures may introduce structure bias to code suspiciousness and negatively influence the output of SBFL techniques. To mitigate these effects and improve the performance of fault localization, we propose Delta4Ts, a structure-aware technique. Delta4Ts references debugging history to alleviate the impact of structure bias in the calculation of code suspiciousness. It reasons from the observable suspicious value towards the desired suspicious value and the impact of structure bias. To evaluate Delta4Ts under practical constraints, we conduct a controlled experiment using nine widely-studied SBFL formulae on 12 C programs and 6 Java programs. The experiment results show that Delta4Ts can significantly improve the accuracy of the studied SBFL formulae by an average of 34.8% on 12 C programs and 30.6% on 6 Java programs, and improve more on subject programs associated with more history versions or having larger code sizes. [ABSTRACT FROM AUTHOR]

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different state-of-the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method. [ABSTRACT FROM AUTHOR]

Abstract: The problem of testing and debugging learning neural network systems is discussed. Differences of these systems from program implementations of algorithms from the point of view of testing are noted. Requirements to the testing systems are identified. Specific features of various neural network models from the point of view of selection of the testing technique and determination of tested parameters are analyzed. It is discussed how to get rid of the noted drawbacks of the systems under study. The discussion is illustrated by an example. [ABSTRACT FROM AUTHOR]

Demand for highly reliable software is increasing day by day which in turn has increased the pressure on the software firms to provide reliable software in no time. Ensuring high reliability of the software can only be done by prolonged testing which in result consumes more resources which is not feasible in the existing market situation. To overcome this, software firms are providing patches after software release so as to fix the remaining number of bugs and to give better product experience to users. An update/fix is a minor portion of software to repair the bugs. With such patches, organizations enhance the performance of the software. Delivering patches after release demands extra effort and resources which are costly and hence not economical for the firms. Also, early patch release might cause improper fixation of bugs, on the other hand, delayed release may increase the chances of more failure during the operational phase. Therefore, determining optimal patch release time is imperative. To overcome the above issues we have formulated a two-dimensional time and effort-based cost model to regulate the optimum release and patch time of software, so that the total cost is minimized. Proposed model is validated on a real life data set. [ABSTRACT FROM AUTHOR]

The Linux kernel does not export a stable, well-defined kernel interface, complicating the development of kernel-level services, such as device drivers and file systems. While there does exist a set of functions that are exported to external modules, this set of functions frequently changes, and the functions have implicit, ill-documented preconditions. No specific debugging support is provided. We present Diagnosys, an approach to automatically constructing a debugging interface for the Linux kernel. First, a designated kernel maintainer uses Diagnosys to identify constraints on the use of the exported functions. Based on this information, developers of kernel services can then use Diagnosys to generate a debugging interface specialized to their code. When a service including this interface is tested, it records information about potential problems. This information is preserved following a kernel crash or hang. Our experiments show that the generated debugging interface provides useful log information and incurs a low performance penalty. [ABSTRACT FROM AUTHOR]

Test case prioritization is the process of ordering the test cases to be conducted eventually. Prioritizing test cases aids to meet two important constraints namely time and budget in software testing in order to enhance the fault detection rate as early as possible. This results us finding the problem earlier due to improved fault detection rate at early stage and deliver the system at short period of time .Interactions and relationships between a systems or sub system of an user is called as dependency. Few techniques that have been used in the past are namely Random order, greedy technique as well as fine grained and coarse grained. But all these techniques have considered the test cases in a test suite as an independent way. Few techniques have been derived dependency structures among test cases manually. Hence, this paper focuses to provide a set of algorithms to derive dependency structures among functions in a system and find exact number of dependents of each function in a system to prioritize the test cases in automated manner. High priority is assigned to test cases that are more dependent in using graph coverage values. By executing high priority (more complex) test cases earlier, the rate of fault detection will be improved at earlier stage and help to fix the fault earlier. For this experimentation purposes, Siemens test suite has been taken to establish the truth of verifying the given proposed technique. [ABSTRACT FROM AUTHOR]

Testing is a key activity of software development and maintenance that determines the level of reliability. Traceability is the ability to describe and follow the life of software artifacts, and has been promoted as a means for supporting various activities, most importantly testing. Traceability information facilitates the testing and debugging of complex software by modeling the dependencies between code and tests. Actively supplementing traceability to testing enables rectifying defects more reliably and efficiently. Despite its importance, the development of test-to-code traceability has not been sufficiently addressed in the literature, and even worse there is currently no organized review of traceability studies in this field. In this work, we have investigated the main conferences, workshops, and journals of the requirements engineering, testing, and reliability, and identified those contributions that refer to traceability topics. From that starting point, we characterized and analyzed the chosen contributions against three research questions by utilizing a comparative framework including nine criteria. As a result, our study arrives to some interesting points, and outlines a number of potential research directions. This, in turn, can pave the way for facilitating and empowering traceability research in this domain to assist software engineers and testers in test management. [ABSTRACT FROM AUTHOR]

This paper introduces a new metric to characterize test sets in terms of their diagnostic power. Our method uses much less space compared to the existing ones and is quite accurate. The metric can be utilized to increase the diagnosability of incompletely specified test sets via don't care filling. The X-filling approach can be integrated with test pattern generation tools to aid in better diagnostic pattern set generation. [ABSTRACT FROM PUBLISHER]

An important research area of Spectrum-Based Fault Localization (SBFL) is the effectiveness of risk evaluation formulas. Most previous studies have adopted an empirical approach, which can hardly be considered as sufficiently comprehensive because of the huge number of combinations of various factors in SBFL. Though some studies aimed at overcoming the limitations of the empirical approach, none of them has provided a completely satisfactory solution. Therefore, we provide a theoretical investigation on the effectiveness of risk evaluation formulas. We define two types of relations between formulas, namely, equivalent and better. To identify the relations between formulas, we develop an innovative framework for the theoretical investigation. Our framework is based on the concept that the determinant for the effectiveness of a formula is the number of statements with risk values higher than the risk value of the faulty statement. We group all program statements into three disjoint sets with risk values higher than, equal to, and lower than the risk value of the faulty statement, respectively. For different formulas, the sizes of their sets are compared using the notion of subset. We use this framework to identify the maximal formulas which should be the only formulas to be used in SBFL. [ABSTRACT FROM AUTHOR]

Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. Usually, compilation errors and behavioral changes are avoided by preconditions determined for each refactoring transformation. However, to formally define these preconditions and transfer them to program checks is a rather complex task. In practice, refactoring engine developers commonly implement refactorings in an ad hoc manner since no guidelines are available for evaluating the correctness of refactoring implementations. As a result, even mainstream refactoring engines contain critical bugs. We present a technique to test Java refactoring engines. It automates test input generation by using a Java program generator that exhaustively generates programs for a given scope of Java declarations. The refactoring under test is applied to each generated program. The technique uses SafeRefactor, a tool for detecting behavioral changes, as an oracle to evaluate the correctness of these transformations. Finally, the technique classifies the failing transformations by the kind of behavioral change or compilation error introduced by them. We have evaluated this technique by testing 29 refactorings in Eclipse JDT, NetBeans, and the JastAdd Refactoring Tools. We analyzed 153,444 transformations, and identified 57 bugs related to compilation errors, and 63 bugs related to behavioral changes. [ABSTRACT FROM AUTHOR]

In this paper, we propose two general frameworks for deriving several software reliability growth models based on a non-homogeneous Poisson process (NHPP) in the presence of imperfect debugging and error generation. The proposed models are initially formulated for the case when there is no differentiation between failure observation and fault removal testing processes, and then extended for the case when there is a clear differentiation between failure observation and fault removal testing processes. During the last three decades, many software reliability growth models (SRGM) have been developed to describe software failures as a random process, and can be used to evaluate development status during testing. With SRGM, software engineers can easily measure (or forecast) the software reliability (or quality), and plot software reliability growth charts. It is not easy to select the best model from a plethora of models available. There are few SRGM in the literature of software engineering that differentiates between failure observation and fault removal processes. In real software development environments, the number of failures observed need not be the same as the number of faults removed. Due to the complexity of software systems, and an incomplete understanding of software, the testing team may not be able to remove the fault perfectly on observation of a failure, and the original fault may remain, resulting in a phenomenon known as imperfect debugging, or get replaced by another fault causing error generation. In the case of imperfect debugging, the fault content of the software remains the same; while in the case of error generation, the fault content increases as the testing progresses. Removal of observed faults may result in the introduction of new faults. [ABSTRACT FROM AUTHOR]

Recent advances in mechanical techniques for systematic testing have increased our ability to automatically find subtle bugs, and hence, to deploy more dependable software. This paper builds on one such systematic technique, scope-bounded testing, to develop a novel specification-based approach for efficiently generating tests for products in a software product line. Given properties of features as first-order logic formulas in Alloy, our approach uses SAT-based analysis to automatically generate test inputs for each product in a product line. To ensure soundness of generation, we introduce an automatic technique for mapping a formula that specifies a feature into a transformation that defines incremental refinement of test suites. Our experimental results using different data structure product lines show that an incremental approach can provide an order of magnitude speedup over conventional techniques. We also present a further optimization using dedicated integer constraint solvers for feature properties that introduce integer constraints, and show how to use a combination of solvers in tandem for solving Alloy formulas. [ABSTRACT FROM AUTHOR]

The difficulty of developing wireless sensor systems is widely recognized. Problems associated with testing, debugging, and profiling are key contributing factors. While network simulators have proven useful, they are unable to capture the subtleties of underlying hardware, nor the dynamics of wireless signal propagation and interference; and physical experimentation remains a necessity. To this end, developers increasingly rely on shared deployments exposed for physical experimentation. Sensor network testbeds are under development across the world. We present a complementary testbed architecture that derives its novelty from three characteristics. First, the system is interactive; users can profile source and network level components across a network in real time, as well as inject transient state faults and external network traffic. Second, the system is source-centric; it enables automated source code analysis, instrumentation, and compilation. Finally, the design is open; developers can extend the set of exposed inter faces as appropriate to particular projects without modifying the underlying middleware. We present the testbed design and implementation, a graphical user interface, a shell-based macro programming interface, example scenarios that illustrate their use, and discuss the testbed's application in the research and teaching activities at client institutions. [ABSTRACT FROM AUTHOR]

With feature sizes steadily shrinking, manufacturing defects and parameter variations often cause design timing failures. It is essential that those errors be correctly and quickly diagnosed. In this paper, we analyze the multiple-delay-fault diagnosis problem and propose a novel approach to solve it. We enhance the diagnostic resolution by processing failure logs at various slower-than-nominal clock frequencies. We evaluate the utility of n-detection and timing-aware automatic-test-pattern-generated (ATPG) sets. Experimental results show that using timing-aware ATPG sets yields better diagnostic resolution and results in better delay-defect-size estimations compared to n-detection ATPG sets. We experimentally determined our diagnosis algorithm's sensitivity to delay variations. [ABSTRACT FROM AUTHOR]

The article presents information on an computer processor, Bugnet which helps the software developers to deterministically replay and debug an application while executing an computer program. The article author says that the difficult part of debugging is reproducing the bugs at a customer's site because it relys on core dumps which contain the final state of the system before the crash. The article authors' say that they propose the BugNet architecture to assist program developers to continuously record a program's execution as it runs. The Deterministic Replay Debugging technique helps to replay the exact same sequence of instructions that led up to the crash. The processor Bugnet continuously records a program's execution without interfering with the execution of the monitored program. The BugNet writes the logs to a fixed-sized buffer allocated in main memory so that software developers can then use these logs at the time of crash to deterministically replay the last part of execution as many times as required to fix the bug.

This article discusses the drawbacks of using a single test to measure the reliability of computer programs and quality of computer software. A one-test policy contradicts common wisdom about testing and mathematical analyses of testing. A one-test policy has a legitimate use in two situations, namely at the early stages of debugging when large faults are predicted, and when a single test can be comprehensive and thorough. It is possible that this policy is a software development department's reaction to the uncertainty associated with figuring out how many tests is enough.

In this article we introduce the problem of computer software reliability and discuss a probabilistic model for describing the failure of software. We suggest a procedure for estimating the parameters of the model and propose a stopping rule for debugging the software. We apply our procedures to some published data on software failures. [ABSTRACT FROM AUTHOR]

When selecting the set of conditions to be tested in modular white-box testing of fundamental program functions, various kinds of criteria are available, including the branch coverage criterion and the all-uses criterion. But there has been very little research on the relationship of these criteria to the overlooking of program faults, and as a consequence many program faults have been discovered only in the later phases of development or at customer sites, where conditions not covered by the original testing occurred. In this paper, an attempt is made to identify the relationship between test conditions and program fault overlooks by expressing the data flows of the program as chains of define-use relationships between two variables. The approach is tested on commercial software products for the control of measuring equipment. The results are used to develop a method of choosing test paths so as to prevent the principal overlook types, and the detection ability and computational efficiency are compared with those of conventional selection criteria. The number of test required in the new method is comparable with that in the all-uses method, and it is capable of preventing almost all of the overlooks that arise from the method used for test path selection. [ABSTRACT FROM AUTHOR]

A commonly used approach for detecting defects in a program is to generate a suite of test inputs which exercises the entire program under a given testing criterion. Current techniques to generate test cases automatically that satisfy a testing criterion choose test input data by considering the testing requirements in an arbitrary order. In this paper, a technique is presented that groups the requirements and orders the generation of the test cases to reduce the number of test cases generated. The grouping is based on the statically determined property of post-dominance. Test cases for each group are generated taking into account requirements for other groups. The use of this technique to generate test cases can be expected to produce fewer test cases that have to be retained with the software. A smaller number of test cases will require less effort and resources to test the software. The technique is especially useful in the maintenance environment, whether retesting of a changed program is done exhaustively or incrementally. The technique not only attempts to minimize the number of test cases generated but also reduces the effort to generate the test cases using the concept of a program slice. [ABSTRACT FROM AUTHOR]

It is argued that the goal in verification and validation is not correctness, but the detection of the occurrence of errors in the program construction process. Different methods, such as proofs and testing, should not be viewed as competing techniques, but as complementary methods used to detect different kinds of error. Proofs, for example, are appropriate for the detection of logical errors made during algorithm design, but are not appropriate for detecting detailed programming errors. Testing is useful for detecting other kinds of error. Mutation testing, for example, can be used to detect "typographical errors", such as the use of the wrong arithmetic operator. Functional static analysis can be used to detect decomposition errors in which a programmer working on one part of a system makes a false assumption about the properties of objects that are established in some other part of the system. A brief summary of different methods is given, along with a description of the kind of error classes for which they are appropriate. An integrated approach to verification and validation is suggested in which, error classes are characterized, and methods, identified as being particularly appropriate for such classes, are combined. [ABSTRACT FROM AUTHOR]

Phillip G. Armour responds to "Differential Debugging" in the Tools of the Trade column September/October issue of IEEE Software to discuss the process of predicting defects. [ABSTRACT FROM PUBLISHER]

Complex product assortments offered by online stores and electronic marketplaces make the identification of appropriate solutions a challenging task. Customers can differ greatly in their level of product domain knowledge. Therefore, recommender applications that support the product selection process are required. Knowledge-based recommender technologies allow flexible mapping of product, marketing and sales knowledge to the formal representation of a knowledge base and thus support customer-oriented sales dialogues in online selling environments. Although these technologies are frequently applied in industrial settings, related development processes are still very time-consuming. This paper presents the Koba4MS environment that supports graphical knowledge acquisition for web-based recommender applications and thus improves the effectiveness of related development and maintenance processes. The results of an experiment that investigated the effects of applying automated testing and debugging techniques to recommender knowledge base development clearly show the applicability and effectiveness of our approach in terms of time savings and reduced error rates. [ABSTRACT FROM AUTHOR]