Search

Property-based testing has gained popularity in recent years in many areas of software development. The specification of assertions/properties helps to understand the semantics of pieces of code, and in modern programming environments, it can serve to test the program behavior. In this paper an XQuery property-based testing tool is presented, which enables to automatically test XQuery programs. The tool is able to systematically generate XML instances (i.e., test cases) from a given XML schema, and to filter XML instances with input properties specified by the programmer. Additionally, the tool automatically checks output (respectively, input-output) properties in each output instance (respectively, each pair of input-output instances). The tool is able to report whether the XQuery program passes the test, that is, if all the test cases satisfy the (input-)output property, as well as the number of test cases used for testing. In addition, if the XQuery program fails the test, the tool shows counterexamples found in the test cases. Properties are specified with XQuery Boolean functions, and the testing tool has been implemented in XQuery. Additionally, an XQuery path validation tool is presented. This tool is able to detect wrong paths in XQuery expressions. The path validation tool takes as input an XML schema, and it reports those paths on the XQuery program that do not match the XML schema. The path validation tool is a complement to the testing tool rejecting XQuery programs that do not conform to the XML schema. The path validation tool has been also implemented in XQuery. Finally, a web tool has been developed enabling to test and validate XQuery programs. [ABSTRACT FROM AUTHOR]

SUMMARY Automation of testing is an essential requirement to render it viable for software development. Although there are several testing techniques and criteria in many different domains, developing methods to test programs with complex outputs remains an unsolved challenge. This setting includes programs with graphical output, which produce images or interface windows. One possible approach towards automating the testing activity is the use of automatic oracles in which a reference image, taken as correct, can be used to establish a correctness measure in the tested program execution. A method that uses concepts of content-based image retrieval to facilitate oracle automation in the domain of programs with graphics output is presented. Two case studies, one using a computer-aided diagnostic system and one using a Web application, are presented, including some reflections and discussions that demonstrate the feasibility of the proposed approach. Copyright © 2011 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Mutation testing is a technique for testing software units that has great potential for improving the quality of testing, and thereby increasing the ability to assure the high reliability of critical software. It will be shown that recent advances in mutation research have brought a practical mutation testing system closer to reality. One recent advance is a partial solution to the problem of automatically detecting equivalent mutant programs. Equivalent mutants are currently detected by hand, which makes it very expensive and time-consuming. The problem of detecting equivalent mutants is a specific instance of a more general problem, commonly called the feasible path problem, which says that for certain structural testing criteria some of the test requirements are infeasible in the sense that the semantics of the program imply that no test case satisfies the test requirements. Equivalent mutants, unreachable statements in path testing techniques, and infeasible DU-pairs in data flow testing are all instances of the feasible path problem. This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to detect some equivalent mutants and infeasible paths automatically. © 1997 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

Fault-based testing attempts to show that particular faults cannot exist in software by using test sets that differentiate between the original program (hypothesized to be correct) and faulty alternate programs. The success of this approach depends on a number of assumptions, notably that programmers are competent insofar as they only commit relatively trivial faults, and that faults only couple infrequently. Fault coupling occurs when test sets are able to differentiate between the original program and faulty alternate programs when faults occur in isolation, but not when they occur in combination; it is a complicating factor in fault-based testing. Fault coupling is studied here within the context of finite bijective functions. A complete mathematical solution of the problem is possible in this simplified case; the results indicate that fault coupling does indeed occur infrequently, and are thus in agreement with the empirical results obtained by others in the field. One surprising result is that certain kinds of test set are able to avoid fault coupling altogether. [ABSTRACT FROM AUTHOR]

A commonly used approach for detecting defects in a program is to generate a suite of test inputs which exercises the entire program under a given testing criterion. Current techniques to generate test cases automatically that satisfy a testing criterion choose test input data by considering the testing requirements in an arbitrary order. In this paper, a technique is presented that groups the requirements and orders the generation of the test cases to reduce the number of test cases generated. The grouping is based on the statically determined property of post-dominance. Test cases for each group are generated taking into account requirements for other groups. The use of this technique to generate test cases can be expected to produce fewer test cases that have to be retained with the software. A smaller number of test cases will require less effort and resources to test the software. The technique is especially useful in the maintenance environment, whether retesting of a changed program is done exhaustively or incrementally. The technique not only attempts to minimize the number of test cases generated but also reduces the effort to generate the test cases using the concept of a program slice. [ABSTRACT FROM AUTHOR]

Systematic testing techniques based on the structure of code, i.e. white-box methods, are well known. Some, such as those based on the control and data flow characteristics of the code, are frequently used as checks on the adequacy of the test data. These approaches to testing, while precise in their description, take no account of the specification against which the program is being tested, i.e. they are specification independent. Specification-based approaches, whilst generally preferred, tend to be imprecise because the specifications themselves are not formal. Whilst formal specification methods are not widely used at present, there is evidence of a growth in the attention being paid to them. Formal methods are frequently used to specify data abstractions, for example. Were software specifications to be formal, what kind of precise strategies, based on specification structure, might be useful? This question is explored in this paper. Specifically, specification-based analogues to control and data flow white-box methods are considered. A model-based approach to formal specification is adopted, and an illustration of the way in which these analogue methods can be used to specify test conditions for data abstraction modules is provided. Subsequently, the analogue testing methods are evaluated intuitively, theoretically, and empirically. The intuitive evaluation reveals that there is no guarantee that desirable test cases will be selected. The theoretical evaluation shows that the relative power of the different methods is similar to, but not identical with that of their white-box counterparts. The empirical evaluation provides preliminary data on the methods' abilities to expose defects actually observed in implementing data abstractions. [ABSTRACT FROM AUTHOR]

The aim of any software testing method is to maximize fault detection, whilst minimizing testing cost. A knowledge base of common coding faults can be used to guide the testing process. Information on faults already found, the purpose of the code, and even the experience of the programmer and tester should be used to make the test efficient and systematic. Using mutation analysis to search primarily for the statistically most likely faults can reduce testing time. Further savings can be made by reducing the number of test input applications and achieving the optimum rate of live mutants generated per mutation applied. The question, 'when should we stop testing?' is often posed; but here the question addressed is, 'where should we start testing?'. [ABSTRACT FROM AUTHOR]

The paper considers some of the reasons advanced by commercial and industrial institutions in order to avoid using analysis and test tools. These insights serve to explain why so few tools are currently in use in such organisations. There is clearly a challenge for the testing community to meet, and this paper is a faltering step towards formulating the goals. [ABSTRACT FROM AUTHOR]

It is argued that the goal in verification and validation is not correctness, but the detection of the occurrence of errors in the program construction process. Different methods, such as proofs and testing, should not be viewed as competing techniques, but as complementary methods used to detect different kinds of error. Proofs, for example, are appropriate for the detection of logical errors made during algorithm design, but are not appropriate for detecting detailed programming errors. Testing is useful for detecting other kinds of error. Mutation testing, for example, can be used to detect "typographical errors", such as the use of the wrong arithmetic operator. Functional static analysis can be used to detect decomposition errors in which a programmer working on one part of a system makes a false assumption about the properties of objects that are established in some other part of the system. A brief summary of different methods is given, along with a description of the kind of error classes for which they are appropriate. An integrated approach to verification and validation is suggested in which, error classes are characterized, and methods, identified as being particularly appropriate for such classes, are combined. [ABSTRACT FROM AUTHOR]

Summary: As the scale and complexity of software increases, the number of tests needed for effective validation becomes extremely large. Executing these large test suites is expensive, both in terms of time and energy. Cache misses are a significant contributing factor to execution time of software. We propose an approach that helps order test executions in a test suite in such a way that instruction cache misses are reduced. We also ensure that the approach scales to large test suite sizes. We conduct an empirical evaluation with 20 subject programs and test suites from the SIR repository, EEMBC suite, and LLVM Symbolizer, comparing execution times and cache misses with test orderings maximising instruction locality versus a traditional ordering maximising coverage and random permutations. We also assess overhead of algorithms in generating the orderings that optimise cache locality. Nature of programs and tests impact the performance gained with our approach. Performance gains were considerable for programs and test suites where the average number of different instructions executed between tests was high. We achieved an average execution speedup of 6.83% and a maximum execution speedup of 17% over subject programs with differing control flow between test executions. [ABSTRACT FROM AUTHOR]