Search

Service-Oriented Architecture (SOA) is a technology for designing software systems. By using this architecture, new challenges appeared for software testing. Functional testing of services assures the quality of service-oriented applications. Herein, we introduce a new test web service named TaaWS (Test as a Web Service), which is a combination of both online testing and testing as a service to overcome SOA testing challenges. Each web service can have its test web service. Thus, the consumer can call TaaWS to assure the accuracy of the web services used in the application. TaaWS consists of semantic and structural test cases extracted from the extended-attribute grammar of the web service called Method-Based Attribute Grammar (MBAG). MBAG is created in five steps, including the definition of the input/output variables, the creation of sections in the web method, and the application of rules to specify conditions and operations on the attributes. In addition, a new coverage criterion is introduced for application on MBAG to extract test cases. Using TaaWS, we have applied semantic and structural test cases on the PersonalIDCode-INFO web method as a case study. In our proposed method, a developed program called Sharif-TaaWS as a tool automates MBAG generation steps, test cases, and TaaWS creation. To evaluate TaaWS, other functional testing techniques have been compared to our method by measuring the effectiveness of the test cases (ETC value). This value shows that TaaWS can find more faults compared to other techniques. We propose that TaaWS has advantages such as creating both semantic and structural test cases through MBAG, detecting changes in web services, running regression tests, and managing test results to inform the consumer. However, TaaWS has a drawback; the response time to the user may increase through calling TaaWS. [ABSTRACT FROM AUTHOR]

Software testing is key for quality assurance of embedded systems. However, with increased development pace, the amount of test results data risks growing to a level where exploration and visualization of the results are unmanageable. This paper covers a tool, Tim, implemented at a company developing embedded systems, where software development occurs in parallel branches and nightly testing is partitioned over software branches, test systems and test cases. Tim aims to replace a previous solution with problems of scalability, requirements and technological flora. Tim was implemented with a reference group over several months. For validation, data were collected both from reference group meetings and logs from the usage of the tool. Data were analyzed quantitatively and qualitatively. The main contributions from the study include the implementation of eight views for test results exploration and visualization, the identification of four solutions patterns for these views (filtering, aggregation, previews and comparisons), as well as six challenges frequently discussed at reference group meetings (expectations, anomalies, navigation, integrations, hardware details and plots). Results are put in perspective with related work and future work is proposed, e.g., enhanced anomaly detection and integrations with more systems such as risk management, source code and requirements repositories. [ABSTRACT FROM AUTHOR]

Tool competitions are a special form of comparative evaluation, where each tool has a team of developers or supporters associated that makes sure the tool is properly configured to show its best possible performance. In several research areas, tool competitions have been a driving force for the development of mature tools that represent the state of the art in their field. This paper describes and reports the results of the 1 st International Competition on Software Testing (Test-Comp 2019), a comparative evaluation of automatic tools for software test generation. Test-Comp 2019 was presented as part of TOOLympics 2019, a satellite event of the conference TACAS. Nine test generators were evaluated on 2 356 test-generation tasks. There were two test specifications, one for generating a test that covers a particular function call and one for generating a test suite that tries to cover the branches of the program. [ABSTRACT FROM AUTHOR]

ESBMC is an SMT-based bounded model checker that provides a bit-precise verification of both C and C++ programs. Bounded model checking (BMC) was developed to provide faster results when finding property violations; BMC achieves this by limiting the number of loop unwindings and recursion depth. The technique, however, is unable to prove correctness unless all loops and recursions are fully unwound, which might not be possible for some programs (e.g., infinite loops). The version of ESBMC described here is designed to avoid the problem of guessing the number of unwindings, which leads to a property violation; it incrementally verifies the program, searching only for property violations. Once ESBMC has found a property violation, it produces a test suite that contains at least one test to expose a bug. ESBMC can correctly produce 312 test cases, which are confirmed by the test validator employed by Test-Comp 2019. [ABSTRACT FROM AUTHOR]

Verification techniques are well-suited for automatic test-case generation. They basically need to check the reachability of every test goal and generate test cases for all reachable goals. This is also the basic idea of our CoVeriTest submission. However, the set of test goals is not fixed in CoVeriTest , instead we can configure the set of test goals. For Test-Comp'19, we support the set of all __VERIFIER_error() calls as well as the set of all branches. Thus, we can deal with the two test specifications considered in Test-Comp'19. Since the tasks in Test-Comp are diverse and verification techniques have different strengths and weaknesses, we also do not stick to a single verification technique, but use a hybrid approach that combines multiple techniques. More concrete, CoVeriTest interleaves different verification techniques and allows to configure the cooperation (i.e., information exchange and time limits). To choose from a large set of verification techniques, CoVeriTest is integrated into the analysis framework CPAchecker. For the competition, we interleave CPAchecker's value and predicate analysis and let both analyses resume their analysis performed in the previous iteration. [ABSTRACT FROM AUTHOR]

Research competitions and challenges are a driving force in transferring theoretical results into working software tools that demonstrate the state of the art in the respective field of research. Regular comparative evaluations provide guidance to practitioners that have to select new technology and tools for their development process. In order to support competitions and challenges with an appropriate publication venue, a new theme of issues in the International Journal on Software Tools for Technology Transfer was created. This issue is the inaugural issue of the newly introduced theme on "Competitions and Challenges" (CoCha). Test-Comp, the International Competition on Software Testing, is an example of a tool competition, where the research teams submit tools for test-generation, and the competition evaluates the tools and assigns scores according to achieved coverage. Test-Comp 2019 was part of the TOOLympics event, which took place as part of the 25-year celebration of the conference TACAS. Thus, it is most natural to start the new STTT-CoCha theme with a special issue that describes the results and participating systems of Test-Comp 2019. There will be a second issue on TOOLympics with contributions from other competitions. [ABSTRACT FROM AUTHOR]

Testing is a widely applied technique to evaluate software quality, and coverage criteria are often used to assess the adequacy of a generated test suite. However, manually constructing an adequate test suite is typically too expensive, and numerous techniques for automatic test-suite generation were proposed. All of them come with different strengths. To build stronger test-generation tools, different techniques should be combined. In this paper, we study cooperative combinations of verification approaches for test generation, which exchange high-level information. We present CoVeriTest, a hybrid technique for test-suite generation. CoVeriTest iteratively applies different conditional model checkers and allows users to adjust the level of cooperation and to configure individual time limits for each conditional model checker. In our experiments, we systematically study different CoVeriTest cooperation setups, which either use combinations of explicit-state model checking and predicate abstraction, or bounded model checking and symbolic execution. A comparison with state-of-the-art test-generation tools reveals that CoVeriTest achieves higher coverage for many programs (about 15%). [ABSTRACT FROM AUTHOR]

Data structure properties are important for many testing and analysis tasks. For example, model checkers use these properties to find program faults. These properties are often written manually which can be error prone and lead to false alarms. This paper presents the results of controlled experiments performed using existing machine learning (ML) models on various data structures. These data structures are dynamic and reside on the program heap. We use ten data structure subjects and ten ML models to evaluate the learnability of data structure properties. The study reveals five key findings. One, most of the ML models perform well in learning data structure properties, but some of the ML models such as quadratic discriminant analysis and Gaussian naive Bayes are not suitable for learning data structure properties. Two, most of the ML models have high performance even when trained on just 1% of data samples. Three, certain data structure properties such as binary heap and red black tree are more learnable than others. Four, there are no significant differences between the learnability of varied-size (i.e., up to a certain size) and fixed-size data structures. Five, there can be significant differences in performance based on the encoding used. These findings show that using machine learning models to learn data structure properties is very promising. We believe that these properties, once learned, can be used to provide a run-time check to see whether a program state at a particular point satisfies the learned property. Learned properties can also be employed in the future to automate static and dynamic analysis, which would enhance software testing and verification techniques. [ABSTRACT FROM AUTHOR]

This is the second issue in the new "Competitions and Challenges" (CoCha) theme of the International Journal on Software Tools for Technology Transfer. The new theme was established to support competitions and challenges with an appropriate publication venue. The first issue presented the competition on software testing Test-Comp 2019, which was part of the TOOLympics 2019 event. In this second issue for TOOLympics, we present selected competition reports. The TOOLympics event took place as part of the 25-years celebration of the conference TACAS. The goal of the event was to provide an overview of competitions and challenges in the area of formal methods. [ABSTRACT FROM AUTHOR]

Automatic test-suite generation tools are often complex and their behavior is not predictable. To provide a minimum baseline that test-suite generators should be able to surpass, we present PRTest, a random black-box test-suite generator for C programs: To create a test, PRTest natively executes the program under test and creates a new, random test value whenever an input value is required. After execution, PRTest checks whether any new program branches were covered and, if this is the case, the created test is added to the test suite. This way, tests are rapidly created either until a crash is found, or until the user aborts the creation. While this naive mechanism is not competitive with more sophisticated, state-of-the-art test-suite generation tools, it is able to provide a good baseline for Test-Comp and a fast alternative for automatic test-suite generation for programs with simple control flow. PRTest is publicly available and open source. [ABSTRACT FROM AUTHOR]

The state-of-the-practice in software development is driven by constant change fueled by continues integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is "good enough", and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage and that it is able to do so with an acceptable performance overhead during project build. [ABSTRACT FROM AUTHOR]

Software testing continues to be regarded as a necessary and critical step in the software development life cycle. Among the multitudes of existing techniques, particle swarm optimization (PSO) algorithm, in particular, has shown superior merits for automatically generating software test cases for its easy implementation and for relying on fewer parameters that require tuning. Hence, several state-of-the-art PSO-based algorithms have been successfully used as a test data generator. On the other hand, greedy-based algorithms, which are commonly used to solve complex and multi-step combinatorial problems, are starting to gain momentum as a solution for the complexity problem of software testing. Greedy algorithms favored over other techniques when the solution of the problem is guaranteed to be near-optimal. As a result, the utilization of both greedy and PSO algorithms in a single solution for automatically generating test data represents a strong candidate if designed carefully. In this paper, we propose a novel hybrid greedy and PSO algorithm (GPSO) that jointly guarantees the effectiveness and close to optimality results for generating a minimum number of test data. Compared with the widely employed genetic algorithm (GA), our proposed GPSO outperforms the GA in terms of the average number of iterations, execution time, and coverage percentage. Experimental trials with six different typical Java card applications show that the use of the proposed GPSO as a test data generator results in an outstanding performance. [ABSTRACT FROM AUTHOR]

When custom modeling tools are used for designing complex safety-critical systems (e.g., critical cyber-physical systems), the tools themselves need to be validated by systematic testing to prevent tool-specific bugs reaching the system. Testing of such modeling tools relies upon an automatically generated set of models as a test suite. While many software testing practices recommend that this test suite should be diverse, model diversity has not been studied systematically for graph models. In the paper, we propose different diversity metrics for models by generalizing and exploiting neighborhood and predicate shapes as abstraction. We evaluate such shape-based diversity metrics using various distance functions in the context of mutation testing of graph constraints and access policies for two separate industrial DSLs. Furthermore, we evaluate the quality (i.e., bug detection capability) of different (random and consistent) model generation techniques for mutation testing purposes. [ABSTRACT FROM AUTHOR]

Web applications are the target of many well-known exploits and also a fertile ground for the discovery of security vulnerabilities. Yet, the success of an exploit depends both on the vulnerability in the application source code and the environment in which the application is deployed and run. As execution environments are complex (application servers, databases and other supporting applications), we need to have a reliable framework to test whether known exploits can be reproduced in different settings, better understand their effects, and facilitate the discovery of new vulnerabilities. In this paper, we present TESTREX—a framework that allows for highly automated, easily repeatable exploit testing in a variety of contexts, so that a security tester may quickly and efficiently perform large-scale experiments with vulnerability exploits. It supports packing and running applications with their environments, injecting exploits, monitoring their success, and generating security reports. We also provide a corpus of example applications, taken from related works or implemented by us. [ABSTRACT FROM AUTHOR]

Testing of reactive systems is challenging because long input sequences are often needed to drive them into a state to test a desired feature. This is particularly problematic in on-target testing, where a system is tested in its real-life application environment and the amount of time required for resetting is high. This article presents an approach to discovering a test case chain-a single software execution that covers a group of test goals and minimizes overall test execution time. Our technique targets the scenario in which test goals for the requirements are given as safety properties. We give conditions for the existence and minimality of a single test case chain and minimize the number of test case chains if a single test case chain is infeasible. We report experimental results with our ChainCover tool for C code generated from Simulink models and compare it to state-of-the-art test suite generators. [ABSTRACT FROM AUTHOR]

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety-critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state space explosion can, however, impact model checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox CompleteTest can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs. [ABSTRACT FROM AUTHOR]

Among the software quality assurance techniques, testing stands out as one of the most frequently used techniques in practice. Consequently, testing is also widely studied in research. An important aspect in testing that receives much attention is the problem of automated test generation. This introductory paper summarizes the papers selected from the 25th IFIP International Conference on Testing Software and Systems (ICTSS 2013), which all deal with this topic. The field is briefly introduced by pointing out to some of the existing survey articles and relating the selected papers in this issue to each other and to the known body of knowledge. [ABSTRACT FROM AUTHOR]

We present a range of testing techniques for the Abstract Behavioral Specification (ABS) language and apply them to an industrial case study. ABS is a formal modeling language for highly variable, concurrent, component-based systems. The nature of these systems makes them susceptible to the introduction of subtle bugs that are hard to detect in the presence of steady adaptation. While static analysis techniques are available for an abstract language such as ABS, testing is still indispensable and complements analytic methods. We focus on fully automated testing techniques including black-box and glass-box test generation as well as runtime assertion checking, which are shown to be effective in an industrial setting. [ABSTRACT FROM AUTHOR]

Model-based testing (MBT) is about testing a software system using a model of its behaviour. To benefit fully from MBT, automation support is required. The goal of this systematic review is determining the current state of the art of prominent MBT tool support where we focus on tools that rely on state-based models. We automatically searched different source of information including digital libraries and mailing lists dedicated to the topic. Precisely defined criteria are used to compare selected tools and comprise support for test adequacy and coverage criteria, level of automation for various testing activities and support for the construction of test scaffolding. Simple adequacy criteria are supported but not advanced ones; data(-flow) criteria are seldom supported; support for creating test scaffolding varies a great deal. The results of this review should be of interest to a wide range of stakeholders: software companies interested in selecting the most appropriate MBT tool for their needs; organizations willing to invest into creating MBT tool support; researchers interested in setting research directions. [ABSTRACT FROM AUTHOR]

In this article, we report from a 22-months long action research study in which we evaluate the usefulness of a set of software development tools in an industrial setting, a small software company. We focus on how developers in the industry use and adopt these tools, what expectations they have on them, how the tools can be improved, and how the adoption process itself can be improved. We describe these change processes from a methodological perspective, how we monitored the processes, how we reviewed the outcomes, and the strategies that we applied. We show how the processes evolved, intermediate results, and the steps that were taken along the way based on the outcomes. We believe that the described study may inspire other tool-developers and/or researchers to organize similar studies to further our understanding of the complex processes involved in the adoption of software development tools in industry. [ABSTRACT FROM AUTHOR]

Automating the process of software testing is a very popular research topic and of real interest to industry. Test automation can take part on different levels, e.g., test execution, test case generation, test data generation. This survey gives an overview of state-of-the art test data generation tools, either academic or commercial. The survey focuses on white- and gray-box techniques. The list of existing tools was filtered with respect to their public availability, their maturity, and activity. The remaining seven tools, i.e., AgitarOne, CodePro AnalytiX, AutoTest, C++test, Jtest, RANDOOP, and PEX, are briefly introduced and their evaluation results are summarized. For the evaluation we defined 31 benchmark tests, which check the tools capabilities to generate test data that satisfies a given specification: 24 primitive type benchmarks and 7 non-primitive type and more complex with respect to the specification benchmarks. Most of the commercial tools implement a test data strategy that uses constant values found in the method under test or values that are slightly modified by means of mathematical operations. This strategy turns out to be very effective. In general, all tools that combine multiple techniques perform very well. For example PEX uses constraint solving techniques, but in cases where the constraint solver reaches its limitations it uses random based techniques to overcome those limitations. Especially, the two commercial tools AgitarOne and PEX that combine multiple approaches to test data generation are able to pass all 31 tests. This survey reflects the status in 2011. [ABSTRACT FROM AUTHOR]

In many development projects, testing has to be conducted under severe pressure due to limited resources and a challenging time schedule. Risk-based testing, which utilizes identified risks of the system for testing purposes, has a high potential to improve testing as it helps to optimize the allocation of resources and provides decision support for management. But for many organizations, the integration of a risk-based approach into established testing activities is a challenging task, and there are several options to do so. In this article, we analyze how risk is defined, assessed, and applied to support and improve testing activities in projects, products, and processes. We investigate these questions empirically by a multiple case study of currently applied risk-based testing activities in industry. The case study is based on three cases from different backgrounds, i.e., a test project in context of the extension of a large Web-based information system, product testing of a measurement and diagnostic equipment for the electrical power industry, as well as a test process of a system integrator of telecommunication solutions. By analyzing and comparing these different industrial cases, we draw conclusions on the state of risk-based testing and discuss possible improvements. [ABSTRACT FROM AUTHOR]

In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC's characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer's Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution. [ABSTRACT FROM AUTHOR]

Risk analysis and testing are conducted for different purposes. Risk analysis and testing nevertheless involve processes that may be combined to the benefit of both. We may use testing to support risk analysis and risk analysis to support testing. This paper surveys literature on the combined use of risk analysis and testing. First, the existing approaches are identified through a systematic literature review. The identified approaches are then classified and discussed with respect to main goal, context of use and maturity level. The survey highlights the need for more structure and rigor in the definition and presentation of approaches. Evaluations are missing in most cases. The paper may serve as a basis for examining approaches for the combined use of risk analysis and testing, or as a resource for identifying the adequate approach to use. [ABSTRACT FROM AUTHOR]

Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer. [ABSTRACT FROM AUTHOR]

In this paper, we show how to extend our approach to property-driven benchmark generation (PDBG) to support concurrency in the benchmarks: we systematically produce multi-process PROMELA code of known and tailored complexity that can then serve as benchmark for the evaluation of analysis and verification tools for concurrent software systems. Key to this extension is the way in which we control the corresponding property profiles, i.e., the statements about the validity of a predefined set of LTL properties, along the PDBG tool chain. It is based on the idea to introduce intricate structural elements like synchronization and interference at a local and therefore controllable level, only concerning a few parallel components, and subsequently obfuscating the locality of these structures at the global level via property-preserving transformations of both the system components and the properties. This leads to a scalable approach which we illustrate along some concrete examples. [ABSTRACT FROM AUTHOR]

A novel, model-based test case generation approach for validating reactive systems, especially those supporting richly structured data inputs and/or interactions, is presented. Given an executable system model and an extended symbolic grammar specifying plausible system inputs, the approach performs a model-based simulation to (i) ensure the consistency of the model with respect to the specified inputs, and (ii) generate corresponding test cases for validating the system. The model-based simulation produces a state transition diagram (STD) automatically justifying the model runtime behaviors within the test case coverage. The STD can further be transformed to produce an evolved symbolic grammar, which can then be used to incrementally generate a refined set of test cases. As a case study, we present a live sequence chart (LSC) model-based test generator, named LCT in short, for LSC simulation and consistency testing. The evolved symbolic grammar produced by the simulator can either be used to generate practical test cases for software testing, or be further refined by applying our model-based test generation approach again with additional test coverage criteria. We further show that LSCs can also be used to specify and test certain temporal system properties during the model simulation. Their satisfaction, reflected in the STD, can either be served as a directive for selective test generation, or a basis for further temporal property model checking. [ABSTRACT FROM AUTHOR]

Testing remains a widely used validation technique for software systems. However, recent needs in software development (e.g., in terms of security concerns) may require to extend this technique to address a larger set of properties. In this article, we explore the set of testable properties within the Safety-Progress classification where testability means to establish by testing that a relation, between the tested system and the property under scrutiny, holds. We characterize testable properties w.r.t. several relations of interest. For each relation, we give a sufficient condition for a property to be testable. Then, we study and delineate a fine-grain characterization of testable properties: for each Safety-Progress class, we identify the subset of testable properties and their corresponding test oracle. Furthermore, we address automatic test generation for the proposed framework by providing a general synthesis technique that allows to obtain canonical testers for the testable properties in the Safety-Progress classification. Moreover, we show how the usual notion of quiescence can be taken into account in our general framework, and, how quiescence improves the testability results. Then, we list some existing testing approaches that could benefit from this work by addressing a wider set of properties. Finally, we propose Java-PT, a prototype Java toolbox that implements the results introduced in this article. [ABSTRACT FROM AUTHOR]

Model-based testing is focused on testing techniques which rely on the use of models. The diversity of systems and software to be tested implies the need for research on a variety of models and methods for test automation. We briefly review this research area and introduce several papers selected from the 22nd International Conference on Testing Software and Systems (ICTSS). [ABSTRACT FROM AUTHOR]

utomated debugging attempts to locate the reason for a failure. Delta debugging minimizes the difference between two inputs, where one input is processed correctly while the other input causes a failure, using a series of test runs to determine the outcome of applied changes. Delta debugging is applicable to inputs or to the program itself, as long as a correct version of the program exists. However, complex errors are often masked by other program defects, making it impossible to obtain a correct version of the program through delta debugging in such cases. Iterative delta debugging extends delta debugging and removes a series of defects step by step, until the originally unresolved defect is isolated. The method is automated and managed to localize a bug in some real-life examples. [ABSTRACT FROM AUTHOR]

An introduction is presented in which the editor discusses various reports within the issue on topics including software testing through comparative functional coverage, delta debugging of software, and the use of Bayesian networks in improving cross-product coverage in simulation-based hardware verification.

Model-checking is becoming an accepted technique for debugging hardware and software systems. Debugging is based on the “Check/Analyze/Fix” loop: check the system against a desired property, producing a counterexample when the property fails to hold; analyze the generated counterexample to locate the source of the error; fix the flawed artifact—the property or the model. The success of model-checking non-trivial systems critically depends on making this Check/Analyze/Fix loop as tight as possible. In this paper, we concentrate on the Analyze part of the debugging loop. To this end, we present a framework for generating, structuring and exploring counterexamples, implemented in a tool called KEGVis. The framework is based on the idea that the most general type of evidence to why a property holds or fails to hold is a proof. Such proofs can be presented to the user in the form of proof-like counterexamples, without sacrificing any of the intuitiveness and close relation to the model that users have learned to expect from model-checkers. Moreover, proof generation is flexible, and can be controlled by strategies, whether built into the tool or specified by the user, thus enabling generation of the most “interesting” counterexample and its interactive exploration. Moreover, proofs can be used to generate and display all relevant evidence together, a technique referred to as abstract counterexamples. Overall, our framework can be used for explaining the reason why the property failed or succeeded, determining whether the property was correct (“specification debugging”), and for general model exploration. [ABSTRACT FROM AUTHOR]

Abstraction methods have become one of the most interesting topics in the automatic verification of software systems because they can reduce the state space to be explored and allow model checking of more complex systems. Nevertheless, there is a lack of tools actually supporting this technique. One direction for abstracting a system is to transform its formal description (its model) into a simpler version specified in the same language, thus skipping the construction of a specific (model checking) tool for the abstract model. The abstraction of the model should be followed by the abstraction of the temporal formulas to be checked. This paper presents αspin, a tool for the integration of abstraction (for models and formulas) into the well-known model checker spin. We present the theoretical results supporting the implementation together with a case study. [ABSTRACT FROM AUTHOR]

The boundaries of model-checking have been extended through the use of abstraction. These techniques are conservative, in the following sense: when the verification succeeds, the verified property is guaranteed to hold; but when it fails, it may result either from the non satisfaction of the property, or from a too rough abstraction. In case of failure, it is, in general, undecidable whether an abstract trace corresponding to a counter-example has any concrete counterparts. For debugging purposes, one usually desires to go further than giving a “yes/no” answer (actually, a “yes/don’t know” answer!), and look for such concrete counter-examples. We propose a solution in which we apply standard test-pattern generation technology to search for concrete instances of abstract traces. [ABSTRACT FROM AUTHOR]

Program verification and testing techniques are crucial in fruitfully analyzing and validating complex software systems. This is an active research area and has produced various promising techniques in the last decade. However, many challenges lie ahead. We review the research area and summarize six papers selected from the Sixth International Symposium on Theoretical Aspects of Software Engineering (TASE 2012). [ABSTRACT FROM AUTHOR]

Tools that implement automated verification techniques can be used to fruitfully analyze and validate complex software systems. Developing such tools is an active research area that has produced several promising techniques in the last decade: however, many challenges lie ahead. We briefly review the research area and summarize four papers selected from the Eighteenth International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2012). [ABSTRACT FROM AUTHOR]

An introduction to the journal is presented in which the guest editor discusses various reports published within the issue including one by Kinga Dobolyi, Elizabeth Soechting, and Westley Weimer on the SMART regression testing tool, one by AlessandroMarchetto and colleagues on crawler, an automatic web testing technique, and one by Marco Torchiano, Filippo Ricca, and AlessandroMarchetto on the distribution of bugs among different layers of applications.