Search

Random Forests are one of the most popular classifiers in machine learning. The larger they are, the more precise the outcome of their predictions. However, this comes at a cost: it is increasingly difficult to understand why a Random Forest made a specific choice, and its running time for classification grows linearly with the size (number of trees). In this paper, we propose a method to aggregate large Random Forests into a single, semantically equivalent decision diagram which has the following two effects: (1) minimal, sufficient explanations for Random Forest-based classifications can be obtained by means of a simple three step reduction, and (2) the running time is radically improved. In fact, our experiments on various popular datasets show speed-ups of several orders of magnitude, while, at the same time, also significantly reducing the size of the required data structure. [ABSTRACT FROM AUTHOR]

This paper presents a compositional approach to active automata learning of Systems of Procedural Automata (SPAs), an extension of Deterministic Finite Automata (DFAs) to systems of DFAs that can mutually call each other. SPAs are of high practical relevance, as they allow one to efficiently learn intuitive recursive models of recursive programs after an easy instrumentation that makes calls and returns observable. Key to our approach is the simultaneous inference of individual DFAs for each of the involved procedures via expansion and projection: membership queries for the individual DFAs are expanded to membership queries of the entire SPA, and global counterexample traces are transformed into counterexamples for the DFAs of concerned procedures. This reduces the inference of SPAs to a simultaneous inference of the DFAs for the involved procedures for which we can utilize various existing regular learning algorithms. The inferred models are easy to understand and allow for an intuitive display of the procedural system under learning that reveals its recursive structure. We implemented the algorithm within the LearnLib framework in order to provide a ready-to-use tool for practical application which is publicly available on GitHub for experimentation. [ABSTRACT FROM AUTHOR]

This paper reviews the seemingly inevitable trend that software tools are no longer just a means for supporting the design, construction, and analysis of (large-scale) systems, but become so complex that each of them turns into a reality of their own, with its own 'physics', that needs to be studied in its own right. The true effects of combining methodologies as diverse as classical static analysis, model checking, SAT and SMT solving, and dynamic methods such as simulation, runtime verification, testing, and learning, with their dedicated means of optimizations in terms of, e.g., BDD coding, parallelization, and various forms of abstraction and reduction, are very dependent on the particular tools and typically hardly predictable. Corresponding experimental investigations, today often supported by diverse and frequent tool challenges, provide interesting indications about the applied technology, but typically fail to provide sufficient evidence to transfer results to other settings and tools. Moreover, implementation-specific details often dominate the observed effects which thereby become invalid for drawing conceptual conclusions. On the other hand, requiring consequent in-depth analysis of any experimental observation in order to pinpoint the underlying conceptual consequences before publication would slow down the scientific exchange and also hinder the scientific progress. This paper analyzes the situation of today's software tools from a global perspective in terms of a SWOT (Strength, Weaknesses, Opportunities, Treats) analysis, identifies challenges, and establishes a global vision for overcoming current weaknesses. [ABSTRACT FROM AUTHOR]

The jABC is a framework for process modelling and execution according to the XMDD (eXtreme model-driven design) paradigm, which advocates the rigorous use of user-level models in the software development process and software life cycle. We have used the jABC in the domain of scientific workflows for more than a decade now-an occasion to look back and take stock of our experiences in the field. On the one hand, we discuss results from the analysis of a sample of nearly 100 scientific workflow applications that have been implemented with the jABC. On the other hand, we reflect on our experiences and observations regarding the workflow development process with the framework. We then derive and discuss ongoing further developments and future perspectives for the framework, all with an emphasis on simplicity for end users through increased domain specificity. Concretely, we describe how the use of the PROPHETS synthesis plugin can enable a semantics-based simplification of the workflow design process, how with the jABC4 and DyWA frameworks more attention is paid to the ease of data management, and how the Cinco SCCE Meta-Tooling Suite can be used to generate tailored workflow management tools. [ABSTRACT FROM AUTHOR]

In this paper, we present the envisioned style and scope of the new topic "Explanation Paradigms Leveraging Analytic Intuition" (ExPLAIn) with the International Journal on Software Tools for Technology Transfer (STTT). Intention behind this new topic is to (1) explicitly address all aspects and issues that arise when trying to, if possible, reveal and then confirm hidden properties of black-box systems, or (2) to enforce vital properties by embedding them into appropriate system contexts. Machine-learned systems, such as Deep Neural Networks, are particularly challenging black-box systems, and there is a wealth of formal methods for analysis and verification waiting to be adapted and applied. The selection of papers of this first Special Section of ExPLAIn, most of which were co-authored by editorial board members, is an illustrative example of the style and scope envisioned: In addition to methodological papers on verification, explanation, and their scalability, case studies, tool papers, literature reviews, and position papers are also welcome. [ABSTRACT FROM AUTHOR]

We present a systematic approach to the automatic generation of platform-independent benchmarks of realistic structure and tailored complexity for evaluating verification tools for reactive systems. The idea is to mimic a systematic constraint-driven software development process by automatically transforming randomly generated temporal-logic-based requirement specifications on the basis of a sequence of property-preserving, randomly generated structural design decisions into executable source code of a chosen target language or platform. Our automated transformation process steps through dedicated representations in terms of Büchi automata, Mealy machines, decision diagram models, and code models. It comprises LTL synthesis, model checking, property-oriented expansion, path condition extraction, theorem proving, SAT solving, and code motion. This setup allows us to address different communities via a growing set of programming languages, tailored sets of programming constructs, different notions of observation, and the full variety of LTL properties-ranging from mere reachability over general safety properties to arbitrary liveness properties. The paper illustrates the corresponding tool chain along accompanying examples, emphasizes the current state of development, and sketches the envisioned potential and impact of our approach. [ABSTRACT FROM AUTHOR]

In this paper, we show how to extend our approach to property-driven benchmark generation (PDBG) to support concurrency in the benchmarks: we systematically produce multi-process PROMELA code of known and tailored complexity that can then serve as benchmark for the evaluation of analysis and verification tools for concurrent software systems. Key to this extension is the way in which we control the corresponding property profiles, i.e., the statements about the validity of a predefined set of LTL properties, along the PDBG tool chain. It is based on the idea to introduce intricate structural elements like synchronization and interference at a local and therefore controllable level, only concerning a few parallel components, and subsequently obfuscating the locality of these structures at the global level via property-preserving transformations of both the system components and the properties. This leads to a scalable approach which we illustrate along some concrete examples. [ABSTRACT FROM AUTHOR]

In this paper, we present the LearnLib, a library of tools for automata learning, which is explicitly designed for the systematic experimental analysis of the profile of available learning algorithms and corresponding optimizations. Its modular structure allows users to configure their own tailored learning scenarios, which exploit specific properties of their envisioned applications. As has been shown earlier, exploiting application-specific structural features enables optimizations that may lead to performance gains of several orders of magnitude, a necessary precondition to make automata learning applicable to realistic scenarios. [ABSTRACT FROM AUTHOR]

In this paper, we review behavior-based model construction from a point of view characterized by verification, model checking, and abstraction. It turns out that abstract interpretation is the key to scaling known learning techniques for practical applications, that model checking may serve as a teaching aid in the learning process underlying the model construction, and that there are various synergies with other validation and verification techniques. We will illustrate our discussion by means of a realistic telecommunications scenario, where the underlying system has grown over the last two decades, the available system documentation consists of not much more than user manuals and protocol standards, and the revision cycle times are extremely short. In this situation, behavior-based model construction provides a sound basis, e.g., for test-suite design and maintenance, test organization, and test evaluation. [ABSTRACT FROM AUTHOR]

In this paper, our solution to the problem of modelling functionally complex communication systems at the application level, based on lightweight coordination, is extended to seamlessly capture system-level testing as well. This extension could be realized simply by self-application: the bulk of the work for integrating system-level testing into our development environment, the ABC, concerned domain modelling, which can be done using the ABC. Therefore, the extension of the ABC to cover system-level testing was merely an application development on the basis of the ABC, illustrated here in the domain of Computer Telephony Integration. Here the adoption of a coarse-grained approach to test design, which is central to the scalability of the overall testing environment, is the enabling aspect for system-level test automation. Together with our lightweight coordination approach this induces an understandable modelling paradigm of system-wide test cases that is adequate for the needs and requirements of industrial test engineers. In particular, it enables test engineers to graphically design complex test cases that, in addition, can even be automatically checked for their intended purposes via model checking. [ABSTRACT FROM AUTHOR]

This Special Section contains selected papers from ISoLA 2004, the First International Symposium on Leveraging Applications of Formal Methods held in Paphos (Cyprus) October–November 2004. ISoLA 2004 served the need of providing a forum for developers, users, and researchers to discuss issues related to the adoption and use of rigorous tools and methods for the specification, analysis, verification, certification, construction, test, and maintenance of systems from the point of view of their different application domains. The seven selected papers illustrate the intended variety of ISoLA’s scope. [ABSTRACT FROM AUTHOR]

In this paper, we prove that Hennessy–Milner Logic (HML), despite its structural limitations, is sufficiently expressive to specify an initial property φ 0 and a characteristic invariant χ I for an arbitrary finite-state process P such that φ 0 ∧ AG (χ I) is a characteristic formula for P. This means that a process Q, even if infinite state, is bisimulation equivalent to P iff Q ⊧ φ 0 ∧ AG (χ I) . It follows, in particular, that it is sufficient to check an HML formula for each state of a finite-state process to verify that it is bisimulation equivalent to P. In addition, more complex systems such as context-free processes can be checked for bisimulation equivalence with P using corresponding model checking algorithms. Our characteristic invariant is based on so called class-distinguishing formulas that identify bisimulation equivalence classes in P and which are expressed in HML. We extend Kanellakis and Smolka's partition refinement algorithm for bisimulation checking in order to generate concise class-distinguishing formulas for finite-state processes. [ABSTRACT FROM AUTHOR]

In this paper, we present Forest GUMP (for Generalized, Unifying Merge Process) a tool for verification and precise explanation of Random forests. Besides pre/post-condition-based verification and equivalence checking, Forest GUMP also supports three concepts of explanation, the well-known model explanation and outcome explanation, as well as class characterization, i.e., the precise characterization of all samples that are equally classified. Key technology to achieve these results is algebraic aggregation, i.e., the transformation of a Random Forest into a semantically equivalent, concise white-box representation in terms of Algebraic Decision Diagrams (ADDs). The paper sketches the method and demonstrates the use of Forest GUMP along illustrative examples. This way readers should acquire an intuition about the tool, and the way how it should be used to increase the understanding not only of the considered dataset, but also of the character of Random Forests and the ADD technology, here enriched to comprise infeasible path elimination. As Forest GUMP is publicly available all experiments can be reproduced, modified, and complemented using any dataset that is available in the ARFF format. [ABSTRACT FROM AUTHOR]

TADS are a novel, concise white-box representation of neural networks. In this paper, we apply TADS to the problem of neural network verification, using them to generate either proofs or concise error characterizations for desirable neural network properties. In a case study, we consider the robustness of neural networks to adversarial attacks, i.e., small changes to an input that drastically change a neural networks perception, and show that TADS can be used to provide precise diagnostics on how and where robustness errors a occur. We achieve these results by introducing Precondition Projection, a technique that yields a TADS describing network behavior precisely on a given subset of its input space, and combining it with PCA, a traditional, well-understood dimensionality reduction technique. We show that PCA is easily compatible with TADS. All analyses can be implemented in a straightforward fashion using the rich algebraic properties of TADS, demonstrating the utility of the TADS framework for neural network explainability and verification. While TADS do not yet scale as efficiently as state-of-the-art neural network verifiers, we show that, using PCA-based simplifications, they can still scale to medium-sized problems and yield concise explanations for potential errors that can be used for other purposes such as debugging a network or generating new training samples. [ABSTRACT FROM AUTHOR]

In this paper, we present an algebraic approach to the precise and global verification and explanation of Rectifier Neural Networks, a subclass of Piece-wise Linear Neural Networks (PLNNs), i.e., networks that semantically represent piece-wise affine functions. Key to our approach is the symbolic execution of these networks that allows the construction of semantically equivalent Typed Affine Decision Structures (TADS). Due to their deterministic and sequential nature, TADS can, similarly to decision trees, be considered as white-box models and therefore as precise solutions to the model and outcome explanation problem. TADS are linear algebras, which allows one to elegantly compare Rectifier Networks for equivalence or similarity, both with precise diagnostic information in case of failure, and to characterize their classification potential by precisely characterizing the set of inputs that are specifically classified, or the set of inputs where two network-based classifiers differ. All phenomena are illustrated along a detailed discussion of a minimal, illustrative example: the continuous XOR function. [ABSTRACT FROM AUTHOR]

In this paper, we show how our approach to active continuous quality control (ACQC), which employs learning technology to automatically maintain test models along the whole life cycle, can be extended to include risk analysts for supporting risk-based testing. Key to this enhancement is the tailoring of ACQC's characteristic automata learning-based model extraction to prioritize critical aspects. Technically, risk analysts are provided with an abstract modeling level tailored to design test components (learning symbols) that encompass data-flow constraints reflecting a given risk profile. The resulting alphabet models are already sufficient to steer the ACQC process in a fashion that increases the risk coverage, while it at the same time radically reduces the testing effort. We illustrate our approach by means of case studies with Springer's Online Conference Service (OCS) which show the impact of the risk prioritization on the performance: risk-based regression testing tailored for system migration and for pure functional evolution. [ABSTRACT FROM AUTHOR]

This paper (1) summarizes the history of the RERS challenge for the analysis and verification of reactive systems, its profile and intentions, its relation to other competitions, and, in particular, its evolution due to the feedback of participants, and (2) presents the most recent development concerning the synthesis of hard benchmark problems. In particular, the second part proposes a way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors. This gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing. [ABSTRACT FROM AUTHOR]

This paper proposes a simplicity-oriented approach and framework for language-to-language transformation of, in particular, graphical languages. Key to simplicity is the decomposition of the transformation specification into sub-rule systems that separately specify purpose-specific aspects. We illustrate this approach by employing a variation of Plotkin's Structural Operational Semantics (SOS) for pattern-based transformations of typed graphs in order to address the aspect 'computation' in a graph rewriting fashion. Key to our approach are two generalizations of Plotkin's structural rules: the use of graph patterns as the matching concept in the rules, and the introduction of node and edge types. Types do not only allow one to easily distinguish between different kinds of dependencies, like control, data, and priority, but may also be used to define a hierarchical layering structure. The resulting Type-based Structural Operational Semantics (TSOS) supports a well-structured and intuitive specification and realization of semantically involved language-to-language transformations adequate for the generation of purpose-specific views or input formats for certain tools, like, e.g., model checkers. A comparison with the general-purpose transformation frameworks ATL and Groove, illustrates along the educational setting of our graphical WebStory language that TSOS provides quite a flexible format for the definition of a family of purpose-specific transformation languages that are easy to use and come with clear guarantees. [ABSTRACT FROM AUTHOR]

This paper presents dynamic testing, a method that exploits automata learning to systematically test (black box) systems almost without prerequisites. Based on interface descriptions and optional sample test cases, our method successively explores the system under test (SUT), in order to extrapolate a behavioural model. This is in turn used to steer the further exploration process. Due to the applied learning technique, our method is optimal in the sense that the extrapolated models are most concise (i.e. state minimal) in consistently representing all the information gathered during the exploration. Using the LearnLib, our framework for automata learning, our method can be elegantly combined with numerous optimisations of the learning procedure, with various choices of model structures, and with the option of dynamically/interactively enlarging the alphabet underlying the learning process. The latter is important in the Web context, where totally new situations may arise when following links. All these features are illustrated using as a case study the web application Mantis, a bug tracking system widely used in practice. In addition, we present another case study that demonstrates the scalability of the approach. We show how the dynamic testing procedure works and how behavioural models arise that concisely summarize the current testing effort. It turns out that these models reveal the system structure from a user perspective. Besides steering the automatic exploration process, they are ideal for user guidance and to support analyses to improve the system understanding, as they reveal the system structure from a user perspective. [ABSTRACT FROM AUTHOR]

This is the second issue in the new "Competitions and Challenges" (CoCha) theme of the International Journal on Software Tools for Technology Transfer. The new theme was established to support competitions and challenges with an appropriate publication venue. The first issue presented the competition on software testing Test-Comp 2019, which was part of the TOOLympics 2019 event. In this second issue for TOOLympics, we present selected competition reports. The TOOLympics event took place as part of the 25-years celebration of the conference TACAS. The goal of the event was to provide an overview of competitions and challenges in the area of formal methods. [ABSTRACT FROM AUTHOR]

Over the years, schedulability of Cyber-Physical Systems (CPS) has mainly been performed by analytical methods. These techniques are known to be effective but limited to a few classes of scheduling policies. In a series of recent work, we have shown that schedulability analysis of CPS could be performed with a model-based approach and extensions of verification tools such as UPPAAL. One of our main contributions has been to show that such models are flexible enough to embed various types of scheduling policies, which goes beyond those in the scope of analytical tools. However, the specification of scheduling problems with model-based approaches requires a substantial modeling effort, and a deep understanding of the techniques employed in order to understand their results. In this paper we propose simplicity-driven high-level specification and verification frameworks for various scheduling problems. These frameworks consist of graphical and user-friendly languages for describing scheduling problems. The high-level specifications are then automatically translated to formal models, and results are transformed back into the comprehensible model view. To construct these frameworks we exploit a meta-modeling approach based on the tool generator CINCO. Additionally we propose in this paper two new techniques for scheduling analysis. The first performs runtime monitoring using the CUSUM algorithm to detect alarming change in the system. The second performs optimization using efficient statistical techniques. We illustrate our frameworks and techniques on two case studies. [ABSTRACT FROM AUTHOR]

Even with the help of powerful metamodeling frameworks, the development of domain-specific graphical modeling tools is usually a complex, repetitive, and tedious task, which introduces substantial upfront costs often prohibiting such approaches in practice. In order to reduce these costs, the presented Cinco meta tooling suite is designed to provide a holistic approach that greatly simplifies the development of such domain-specific tools. Our solution is based on the idea to apply the concept of domain specialization also to the (meta-)domain of “domain-specific modeling tools”. Important here is our focus on complex graph-based models, comprising various kinds of nodes and edges together with their individual representation, correlations, and interpretation. This focus allows for high-level specifications of the model structures and functionalities as the prerequisite for push-button tool generation. [ABSTRACT FROM AUTHOR]

The article discusses various reports published within the issue, including the experiment for testing embedded software by using a tool to demonstrate that it improves the testing process by Chetali and Nguyen, the survey and evaluation of a number of model checking tools then investigates its appropriateness for use with C code by Schlich and Kowalewski, and a methodology for Systems on a Chip from event B.

The goal of the RERS challenge is to evaluate the effectiveness of various verification and validation approaches on reactive systems, a class of systems that is highly relevant for industrial critical applications. The RERS challenge brings together researchers from different areas of software verification and validation, including static analysis, model checking, theorem proving, symbolic execution, and testing. The challenge provides a forum for experimental comparison of different techniques on specifically designed verification tasks. These benchmarks are automatically synthesized to exhibit chosen properties, and then enhanced to include dedicated dimensions of difficulty, such as conceptual complexity of the properties (e.g., reachability, safety, liveness), size of the reactive systems (a few hundred lines to millions of lines), and complexity of language features (arrays and pointer arithmetic). The STTT special section on RERS describes the results of the evaluations and the different analysis techniques that were used in the RERS challenges 2012 and 2013. [ABSTRACT FROM AUTHOR]

Constraint-based variability modeling is a flexible, declarative approach to managing solution-space variability. Product variants are defined in a top-down manner by successively restricting the admissible combinations of product artifacts until a specific product variant is determined. In this paper, we illustrate the range of constraint-based variability modeling by discussing two of its extreme flavors: constraint-guarded variability modeling and constraint-driven variability modeling. The former applies model checking to establish the global consistency of product variants which are built by manual specification of variations points, whereas the latter uses synthesis technology to fully automatically generate product variants that satisfy all given constraints. Each flavor is illustrated by means of a concrete case study. [ABSTRACT FROM AUTHOR]

Embedded Systems, by their nature, constitute a meeting point for communities with extremely different background. In particular, the high demands for quality and reliability for embedded systems have led to complementary quality assurance efforts: hardware engineers have developed techniques for dynamic verification in terms of co-simulation, which, in particular, addresses the different nature of hardware and software components. Thus these techniques are tailored for the transactional level, which comprises dedicated models for the hardware and the software parts. On the other hand, there is a bulk of work on formal verification techniques, which typically address higher levels of abstraction. These techniques are exhaustive in the sense that they cover all the infinite possible paths of their models, however at the price of neglecting many of the low-level aspects treated by co-simulation. It is the goal of this paper to increase the mutual understanding between these communities and to animate research at this exciting borderline. [ABSTRACT FROM AUTHOR]