Search

Deep learning has emerged as an effective approach for creating modern software systems, with neural networks often surpassing hand-crafted systems. Unfortunately, neural networks are known to suffer from various safety and security issues. Formal verification is a promising avenue for tackling this difficulty, by formally certifying that networks are correct. We propose an SMT-based technique for verifyingbinarized neural networks— a popular kind of neural network, where some weights have been binarized in order to render the neural network more memory and energy efficient, and quicker to evaluate. One novelty of our technique is that it allows the verification of neural networks that include both binarized and non-binarized components. Neural network verification is computationally very difficult, and so we propose here various optimizations, integrated into our SMT procedure as deduction steps, as well as an approach for parallelizing verification queries. We implement our technique as an extension to the Marabou framework, and use it to evaluate the approach on popular binarized neural network architectures.

A broad, long-term research project is described, which will lead to the computer becoming an equal member of the system-development team, continuously making proactive contributions, akin to those expected from an experienced and knowledgeable customer or user, a conscientious QA engineer, a strict regulatory auditor, an engineering -team leader, or the organization’s CTO. The web extra at https://youtu.be/mmrv8ZACbpU describes the authors’ novel “wise computing” approach and demonstrates one possible application. The second web extra at https://youtu.be/DtpvMxMwYPM extends the discussion of the authors’ novel “wise computing” approach and presents a case study.

ScenarioTools is an Eclipse-based tool suite for the scenario-based modeling and analysis of reactive systems. ScenarioTools especially targets the modeling and analysis of systems where the behavior of the components is sensitive to changes in the component structure that can occur at run-time. For example, in a system of communicating cars, the cars' relationships can change due to their movement and influence how cars must interact. The modeling in ScenarioTools is based on the Scenario Modeling Language (SML), an extended variant of Live Sequence Charts (LSCs). For modeling structural changes and conditions, graph transformation rules can be combined with SML. The specifications are executable and can be analyzed by simulation. ScenarioTools further supports a formal synthesis procedure that can find specification inconsistencies or prove the specification's realizability. In this article, we illustrate the features of ScenarioTools by an example and describe its architecture.

Autonomous vehicles are highly complex systems, required to function reliably in a wide variety of situations. Manually crafting software controllers for these vehicles is difficult, but there has been some success in using deep neural networks generated using machine-learning. However, deep neural networks are opaque to human engineers, rendering their correctness very difficult to prove manually; and existing automated techniques, which were not designed to operate on neural networks, fail to scale to large systems. This paper focuses on proving the adversarial robustness of deep neural networks, i.e. proving that small perturbations to a correctly-classified input to the network cannot cause it to be misclassified. We describe some of our recent and ongoing work on verifying the adversarial robustness of networks, and discuss some of the open questions we have encountered and how they might be addressed., In Proceedings FVAV 2017, arXiv:1709.02126

Deep neural networks are revolutionizing the way complex systems are developed. However, these automatically-generated networks are opaque to humans, making it difficult to reason about them and guarantee their correctness. Here, we propose a novel approach for verifying properties of a widespread variant of neural networks, called recurrent neural networks. Recurrent neural networks play a key role in, e.g., speech recognition, and their verification is crucial for guaranteeing the reliability of many critical systems. Our approach is based on the inference of invariants, which allow us to reduce the complex problem of verifying recurrent networks into simpler, non-recurrent problems. Experiments with a proof-of-concept implementation of our approach demonstrate that it performs orders-of-magnitude better than the state of the art.

Deep neural network (DNN) verification is an emerging field, with diverse verification engines quickly becoming available. Demonstrating the effectiveness of these engines on real-world DNNs is an important step towards their wider adoption. We present a tool that can leverage existing verification engines in performing a novel application: neural network simplification, through the reduction of the size of a DNN without harming its accuracy. We report on the work-flow of the simplification process, and demonstrate its potential significance and applicability on a family of real-world DNNs for aircraft collision avoidance, whose sizes we were able to reduce by as much as 10%.

Deep neural networks are increasingly being used as controllers for safety-critical systems. Because neural networks are opaque, certifying their correctness is a significant challenge. To address this issue, several neural network verification approaches have recently been proposed. However, these approaches afford limited scalability, and applying them to large networks can be challenging. In this paper, we propose a framework that can enhance neural network verification techniques by using over-approximation to reduce the size of the network—thus making it more amenable to verification. We perform the approximation such that if the property holds for the smaller (abstract) network, it holds for the original as well. The over-approximation may be too coarse, in which case the underlying verification tool might return a spurious counterexample. Under such conditions, we perform counterexample-guided refinement to adjust the approximation, and then repeat the process. Our approach is orthogonal to, and can be integrated with, many existing verification techniques. For evaluation purposes, we integrate it with the recently proposed Marabou framework, and observe a significant improvement in Marabou’s performance. Our experiments demonstrate the great potential of our approach for verifying larger neural networks.

Deep reinforcement learning (RL) has recently been successfully applied to networking contexts including routing, flow scheduling, congestion control, packet classification, cloud resource management, and video streaming. Deep-RL-driven systems automate decision making, and have been shown to outperform state-of-the-art handcrafted systems in important domains. However, the (typical) non-explainability of decisions induced by the deep learning machinery employed by these systems renders reasoning about crucial system properties, including correctness and security, extremely difficult. We show that despite the obscurity of decision making in these contexts, verifying that deep-RL-driven systems adhere to desired, designer-specified behavior, is achievable. To this end, we initiate the study of formal verification of deep RL and present Verily, a system for verifying deep-RL-based systems that leverages recent advances in verification of deep neural networks. We employ Verily to verify recently-introduced deep-RL-driven systems for adaptive video streaming, cloud resource management, and Internet congestion control. Our results expose scenarios in which deep-RL-driven decision making yields undesirable behavior. We discuss guidelines for building deep-RL-driven systems that are both safer and easier to verify.

Deep neural networks are revolutionizing the way complex systems are designed. Consequently, there is a pressing need for tools and techniques for network analysis and certification. To help in addressing that need, we present Marabou, a framework for verifying deep neural networks. Marabou is an SMT-based tool that can answer queries about a network’s properties by transforming these queries into constraint satisfaction problems. It can accommodate networks with different activation functions and topologies, and it performs high-level reasoning on the network that can curtail the search space and improve performance. It also supports parallel execution to further enhance scalability. Marabou accepts multiple input formats, including protocol buffer files generated by the popular TensorFlow framework for neural networks. We describe the system architecture and main components, evaluate the technique and discuss ongoing work.

International audience; This paper describes SMTCoq, a plug-in for the integration of external solvers into the Coq proof assistant. Based on a checker for generic first-order proof certificates fully implemented and proved correct in Coq, SMTCoq offers facilities to check answers from external SAT and SMT solvers and to increase Coq's automation using such solvers, all in a safe way. The current version supports proof certificates produced by the SAT solver ZChaff, for propositional logic, and the SMT solvers veriT and CVC4, for the quantifier-free fragment of the combined theory of fixed-size bit vectors, functional arrays with extensionality, linear integer arithmetic, and uninterpreted function symbols.

As part of expanding the implementation and use of the behavioral programming (BP) approach in a variety of languages and configurations, we tackle some of the challenges associated with applying the approach in a truly distributed, decentralized manner, where different modules run on separate machines. BP supports the development of reactive applications from modules that are aligned with the desired and undesired scenarios of system behaviors as described, say, in a requirements document, in an enhancement request, or a field problem report. A key advantage of this approach is that it facilitates incremental development where loosely coupled modules are added as requirements are introduced, and meaningful prototype execution can be carried out from early stages of development. In BP, each behavioral module (called a behavior thread) takes care of a separate facet of the requirements, thus control is conceptually decentralized. However, as the underlying principles of BP call for constant synchronization of, and "consultation" with, all behavior threads, efficient implementation in a physically distributed environment is a significant challenge on the road to broader acceptance of BP as a viable new way to develop systems. We begin by describing an implementation of BP in Erlang, where the coordination protocol is implemented via message passing. We demonstrate through examples how developing distributed systems in Erlang can benefit from BP advantages of incremental development and alignment of the code modules with the requirements. Next, we propose general BP design patterns (not limited to Erlang) for using BP in distributed applications, showing how to design applications using BP without forcing full synchronization among all threads at each step of the execution. This allows modules to run at different time scales and to wait for external input without stalling the entire system. Finally, we propose ways to alter the execution mechanism of BP so that execution can progress without necessarily waiting for the synchronization of all the threads. The enhanced execution algorithm has the potential of accelerating the distributed execution of behavioral programs. We present a design pattern for programming behaviorally in Erlang.We present a proof-of-concept for a BP execution infrastructure in Erlang.We present BP design patterns for interfacing with unsynchronized environments.We present BP design patterns for coordinating behaviors at different time scales.We present mechanisms for eager execution in BP.

Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). The technique is based on the simplex method, extended to handle the non-convex Rectified Linear Unit (ReLU) activation function, which is a crucial ingredient in many modern neural networks. The verification procedure tackles neural networks as a whole, without making any simplifying assumptions. We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks verified using existing methods.

This extended abstract reports on current progress of SMTCoq, a communication tool between the Coq proof assistant and external SAT and SMT solvers. Based on a checker for generic first-order certificates implemented and proved correct in Coq, SMTCoq offers facilities both to check external SAT and SMT answers and to improve Coq's automation using such solvers, in a safe way. Currently supporting the SAT solver zChaff, and the SMT solver veriT for the combination of the theories of congruence closure and linear integer arithmetic, SMTCoq is meant to be extendable with a reasonable amount of effort: we present work in progress to support the SMT solver CVC4 and the theory of bit vectors., In Proceedings HaTT 2016, arXiv:1606.05427

Cyber-physical systems are found in many areas, e.g., manufacturing or smart cities. They consist of multiple components that cooperate to provide the desired functionality. This need for cooperation causes complex interactions between components, which makes developing cyber-physical systems difficult, especially systems whose component structure changes dynamically at runtime. We have created a formal, scenario-based specification method which makes it easier to develop distributed cyber-physical systems. We previously presented an approach for the distributed execution of such specifications based on naive and inefficient broadcasting. In this paper we propose a more efficient approach which uses the available network resources more economically.

In behavioral programming, a program consists of separate modules called behavior threads, each representing a part of the system’s allowed, necessary or forbidden behavior. An execution of the program is a series of synchronizations between these threads, where at each synchronization point an event is selected to be carried out. As a result, the execution speed is dictated by the slowest thread. We propose an eager execution mechanism for such programs, which builds upon the realization that it is often possible to predict the outcome of a synchronization point even without waiting for slower threads to synchronize. This allows faster threads to continue running uninterrupted, whereas slower ones catch up at a later time. Consequently, eager execution brings about increased system performance, better support for the modular design of programs, and the ability to distribute programs across several machines. It also allows to apply behavioral programming to a variety of problems that were previously outside its scope. We illustrate the method by concrete examples, implemented in a behavioral programming framework in C + + .