Search

Reinforcement learning (RL) algorithms have been used to learn how to implement tasks in uncertain and partially unknown environments. In practice, environments are usually uncontrolled and may affect task performance in an adversarial way. In this paper, we model the interaction between an RL agent and its potentially adversarial environment as a turn-based zero-sum stochastic game. The task requirements are represented both qualitatively as a linear temporal logic specification, and quantitatively as a reward function. For each case in which the specification is realizable and can be equivalently transformed into a deterministic Buchi automaton, we show that there always exists a memoryless almost-sure winning strategy that is epsilon-optimal for the discountedd sum objective for any arbitrary positive epsilon. We propose a probably approximately correct (PAC) learning algorithm that learns such a strategy efficiently in an online manner with a priori unknown reward functions and unknown transition distributions. To the best of our knowledge, this is the first result on PAC learning in stochastic games with independent quantitative and qualitative objectives.

The majority of existing linear temporal logic (LTL) planning methods rely on the construction of a discrete product automaton, which combines a discrete abstraction of robot mobility and a Buchi automaton that captures the LTL specification. Representing this product automaton as a graph and using graph search techniques, optimal plans that satisfy the LTL task can be synthesized. However, constructing expressive discrete abstractions makes the synthesis problem computationally intractable. In this article, we propose a new sampling-based LTL planning algorithm that does not require any discrete abstraction of robot mobility. Instead, it incrementally builds trees that explore the product state-space, until a maximum number of iterations is reached or a feasible plan is found. The use of trees makes data storage and graph search tractable, which significantly increases the scalability of our algorithm. To accelerate the construction of feasible plans, we introduce bias in the sampling process, which is guided by transitions in the Buchi automaton that belong to the shortest path to the accepting states. We show that our planning algorithm, with and without bias, is probabilistically complete and asymptotically optimal. Finally, we present numerical experiments showing that our method outperforms relevant temporal logic planning methods.

Linear temporal logic (LTL) is extensively used in formal methods, in particular in runtime verification (RV) and in model checking. Its propositional version was shown by Wolper (1983, Inform Control 56(1/2): 72–99) to be limited in expressiveness. Several extensions of propositional LTL, which promote the expressive power to that of Buchi automata, have therefore been proposed; however, none of those, by and large, have been adopted for formal methods. We present an extension of propositional LTL with rules, that is as expressive as these aforementioned extensions. We then show a similar deficiency in the expressiveness of first-order LTL and present an extension of it with rules, which parallels the propositional version. In our work on runtime verification, we focus on execution traces which consist of events that carry data, where a first-order version of LTL is needed, and in particular on past time versions of first-order LTL. In the previous work, we provided an algorithm for past time first-order LTL that uses BDDs to represent relations over data elements, and implemented it as a tool called DejaVu. In this paper, we propose a monitoring algorithm for the extension of past time first-order LTL with rules. This is implemented as an extension of DejaVu, and experimental results are provided.

UNITY is a model for concurrent specifications with a complete logic for proving progress properties of the form “ P leads to Q ”. UNITY is generalized to U-specifications by giving more freedom to specify the steps that are to be taken infinitely often. In particular, these steps can correspond to non-total relations. The generalization keeps the logic sound and complete. The paper exploits the generalization in two ways. Firstly, the logic remains sound when the specification is extended with hypotheses of the form “ F leads to G ”. As the paper shows, this can make the logic incomplete. The generalization is used to show that the logic remains complete, if the added hypotheses “ F leads to G ” satisfy “ F unless G ”. The main result extends the applicability and completeness of UNITY logic to proofs that a given concurrent program satisfies any given formula of LTL, linear temporal logic, without the next-operator which is omitted because it is sensitive to stuttering. For this purpose, the program, written as a UNITY program, is extended with a number of boolean variables. The proof method relies on implementing the LTL formula, i.e., restricting the specification in such a way that only those runs remain that satisfy the formula. This result is a variation of the classical construction of a Büchi automatonfor a given LTL formula that accepts precisely those runs that satisfy the formula.

This paper introduces ROCS 2.0, a control synthesis tool for nonlinear systems with control objectives given as temporal logic formulas. In addition to the basic invariance, reachability, Buchi, and co-Buchi specifications that can be handled in the previous version of ROCS, ROCS 2.0 provides a major upgrade to support the general class of linear temporal logic formulas that can be translated to deterministic Buchi automata. Moreover, ROCS 2.0 not only maintains and accelerates the kernel of its previous version—the engine based on the specification-guided control method—by more efficient implementation, but also integrates a second engine that implements the abstraction-based control method, which is optimized to gain time and memory efficiency. Such a feature gives the user the freedom to choose the control synthesis method that is more suitable for a specific control problem.

We present a unified translation of linear temporal logic (LTL) formulas into deterministic Rabin automata (DRA), limit-deterministic Büchi automata (LDBA), and nondeterministic Büchi automata (NBA). The translations yield automata of asymptotically optimal size (double or single exponential, respectively). All three translations are derived from one single Master Theorem of purely logical nature. The Master Theorem decomposes the language of a formula into a positive Boolean combination of languages that can be translated into ω-automata by elementary means. In particular, Safra’s, ranking, and breakpoint constructions used in other translations are not needed. We further give evidence that this theoretical clean and compositional approach does not lead to large automata per se and in fact in the case of DRAs yields significantly smaller automata compared to the previously known approach using determinisation of NBAs.

This tool paper describes RGMEDD*, a CTL* model checker that computes the set of states (sat-sets) of a Petri net that satisfy a CTL* formula. The tool can be used as a stand-alone program or from the GreatSPN graphical interface. The tool is based on the decision diagram library Meddly, it uses Spot to translate (sub)formulae into Büchi automata and a variation of the Emerson-Lei algorithm to compute the sat-sets. Correctness has been assessed based on the Model Checking Context 2018 results (for LTL and CTL queries), the sat-set computation of GreatSPN (for CTL) and LTSmin (for LTL), and the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu $$\end{document}-calculus model checker of LTSmin for proper CTL* formulae (using a translator from CTL* to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu $$\end{document}-calculus available in LTSmin). As far as we know, RGMEDD* is the only available Büchi-based CTL* model checker.

We characterize the class of nondeterministic \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega $$\end{document}-automata that can be used for the analysis of finite Markov decision processes (MDPs). We call these automata ‘good-for-MDPs’ (GFM). We show that GFM automata are closed under classic simulation as well as under more powerful simulation relations that leverage properties of optimal control strategies for MDPs. This closure enables us to exploit state-space reduction techniques, such as those based on direct and delayed simulation, that guarantee simulation equivalence. We demonstrate the promise of GFM automata by defining a new class of automata with favorable properties—they are Büchi automata with low branching degree obtained through a simple construction—and show that going beyond limit-deterministic automata may significantly benefit reinforcement learning.

The increasing complexity of embedded systems renders software verification more complex, requiring monitoring and formal techniques, like model-checking. However, to use such techniques, system engineers usually need formal expertise to express the software requirements in a formal language. To facilitate the use of model-checking tools by system engineers, our approach uses a UML model interpreter through which the software requirements can directly be expressed in UML as well. Formal requirements are encoded as UML state machines with the transition guards written in a specific observation language, which expresses predicates on the execution of the system model. Each such executable UML specification can model either a Buchi automaton or an observer automaton, and is synchronously composed with the system, to follow its execution during model-checking. Formal verification can continue at runtime for all deterministic observer automata used during offline verification by deploying them on real embedded systems. Our approach has been evaluated on multiple case studies and is illustrated, in this paper, through the user interface model of a cruise-control system. The automata-based verification results are in line with the verification of the equivalent LTL properties. The runtime overhead during monitoring is proportional to the number of monitors.

A common approach to improve software quality is to use programming guidelines to avoid common kinds of errors. In this paper, we consider the problem of enforcing guidelines for Featherweight Java (FJ). We formalize guidelines as sets of finite or infinite execution traces and develop a region-based type and effect system for FJ that can enforce such guidelines. We build on the work by Erbatur, Hofmann and Z\u{a}linescu, who presented a type system for verifying the finite event traces of terminating FJ programs. We refine this type system, separating region typing from FJ typing, and use ideas of Hofmann and Chen to extend it to capture also infinite traces produced by non-terminating programs. Our type and effect system can express properties of both finite and infinite traces and can compute information about the possible infinite traces of FJ programs. Specifically, the set of infinite traces of a method is constructed as the greatest fixed point of the operator which calculates the possible traces of method bodies. Our type inference algorithm is realized by working with the finitary abstraction of the system based on B\"uchi automata., Comment: main part (14 pages) published at PPDP'21; arXiv version contains an appendix on the FJ operational semantics and the extension to support exception handling (15 pages total)

This research proposes an automatic strategy for planning a team of identical robots evolving in a known environment. The robots should satisfy a global task for the whole team, given in terms of a Linear Temporal Logic (LTL) formula over predefined regions of interest. A Robot Motion Petri Net (RMPN) system is used for modeling the evolution of the robotic team in the environment, bringing the advantage of a fixed topology versus the number of robots, with respect to methods based on synchronous automaton products. The algorithmic method iterates the selection of an accepted run that satisfies the specification and the search for RMPN sequences of reachable markings that can produce desired observations. A Buchi automaton witnesses the advancement towards formula fulfillment, and at the core of our methods are three Mixed Integer Linear Programming (MILP) formulations that yield firing sequences and markings of RMPN model. The cost functions of these formulations reduce the number of robot synchronizations and induce collision avoidance. Simulation examples support the computational feasibility of the proposed method.

One of the most simple models of computation suitable for representation of reactive systems behavior is a finite state transducer that operates on an input alphabet of control signals and an output alphabet of basic actions. The behavior of such a reactive system displays itself in the correspondence between the flow of control signals and sequence of basic actions performed by the system. We believe that the behavior of this kind requires more complex and expressive means of specifications than the conventional linear temporal logic ($$LTL$$). In this paper, we define a new language of formal specifications $$\mathcal{L}\mathcal{P}$$-$$LTL$$, which is an extension of LTL, specifically intended for describing the properties of transducer computations. In this extension, the temporal operators are parameterized by sets of words (languages) which describe flows of control signals that impact on a reactive system. Basic predicates in the ℒ𝒫-LTL formulas are also defined by the languages in the alphabet of basic actions; these languages describe the expected response of the system to the environmental influences. In our earlier papers, we considered a verification problem for finite state transducers with regard to specifications represented by the ℒ𝒫-$$LTL$$ and ℒ𝒫-$$CTL$$ formulas. It was shown that the problem is algorithmically solvable for both logics. The aim of this paper is to estimate the expressive power of ℒ𝒫-$$LTL$$ by comparing it with some well-known logics widely used in computer science for specification of reactive systems. Two fragments were isolated in the ℒ𝒫-LTL: ℒ𝒫-1-LTL and ℒ𝒫-n-LTL. We discovered that the specification language of ℒ𝒫-1-$$LTL$$ is more expressive than LTL, while the ℒ𝒫-n-$$LTL$$ fragment has the same expressive power as the monadic second order logic S1S. temporal logics, expressive power, specification, verification, Buchi automata, infinite words

To specify properties of open systems with interval based temporal logics, Alternating Projection Temporal Logic (APTL) is proposed by introducing Concurrent Game Structures (CGS) to Propositional Projection Temporal Logic (PPTL). Further, examples are given to show how properties of open systems can be specified by APTL formulas. Moreover, to establish the automata based model theory for the new proposed logic, Generalized alternating Buchi automata over Concurrent Game structures (GBCGs) are defined, and a transformation from APTL formulas to GBCGs is presented. In addition, a decision procedure for checking the satisfiability of APTL formulas, and a model checking approach for APTL with Concurrent Game Structures (CGSs) being models are presented.

The translation of LTL formula into equivalent Buchi automata plays an important role in many algorithms for LTL model checking, which consist in obtaining a Buchi automaton that is equivalent to the software system specification and another one that is equivalent to the negation of the property. The intersection of the two Buchi automata is empty if the model satisfies the property. Generating the Buchi automaton corresponding to an LTL formula may, in the worst case, be exponential in the size of the formula, making the model checking effort exponential in the size of the original formula. There is no polynomial solution for checking emptiness of the intersection. That comes from the translation step of LTL formula into finite state models. This makes verification methods hard or even impossible to be implemented in practice. In this paper, we propose a subset of LTL formula which can be converted to Buchi automata whose the size is polynomia.

Drawing on business rules for constructing business process models by a constraint-driven methodology is a distinct characteristic of declarative process modeling. Given the intricacies of business rules, there is a pragmatic need to conduct conflict-free assessments for business rules in an automatic manner. In this paper, business rules are stated in terms of restricted English by harnessing a group of predefined business rule templates. With linear temporal logic that serves as a semantic foundation for the business rule templates, a pair of business rules represented as a linear temporal logic specification is translated into an associated Büchi automaton via LTL2BA, LTL3BA and ltl2tgba. A Büchi automaton that accepts the empty language signifies that the two business rules are in conflict with each other. The suitability of the formal framework and the three automated tools is evaluated by an industry-level case study.

Inspired by the results of finite automata working on infinite words, we studied the quantum ω-automata with Buchi, Muller, Rabin and Streett acceptance condition. Quantum finite automata play a pivotal part in quantum information and computational theory. Investigation of the power of quantum finite automata over infinite words is a natural goal. We have investigated the classes of quantum ω-automata from two aspects: the language recognition and their closure properties. It has been shown that quantum Muller automaton is more dominant than quantum Buchi automaton. Furthermore, we have demonstrated the languages recognized by one-way quantum finite automata with different quantum acceptance conditions. Finally, we have proved the closure properties of quantum ω-automata.

International audience; We propose a new algorithm for checking language equivalence of non-deterministic Büchi automata. We start from a construction proposed by Calbrix, Nivat and Podelski, which makes it possible to reduce the problem to that of checking equivalence of automata on finite words. Although this construction generates large and highly non-deterministic automata, we show how to exploit their specific structure and apply state-of-the art techniques based on coinduction to reduce the state-space that has to be explored. Doing so, we obtain algorithms which do not require full determinisation or complementation.

This paper presents a model-free reinforcement learning (RL) algorithm to synthesize a control policy that maximizes the satisfaction probability of linear temporal logic (LTL) specifications. Due to the consideration of environment and motion uncertainties, we model the robot motion as a probabilistic labeled Markov decision process with unknown transition probabilities and unknown probabilistic label functions. The LTL task specification is converted to a limit deterministic generalized B\"uchi automaton (LDGBA) with several accepting sets to maintain dense rewards during learning. The novelty of applying LDGBA is to construct an embedded LDGBA (E-LDGBA) by designing a synchronous tracking-frontier function, which enables the record of non-visited accepting sets without increasing dimensional and computational complexity. With appropriate dependent reward and discount functions, rigorous analysis shows that any method that optimizes the expected discount return of the RL-based approach is guaranteed to find the optimal policy that maximizes the satisfaction probability of the LTL specifications. A model-free RL-based motion planning strategy is developed to generate the optimal policy in this paper. The effectiveness of the RL-based control synthesis is demonstrated via simulation and experimental results., Comment: arXiv admin note: text overlap with arXiv:2109.02791

This paper considers global exponential stabilization (GES) of switched linear discrete-time system under language constraint which is generated by B chi automata. Firstly, the characteristics of the acceptable conditions of B chi automata are considered, which are used to generate the re-entry constrained switching sequences, and the global exponential stability of B chi automata is proposed. Then, by combining multiple Lyapunov functions with the initial state and acceptable state of B chi automata, a sufficient condition for the global exponential stability of the system with step decreasing Lyapunov functions is given. This condition can also be used to study the stability of Muller automata constrained switched systems.

We present a formal verification method that provides a model-based approach to human-robot interaction (HRI) in medical settings by utilizing linear temporal logic (LTL). We define high-level HRI procedures with an LTL-based framework to create algorithmically sound robots that can function independently in dynamic HRI environments. Our approach's theoretical infallibility confers particular advantages for medical robots, where safety and informative communications are crucial. In order to establish the viability of our proposed method, and with the ongoing COVID-19 pandemic in mind, we developed an LTL knowledge base for a medical robot tasked with HRI-intensive roles of patient reception and triage. We designed robotic simulations based on our LTL architecture to test our approach, employing randomized inputs to generate unpredictable HRI environments. We then conducted formal verification via an automata-theoretic approach by evaluating our simulated robot against generalized Buchi automata. We hope our LTL-based approach can enable future achievements in HRI.

This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) with unknown transition probabilities over continuous state and action spaces. Linear temporal logic (LTL) is used to specify high-level tasks over infinite horizon, which can be converted into a limit deterministic generalized B\"uchi automaton (LDGBA) with several accepting sets. The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP by incorporating a synchronous tracking-frontier function to record unvisited accepting sets of the automaton, and to facilitate the satisfaction of the accepting conditions. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states and can overcome the issues of sparse rewards. Rigorous analysis shows that any RL method that optimizes the expected discounted return is guaranteed to find an optimal policy whose traces maximize the satisfaction probability. A modular deep deterministic policy gradient (DDPG) is then developed to generate such policies over continuous state and action spaces. The performance of our framework is evaluated via an array of OpenAI gym environments., Comment: arXiv admin note: text overlap with arXiv:2010.06797

Checking the soundness of cyclic induction reasoning for first-order logic with inductive definitions (FOLID) is decidable but the standard checking method is based on an exponential complement operation for B\"uchi automata. Recently, we introduced a polynomial checking method whose most expensive steps recall the comparisons done with multiset path orderings. We describe the implementation of our method in the Cyclist prover. Referred to as E-Cyclist, it successfully checked all the proofs included in the original distribution of Cyclist. Heuristics have been devised to automatically define, from the analysis of the proof derivations, the trace-based ordering measures that guarantee the soundness property., Comment: In Proceedings SCSS 2021, arXiv:2109.02501

Hyperproperties generalize trace properties by expressing relations between multiple computations. Hyperpropertes include policies from information-flow security, like observational determinism or non-interference, and many other system properties including promptness and knowledge. In this paper, we give an overview on the model checking problem for temporal hyperlogics. Our starting point is the model checking algorithm for HyperLTL, a reduction to Buchi automata emptiness. This basic construction can be extended with propositional quantification, resulting in an algorithm for HyperQPTL. It can also be extended with branching time, resulting in an algorithm for HyperCTL*. However, it is not possible to have both extensions at the same time: the model checking problem of HyperQCTL* is undecidable. An attractive compromise is offered by MPL[E], i.e., monadic path logic extended with the equal-level predicate. The expressiveness of MPL[E] falls strictly between that of HyperCTL* and HyperQCTL*. MPL[E] subsumes both HyperCTL* and HyperKCTL*, the extension of HyperCTL* with the knowledge operator. We show that the model checking problem for MPL[E] is still decidable.

Decoupled search is a state space search method originally introduced in AI Planning. Similar to partial-order reduction methods, decoupled search exploits the independence of components to tackle the state explosion problem. Similar to symbolic representations, it does not construct the explicit state space, but sets of states are represented in a compact manner, exploiting component independence. Given the success of both partial-order reduction and symbolic representations when model checking liveness properties, our goal is to add decoupled search to the toolset of liveness checking methods. Specifically, we show how decoupled search can be applied to liveness verification for composed Büchi automata by adapting, and showing correct, a standard algorithm for detecting lassos (i.e., infinite accepting runs), namely nested depth-first search. We evaluate our approach using a prototype implementation.

We study the problem of determinizing \(\omega \)-automata whose acceptance condition is defined on the transitions using Boolean formulas, also known as transition-based Emerson-Lei automata (TELA). The standard approach to determinize TELA first constructs an equivalent generalized Buchi automaton (GBA), which is later determinized. We introduce three new ways of translating TELA to GBA. Furthermore, we give a new determinization construction which determinizes several GBA separately and combines them using a product construction. An experimental evaluation shows that the product approach is competitive when compared with state-of-the-art determinization procedures. We also study limit-determinization of TELA and show that this can be done with a single-exponential blow-up, in contrast to the known double-exponential lower-bound for determinization. Finally, one version of the limit-determinization procedure yields good-for-MDP automata which can be used for quantitative probabilistic model checking.

This paper investigates an unified model checking procedure for Alternating Projection Temporal Logic (APTL) based on automaton. The method for transforming an APTL formula to an alternating Buchi automaton over Concurrent Game structure (BCG) has been illustrated. The algorithm for the product of two BCGs is presented in this paper. In addition, the APTL model checking algorithm for open system is given and examples are also presented to illustrate how the procedure works.

In this paper, we investigate the problem of planning an optimal infinite path for a single robot to achieve a linear temporal logic (LTL) task with security guarantee. We assume that the external behavior of the robot, specified by an output function, can be accessed by a passive intruder (eavesdropper). The security constraint requires that the intruder should never infer that the robot was started from a secret location. We provide a sound and complete algorithmic procedure to solve this problem. Our approach is based on the construction of the twin weighted transition systems (twin-WTS) that tracks a pair of paths having the same observation. We show that the security-aware path planning problem can be effectively solved based on graph search techniques in the product of the twin-WTS and the B\"{u}chi automaton representing the LTL formula. The complexity of the proposed planning algorithm is polynomial in the size of the system model. Finally, we illustrate our algorithm by a simple robot planning example., Comment: This work has been accepted by 59th IEEE Conference on Decision and Control for publication

Omega-regular properties—specified using linear time temporal logic or various forms of omega-automata—find increasing use in specifying the objectives of reinforcement learning (RL). The key problem that arises is that of faithful and effective translation of the objective into a scalar reward for model-free RL. A recent approach exploits Buchi automata with restricted nondeterminism to reduce the search for an optimal policy for an Open image in new window-regular property to that for a simple reachability objective. A possible drawback of this translation is that reachability rewards are sparse, being reaped only at the end of each episode. Another approach reduces the search for an optimal policy to an optimization problem with two interdependent discount parameters. While this approach provides denser rewards than the reduction to reachability, it is not easily mapped to off-the-shelf RL algorithms. We propose a reward scheme that reduces the search for an optimal policy to an optimization problem with a single discount parameter that produces dense rewards and is compatible with off-the-shelf RL algorithms. Finally, we report an experimental comparison of these and other reward schemes for model-free RL with omega-regular objectives.

An automaton is unambiguous if for every input it has at most one accepting computation. An automaton is k-ambiguous (for k > 0) if for every input it has at most k accepting computations. An automaton is boundedly ambiguous if there is k ∈ ℕ, such that for every input it has at most k accepting computations. An automaton is finitely (respectively, countably) ambiguous if for every input it has at most finitely (respectively, countably) many accepting computations. The degree of ambiguity of a regular language is defined in a natural way. A language is k-ambiguous (respectively, boundedly, finitely, countably ambiguous) if it is accepted by a k-ambiguous (respectively, boundedly, finitely, countably ambiguous) automaton. Over finite words every regular language is accepted by a deterministic automaton. Over finite trees every regular language is accepted by an unambiguous automaton. Over ω-words every regular language is accepted by an unambiguous Büchi automaton [Arnold, 1983] and by a deterministic parity automaton. Over infinite trees there are ambiguous languages [Carayol et al., 2010]. We show that over infinite trees there is a hierarchy of degrees of ambiguity: For every k > 1 there are k-ambiguous languages which are not k-1 ambiguous; there are finitely (respectively countably, uncountably) ambiguous languages which are not boundedly (respectively finitely, countably) ambiguous.

We present planning and control techniques for non-periodic locomotion tasks specified by temporal logic in rough cluttered terrains. Our planning approach is based on a discrete set of motion primitives for the center of mass (CoM) of a general bipedal robot model. A deterministic shortest path problem is solved over the Buchi automaton of the temporal logic task specification, composed with the graph of CoM keyframe states generated by the motion primitives. A low- level controller based on quadratic programming is proposed to track the resulting CoM and foot trajectories. We demonstrate dynamically stable, non-periodic locomotion of a kneed compass gait bipedal robot satisfying complex task specifications.

This letter proposes a novel reinforcement learning method for the synthesis of a control policy satisfying a control specification described by a linear temporal logic formula. We assume that the controlled system is modeled by a Markov decision process (MDP). We convert the specification to a limit-deterministic generalized B\"uchi automaton (LDGBA) with several accepting sets that accepts all infinite sequences satisfying the formula. The LDGBA is augmented so that it explicitly records the previous visits to accepting sets. We take a product of the augmented LDGBA and the MDP, based on which we define a reward function. The agent gets rewards whenever state transitions are in an accepting set that has not been visited for a certain number of steps. Consequently, sparsity of rewards is relaxed and optimal circulations among the accepting sets are learned. We show that the proposed method can learn an optimal policy when the discount factor is sufficiently close to one., Comment: 7 pages, 6 figures; an extended version of a manuscript accepted to IEEE L-CSS

We prove that ω-regular languages accepted by Buchi or Muller automata satisfy an effective automata-theoretic version of the Baire property. Then we use this result to obtain a new effective property of rational functions over infinite words which are realized by finite state Buchi transducers: for each such function F : Σ^ω → Γ^ω , one can construct a deterministic Buchi automaton A accepting a dense Π^0_2-subset of Σ^ω such that the restriction of F to L(A) is continuous.

In this paper, we provide a compositional framework for synthesizing hybrid controllers for interconnected discrete-time control systems enforcing specifications expressed by co-Buchi automata. In particular, we first decompose the given specification to simpler reachability tasks based on automata representing the complements of original co-Buchi automata. Then, we provide a systematic approach to solve those simpler reachability tasks by computing corresponding control barrier functions. We show that such control barrier functions can be constructed compositionally by assuming some small-gain type conditions and composing so-called local control barrier functions computed for subsystems. We provide two systematic techniques to search for local control barrier functions for subsystems based on the sum-of-squares optimization program and counter-example guided inductive synthesis approach. Finally, we illustrate the effectiveness of our results through two large-scale case studies., Comment: 18 pages, 10 figures, HSCC 2020

The combination of data-driven learning methods with formal reasoning has seen a surge of interest, as either area has the potential to bolstering the other. For instance, formal methods promise to expand the use of state-of-the-art learning approaches in the direction of certification and sample efficiency. In this work, we propose a deep Reinforcement Learning (RL) method for policy synthesis in continuous-state/action unknown environments, under requirements expressed in Linear Temporal Logic (LTL). We show that this combination lifts the applicability of deep RL to complex temporal and memory-dependent policy synthesis goals. We express an LTL specification as a Limit Deterministic Buchi Automaton (LDBA) and synchronise it on-the-fly with the agent/environment. The LDBA in practice monitors the environment, acting as a modular reward machine for the agent: accordingly, a modular Deep Deterministic Policy Gradient (DDPG) architecture is proposed to generate a low-level control policy that maximises the probability of the given LTL formula. We evaluate our framework in a cart-pole example and in a Mars rover experiment, where we achieve near-perfect success rates, while baselines based on standard RL are shown to fail in practice.

The search for a proof of correctness and the search for counterexamples (bugs) are complementary aspects of verification. In order to maximize the practical use of verification tools it is better to pursue them at the same time. While this is well-understood in the termination analysis of programs, this is not the case for the language inclusion analysis of Buchi automata, where research mainly focused on improving algorithms for proving language inclusion, with the search for counterexamples left to the expensive complementation operation.

Probabilistic Büchi automata are a natural generalization of PFA to infinite words, but have been studied in-depth only rather recently and many interesting questions are still open. PBA are known to accept, in general, a class of languages that goes beyond the regular languages. In this work we extend the known classes of restricted PBA which are still regular, strongly relying on notions concerning ambiguity in classical $$\omega $$ ω -automata. Furthermore, we investigate the expressivity of the not yet considered but natural class of weak PBA, and we also show that the regularity problem for weak PBA is undecidable.

We develop a complementation procedure and an equivalence checker for nondeterministic Buchi automata. Both are formally verified using the proof assistant Isabelle/Hol. The verification covers everything from the abstract correctness proof down to the generated Sml code.

We present a new approach to conformance testing of black-box reactive systems. We consider system specifications written as linear temporal logic formulas to generate tests as sequences of input/output pairs: inputs are extracted from the Buchi automata corresponding to the specifications, and outputs are obtained by feeding the inputs to the systems. Conformance is checked by comparing input/output sequences with automata traces to detect violations of the specifications. We consider several criteria for extracting tests and for stopping generation, and we compare them experimentally using both indicators of coverage and error-detection. The results show that our methodology can generate test suites with good system coverage and error-detection capability.