Search

In this paper, we address the problem of steering a team of agents under stochastic linear dynamics to prescribed final state means and covariances. The agents operate in a common environment where inter-agent constraints may also be present. In order for our method to be scalable to large-scale systems and computationally efficient, we approach the problem in a distributed control framework using the Alternating Direction Method of Multipliers (ADMM). Each agent solves its own covariance steering problem in parallel, while additional copy variables for its closest neighbors are introduced to ensure that the inter-agent constraints will be satisfied. The inclusion of these additional variables creates a requirement for consensus between original and copy variables that involve the same agent. For this reason, we employ a variation of ADMM for consensus optimization. Simulation results on multi-vehicle systems under uncertainty with collision avoidance constraints illustrate the effectiveness of our algorithm. The substantially improved scalability of our distributed approach with respect to the number of agents is also demonstrated, in comparison with an equivalent centralized scheme.

In this paper, we discuss the methodology of generalizing the optimal control law from learned component tasks to unlearned composite tasks on Multi-Agent Systems (MASs), by using the linearity composition principle of linearly solvable optimal control (LSOC) problems. The proposed approach achieves both the compositionality and optimality of control actions simultaneously within the cooperative MAS framework in both discrete- and continuous-time in a sample-efficient manner, which reduces the burden of re-computation of the optimal control solutions for the new task on the MASs. We investigate the application of the proposed approach on the MAS with coordination between agents. The experiments show feasible results in investigated scenarios, including both discrete and continuous dynamical systems for task generalization without resampling., Accepted to the 2021 American Control Conference (ACC)

In this paper we develop a novel optimal control framework for uncertain mechanical systems. Our work extends differential dynamic programming and handles uncertainty through generalized polynomial...

We present a trajectory optimization approach to reinforcement learning in continuous state and action spaces, called probabilistic differential dynamic programming (PDDP). Our method represents systems dynamics using Gaussian processes (GPs), and performs local dynamic programming iteratively around a nominal trajectory in Gaussian belief spaces. Different from model-based policy search methods, PDDP does not require a policy parameterization and learns a time-varying control policy via successive forward-backward sweeps. A convergence analysis of the iterative scheme is given, showing that our algorithm converges to a stationary point globally under certain conditions. We show that prior model knowledge can be incorporated into the proposed framework to speed up learning, and a generalized optimization criterion based on the predicted cost distribution can be employed to enable risk-sensitive learning. We demonstrate the effectiveness and efficiency of the proposed algorithm using nontrivial tasks. Compared with a state-of-the-art GP-based policy search method, PDDP offers a superior combination of learning speed, data efficiency, and applicability.

A distributed stochastic optimal control solution is presented for cooperative multi-agent systems. The network of agents is partitioned into multiple factorial subsystems, each of which consists of a central agent and neighboring agents. Local control actions that rely only on agents' local observations are designed to optimize the joint cost functions of subsystems. When solving for the local control actions, the joint optimality equation for each subsystem is cast as a linear partial differential equation and solved using the Feynman-Kac formula. The solution and the optimal control action are then formulated as path integrals and approximated by a Monte-Carlo method. Numerical verification is provided through a simulation example consisting of a team of cooperative UAVs., To appear in American Control Conference 2021, New Orleans, LA, USA

There is a rising interest in Spatio-temporal systems described by Partial Differential Equations (PDEs) among the control community. Not only are these systems challenging to control, but the sizing and placement of their actuation is an NP-hard problem on its own. Recent methods either discretize the space before optimziation, or apply tools from linear systems theory under restrictive linearity assumptions. In this work we consider control and actuator placement as a coupled optimization problem, and derive an optimization algorithm on Hilbert spaces for nonlinear PDEs with an additive spatio-temporal description of white noise. We study first and second order systems and in doing so, extend several results to the case of second order PDEs. The described approach is based on variational optimization, and performs joint RL-type optimization of the feedback control law and the actuator design over episodes. We demonstrate the efficacy of the proposed approach with several simulated experiments on a variety of SPDEs., 19 pages, 5 figures

Learning-based control aims to construct models of a system to use for planning or trajectory optimization, e.g. in model-based reinforcement learning. In order to obtain guarantees of safety in this context, uncertainty must be accurately quantified. This uncertainty may come from errors in learning (due to a lack of data, for example), or may be inherent to the system. Propagating uncertainty forward in learned dynamics models is a difficult problem. In this work we use deep learning to obtain expressive and flexible models of how distributions of trajectories behave, which we then use for nonlinear Model Predictive Control (MPC). We introduce a deep quantile regression framework for control that enforces probabilistic quantile bounds and quantifies epistemic uncertainty. Using our method we explore three different approaches for learning tubes that contain the possible trajectories of the system, and demonstrate how to use each of them in a Tube MPC scheme. We prove these schemes are recursively feasible and satisfy constraints with a desired margin of probability. We present experiments in simulation on a nonlinear quadrotor system, demonstrating the practical efficacy of these ideas., RSS 2020 Camera Ready Version

We consider covariance control problems for nonlinear stochastic systems. Our objective is to find an optimal control strategy to steer the state from an initial distribution to a terminal one with specified mean and covariance. This problem is considerably more complicated than previous studies on covariance control for linear systems. We leverage a widely used technique - differential dynamic programming - in nonlinear optimal control to achieve our goal. In particular, we adopt the stochastic differential dynamic programming framework to handle the stochastic dynamics. Additionally, to enforce the terminal statistical constraints, we construct a Lagrangian and apply a primal-dual type algorithm. Several examples are presented to demonstrate the effectiveness of our framework., 7 pages, 5 figures

Differential Dynamic Programming (DDP) has become a well established method for unconstrained trajectory optimization. Despite its several applications in robotics and controls however, a widely successful constrained version of the algorithm has yet to be developed. This paper builds upon penalty methods and active-set approaches, towards designing a Dynamic Programming-based methodology for constrained optimal control. Regarding the former, our derivation employs a constrained version of Bellman's principle of optimality, by introducing a set of auxiliary slack variables in the backward pass. In parallel, we show how Augmented Lagrangian methods can be naturally incorporated within DDP, by utilizing a particular set of penalty-Lagrangian functions that preserve second-order differentiability. We demonstrate experimentally that our extensions (individually and combinations thereof) enhance significantly the convergence properties of the algorithm, and outperform previous approaches on a large number of simulated scenarios., 13 pages, 9 figures

The aim of this work is to present a sampling-based algorithm designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). Specifically, we address the class of problems in which the running cost of the performance index involves an L 1 -type minimization problem in terms of the control effort. Such problems are typically called minimum fuel problems in optimal control literature. By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques.

The aim of this work is to present a novel sampling-based numerical scheme designed to solve a certain class of stochastic optimal control problems, utilizing forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman–Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton–Jacobi–Bellman equation, expressed in the form of a system of decoupled FBSDEs. This system of FBSDEs can be solved by employing linear regression techniques. The proposed framework relaxes some of the restrictive conditions present in recent sampling based methods within the Linearly Solvable Optimal Control framework, and furthermore addresses problems in which the time horizon is not prespecified. To enhance the efficiency of the proposed scheme when treating more complex nonlinear systems, we then derive an iterative algorithm based on Girsanov’s theorem on the change of measure, which features importance sampling. This scheme is shown to be capable of learning the optimal control without requiring an initial guess.

This paper presents a real-time implementable trajectory optimization framework for autonomous suspended load operations in outdoor environments. The framework solves the posed optimal control problem with the iteration-based differential dynamic programming algorithm. The algorithm uses a variational integrator to propagate the modeled system’s state configuration and linearize the resulting discrete dynamics. The variational integrator is an excellent candidate for real-time implementation because it remains accurate despite relatively large discretization time steps. Therefore, the computational effort of the differential dynamic programming algorithm can be mitigated through the reduction of discrete time points. The state of the slung load is estimated via an augmentation to the existing navigation system that only uses vision-based measurements of the load. Simulation studies and a flight test are presented to demonstrate the effectiveness of the proposed framework.

In this paper, a model predictive path integral control algorithm based on a generalized importance sampling scheme is developed and parallel optimization via sampling is performed using a graphics...

In this paper we propose a new methodology for decision-making under uncertainty using recent advancements in the areas of nonlinear stochastic optimal control theory, applied mathematics, and machine learning. Grounded on the fundamental relation between certain nonlinear partial differential equations and forward-backward stochastic differential equations, we develop a control framework that is scalable and applicable to general classes of stochastic systems and decision-making problem formulations in robotics and autonomy. The proposed deep neural network architectures for stochastic control consist of recurrent and fully connected layers. The performance and scalability of the aforementioned algorithm are investigated in three non-linear systems in simulation with and without control constraints. We conclude with a discussion on future directions and their implications to robotics.

We propose a sampling-based trajectory optimization methodology for constrained problems. We extend recent works on stochastic search to deal with box control constraints, as well as nonlinear state constraints for discrete dynamical systems. Regarding the former, our strategy is to optimize over truncated parameterized distributions on control inputs. Furthermore, we show how non-smooth penalty functions can be incorporated into our framework to handle state constraints. Simulations on cartpole and quadcopter show that our approach outperforms previous methods on constrained sampling-based optimization, in terms of quality of solutions and convergence speed.

Rapidly Exploring Random Trees (RRTs) have gained significant attention due to provable properties such as completeness and asymptotic optimality. However, offline methods are only useful when the entire problem landscape is known a priori. Furthermore, many real world applications have problem scopes that are orders of magnitude larger than typical mazes and bug traps that require large numbers of samples to match typical sample densities, resulting in high computational effort for reasonably low-cost trajectories. In this paper we propose an online trajectory optimization algorithm for uncertain large environments using RRTs, which we call Locally Adaptive Rapidly Exploring Random Tree (LARRT). This is achieved through two main contributions. We use an adaptive local sampling region and adaptive sampling scheme which depend on states of the dynamic system and observations of obstacles. We also propose a localized approach to planning and re-planning through fixing the root node to the current vehicle state and adding tree update functions. LARRT is designed to leverage local problem scope to reduce computational complexity and obtain a total lower-cost solution compared to a classical RRT of a similar number of nodes. Using this technique we can ensure that popular variants of RRT will remain online even for prohibitively large planning problems by transforming a large trajectory optimization approach to one that resembles receding horizon optimization. Finally, we demonstrate our approach in simulation and discuss various algorithmic trade-offs of the proposed approach.

In this paper we present an information theoretic approach to stochastic optimal control problems for systems with compound Poisson noise. We generalize previous work on information theoretic path integral control to discontinuous dynamics with compound Poisson noise. We also derive a control update law of the same form using a stochastic optimization approach. We develop a sampling-based iterative model predictive control (MPC) algorithm. The proposed algorithm is parallelizable and when implemented on a Graphical Processing Unit (GPU) can run in real time. We test the performance of the proposed algorithm in simulation for two control tasks using a cartpole and a quadrotor system. Our simulations demonstrate improved performance of the new scheme and indicate the importance of incorporating the statistical characteristics of stochastic disturbances in the computation of the stochastic optimal control policies.

Empirically derived continuum models of collective behavior among large populations of dynamic agents are a subject of intense study in several fields, including biology, engineering and finance. We formulate and study a mean-field game model whose behavior mimics an empirically derived non-local homogeneous flocking model for agents with gradient self-propulsion dynamics. The mean-field game framework provides a non-cooperative optimal control description of the behavior of a population of agents in a distributed setting. In this description, each agent's state is driven by optimally controlled dynamics that result in a Nash equilibrium between itself and the population. The optimal control is computed by minimizing a cost that depends only on its own state, and a mean-field term. The agent distribution in phase space evolves under the optimal feedback control policy. We exploit the low-rank perturbative nature of the non-local term in the forward-backward system of equations governing the state and control distributions, and provide a linear stability analysis demonstrating that our model exhibits bifurcations similar to those found in the empirical model. The present work is a step towards developing a set of tools for systematic analysis, and eventually design, of collective behavior of non-cooperative dynamic agents via an inverse modeling approach., Comment: Revised to incorporate reviewers' suggestions. Accepted to Chaos journal

Recent work from the reinforcement learning community has shown that Evolution Strategies are a fast and scalable alternative to other reinforcement learning methods. In this paper we show that Evolution Strategies are a special case of model-based stochastic search methods. This class of algorithms has nice asymptotic convergence properties and known convergence rates. We show how these methods can be used to solve both cooperative and competitive multi-agent problems in an efficient manner. We demonstrate the effectiveness of this approach on two complex multi-agent UAV swarm combat scenarios: where a team of fixed wing aircraft must attack a well-defended base, and where two teams of agents go head to head to defeat each other., Video at http://goo.gl/dWvQi7 Code freely available at http://github.com/ddfan/swarm_evolve

Trajectory optimization of a controlled dynamical system is an essential part of autonomy, however many trajectory optimization techniques are limited by the fidelity of the underlying parametric model. In the field of robotics, a lack of model knowledge can be overcome with machine learning techniques by utilizing measurements to build a dynamical model from the data. This paper aims to take the middle ground between these two approaches by introducing a semi-parametric representation of the underlying system dynamics. Our goal is to leverage the considerable information contained in a traditional physics based model and combine it with a data-driven, non-parametric regression technique known as a Gaussian Process. Integrating this semi-parametric model with PseudoSpectral Optimal Control (PSOC), we demonstrate model learning in an episodic and receding horizon fashion. In order to manage parametric uncertainty, we introduce an algorithm that utilizes Sparse Spectrum Gaussian Processes (SSGP) for incremental learning after each rollout. The goal of this paper is to motivate and demonstrate the constrained optimization techniques with semi-parametric models for online learning.

In this paper, we present connections between recent developments on the linearly-solvable stochastic optimal control framework with early work in control theory based on the fundamental dualities between free energy and relative entropy. We extend these connections to nonlinear stochastic systems with non-affine controls by using the generalized version of the Feynman–Kac lemma. We present alternative formulations of the linearly-solvable stochastic optimal control framework and discuss information theoretic and thermodynamic interpretations. On the algorithmic side, we present iterative stochastic optimal control algorithms and applications to nonlinear stochastic systems. We conclude with an overview of the frameworks presented and discuss limitations, differences and future directions.

Recent advances in sampling-based Model Predictive Control (MPC) methods have enabled the control of nonlinear stochastic dynamical systems with complex and non-smooth cost functions. However, the main drawback of these methods is that they can be myopic with respect to high-level tasks, since MPC relies on predicting dynamics within a short time horizon. Furthermore, designing cost functions which capture high-level information may be prohibitive for complex tasks, especially multi-agent scenarios. Here we propose a hierarchical approach to this problem where the NeuroEvolution of Augmenting Topologies (NEAT) algorithm is used to build cost functions for an MPC trajectory optimization algorithm known as Model-Predictive Path Integral (MPPI) control. MPPI and NEAT are particularly well-suited to one another since MPPI can control an agent in a way that minimizes a non-differentiable cost function (including logic or non-smooth functions), while NEAT can build a neural network comprised of any arbitrary activation functions, including those which are non-differentiable or logic-based. We utilize this approach in controlling agile swarms of unmanned aerial vehicles (UAVs) in a simulated swarm vs. swarm combat scenario.

The Hamilton Jacobi Bellman (HJB) PDE for the stochastic optimal control (SOC) problem for diffusion SDE dynamics which have affine controls and state and control multiplicative noise is a second order fully nonlinear PDE. The previously known linearly solvable optimal control framework as well as the first order forward backward SDEs (FBSDEs) frameworks therefore have a characteristic inadequacy to support sampling algorithms for this SOC problem. We present the framework of second order FBSDEs for solving this SOC problem in this paper. We derive the nonlinear Feynman Kac representation of the second order fully nonlinear HJB PDE corresponding to diffusions with state and control multiplicative noise. The Feynman Kac representation enables a sampling based scheme for solving this SOC problem. This scheme is then leveraged to develop a least squares Monte Carlo regression based algorithm for implementations. The algorithm is validated by examples of simulated control of an underactuated system and comparison against an analytical characterization.

We present a sampling-based stochastic optimal control (SOC) framework for systems with unknown dynamics based on the path integral formulation and probabilistic inference. This work is motivated by three major limitations of related SOC methods: first, full knowledge of the dynamics model is usually required. Second, model uncertainty is neglected. Third, convergence of the iterative scheme is quite slow. In order to cope with these issues, our method performs sampling in belief space using approximate inference in probabilistic models. When performing probability-weighted averaging, each sample is weighted by its predictive uncertainty. In addition, our method leverages covariance matrix adaptation to achieve faster convergence. We demonstrate the effectiveness and efficiency of the proposed method using a simulated cart-pole swing up task.

We propose a novel methodology for stochastic trajectory optimization which is based on merging the theory of spectral expansions with Differential Dynamic Programming. Specifically, we employ polynomial chaos expansions to handle parametric uncertainties and utilize the Karhunen-Loeve transformation to represent stochastic forces. This allows us to build a generic framework and avoid relying on restrictive assumptions regarding the form of stochasticity. In addition, Differential Dynamic Programming provides an iterative algorithm for finding optimal controls which attains scalability and, under mild assumptions, fast convergence. The obtained method is capable of controlling the distribution of the system's trajectory and can be used in both planning and control. Last but not least, we formulate a variational integration scheme for uncertain systems that are represented by spectral expansions. Simulated examples validate the applicability and numerical efficiency of the proposed approach.

In this work we present a sampling-based algorithm designed to solve game-theoretic control problems and risk-sensitive stochastic optimal control problems. The cornerstone of the proposed approach is the formulation of the problem in terms of forward and backward stochastic differential equations (FBSDEs). By means of a nonlinear version of the Feynman-Kac lemma, we obtain a probabilistic representation of the solution to the nonlinear Hamilton-Jacobi-Isaacs equation, expressed in the form of a decoupled system of FBSDEs. This system of FBSDEs can then be simulated by employing linear regression techniques. Utilizing the connection between stochastic differential games and risk-sensitive optimal control, we demonstrate that the proposed algorithm is also applicable to the latter class of problems. Simulation results validate the algorithm.

A Stochastic Game Theoretic Differential Dynamic Programming (SGT-DDP) algorithm is derived to solve a differential game under stochastic dynamics. We present the update law for the minimizing and maximizing controls for both players and provide a set of backward differential equations for the second order value function approximation. We compute the extra terms in the backward propagation equations that arise from the stochastic assumption compared with the original GTDDP. We present the SGT-DDP algorithm and analyze how the design of the cost function affects the feed-forward and feedback parts of the control policies under the game theoretic formulation. The performance of SGT-DDP is then investigated through simulations on two examples, namely, a first order nonlinear system, the inverted pendulum and the cart pole problems with conflicting controls. We conclude with some possible future extensions.

We present the infinite dimensional approach to control of a general class of doubly stochastic or otherwise known Q-mark Markov Jump Diffusion (Q-MJD) processes. The governing dynamics for the the probability density function (PDF) of this class of Q-MJD processes is a Partial Integro Differential Equation (PIDE). The infinite dimensional Minimum Principle (MP) is applied to control these PIDE dynamics. We qualitatively compare the infinite dimensional MP and the stochastic Dynamic Programming Principle (DPP) frameworks as applied to control of Q-MJD processes. The developed sampling based algorithms illustrate how the presented framework is a multi trajectory optimization method to solve nonlinear stochastic optimal control problems for Q-MJD processes.

In this paper a novel method towards solving the Stochastic Optimal Control problem is proposed, which is based on the combination of the generalized Polynomial Chaos theory and the Differential Dynamic Programming framework. Utilizing the Polynomial Chaos theory allows us to handle a wide range of uncertainties, without having to rely on limiting assumptions regarding the form of stochasticity. In addition, the Differential Dynamic Programming framework provides an iterative algorithm for finding optimal controls, which attains scalability and, under mild assumptions, fast convergence. Last but not least, towards increasing the numerical stability of our algorithm, the concept of Variational Integrators is incorporated. Numerical examples validate the applicability of the proposed approach.

In this paper we present a model predictive control algorithm designed for optimizing non-linear systems subject to complex cost criteria. The algorithm is based on a stochastic optimal control framework using a fundamental relationship between the information theoretic notions of free energy and relative entropy. The optimal controls in this setting take the form of a path integral, which we approximate using an efficient importance sampling scheme. We experimentally verify the algorithm by implementing it on a Graphics Processing Unit (GPU) and apply it to the problem of controlling a fifth-scale Auto-Rally vehicle in an aggressive driving task.

We present a novel trajectory optimization framework to address the issue of robustness, scalability and efficiency in optimal control and reinforcement learning. Based on prior work in Cooperative Stochastic Differential Game (CSDG) theory, our method performs local trajectory optimization using cooperative controllers. The resulting framework is called Cooperative Game-Differential Dynamic Programming (CG-DDP). Compared to related methods, CG-DDP exhibits improved performance in terms of robustness and efficiency. The proposed framework is also applied in a data-driven fashion for belief space trajectory optimization under learned dynamics. We present experiments showing that CG-DDP can be used for optimal control and reinforcement learning under external disturbances and internal model uncertainties.

We present a Bayesian nonparametric trajectory optimization framework for systems with unknown dynamics using Gaussian Processes (GPs), called Gaussian Process Differential Dynamic Programming (GPDDP). Rooted in the Dynamic Programming principle and second-order local approximations of the value function, GPDDP learns time-varying optimal control policies from sampled data. Based on this framework, we propose two algorithms for implementations. We demonstrate the effectiveness and efficiency of the proposed framework using three numerical examples.

In this work, we derive a Game Theoretic Differential Dynamic Programming (GT-DDP) algorithm in continuous time. We provide a set of backward differential equations for the value function expansion without assuming closeness of the initial nominal control to the optimal control solution, and derive the update law for the controls. We introduce the GT-DDP algorithm and analyze the effect of the game theoretic formulation in the feed-forward and feedback parts of the control policies. Furthermore, we investigate the performance of GT-DDP through simulations on the inverted pendulum with conflicting controls and we apply the control gains on a stochastic system to demonstrate the effect of the design of the cost function to the feed-forward and feedback parts of the control policies. Finally, we conclude with some possible future directions.

In this work, we revisit the continuous-time Differential Dynamic Programming (DDP) approach for solving optimal control problems with terminal state constraints. We derive two algorithms, each for different order of expansion of the system dynamics and we investigate their performance in terms of their convergence speed. Compared to previous work, we provide a set of backward differential equations for the value function expansion by relaxing the assumption that the initial nominal control must be very close to the optimal control solution. We apply the derived algorithms to two classical optimal control problems, namely, the inverted pendulum and the Dreyfus rocket problem and show the benefit of second order expansion.

This paper considers optimal control of dynamical systems which are represented by nonlinear stochastic differential equations. It is well-known that the optimal control policy for this problem can be obtained as a function of a value function that satisfies a nonlinear partial differential equation, namely, the Hamilton-Jacobi-Bellman equation. This nonlinear PDE must be solved backwards in time, and this computation is intractable for large scale systems. Under certain assumptions, and after applying a logarithmic transformation, an alternative characterization of the optimal policy can be given in terms of a path integral. Path Integral (PI) based control methods have recently been shown to provide elegant solutions to a broad class of stochastic optimal control problems. One of the implementation challenges with this formalism is the computation of the expectation of a cost functional over the trajectories of the unforced dynamics. Computing such expectation over trajectories that are sampled uniformly may induce numerical instabilities due to the exponentiation of the cost. Therefore, sampling of low-cost trajectories is essential for the practical implementation of PI-based methods. In this paper, we use incremental sampling-based algorithms to sample useful trajectories from the unforced system dynamics, and make a novel connection between Rapidly-exploring Random Trees (RRTs) and information-theoretic stochastic optimal control. We show the results from the numerical implementation of the proposed approach to several examples., 18 pages

We present two nonparametric approaches to Kullback-Leibler (KL) control, or linearly-solvable Markov decision problem (LMDP) based on Gaussian processes (GP) and Nystr\"{o}m approximation. Compared to recently developed parametric methods, the proposed data-driven frameworks feature accurate function approximation and efficient on-line operations. Theoretically, we derive the mathematical connection of KL control based on dynamic programming with earlier work in control theory which relies on information theoretic dualities for the infinite time horizon case. Algorithmically, we give explicit optimal control policies in nonparametric forms, and propose on-line update schemes with budgeted computational costs. Numerical results demonstrate the effectiveness and usefulness of the proposed frameworks.

We develop a general class of stochastic optimal control problems for which the problem of designing optimal linear feedback gains is convex. The class of problems includes arbitrary time varying linear systems and costs that are mixtures of exponentiated quadratics. This allows us to model problems with quadratic state costs and linear constraints on states and state transitions. Further, convexity in the feedback gains lets us impose arbitrary convex constraints or penalties on the feedback matrix: Thus we can model problems like distributed control (by imposing a sparsity structure on the feedback matrix) and variable-stiffness control (by applying time-varying penalties to feedback gain matrices). We show that the convex optimization problem can be solved efficiently by using the structure of the matrices involved. Finally, we present an application of these ideas to a practical problem arising in distributed control of power systems.

This paper presents a new approach for Active Simultaneous Localization and Mapping that uses the Relative Entropy(RE) optimization method to select trajectories which minimize both the localization error and the corresponding uncertainty bounds. To that end we construct a planning cost function which includes, besides the state and control cost, a term that encapsulates the uncertainty of the state. This term is the trace of the state covariance matrix produced by the estimator, in this case an Extended Kalman Filter. The role of the RE method is to iteratively guide the selection of the trajectories towards the ones minimizing the aforementioned cost. Once the method has converged, the result is a near-optimal path in terms of achieving the pre-defined goal in the state space while also improving the localization error and the total uncertainty. In essence the method integrates motion planning with robot localization. To evaluate the approach we consider scenarios with single and multiple robots navigating in presence of obstacles and various conditions of landmark densities. The results show a behavior consistent with our expectations.

Despite the plethora of reinforcement learning algorithms in machine learning and control, the majority of the work in this area relies on discrete time formulations of stochastic dynamics. In this work we present a new policy gradient algorithm for reinforcement learning in continuous state action spaces and continuous time for free energy-like cost functions. The derivation is based on successive application of Girsanov's theorem and the use of the Radon Nikodym derivative as formulated for Markov diffusion processes. The resulting policy gradient is reward weighted. The use of Radon Nikodym extends analysis and results to more general models of stochasticity in which jump diffusions processes are considered. We apply the resulting algorithm in two simple examples for learning attractor landscapes in rhythmic and discrete movements.

This paper integrates recent work on Path Integral (PI) and Kullback Leibler (KL) divergence stochastic optimal control theory with earlier work on risk sensitivity and the fundamental dualities between free energy and relative entropy. We derive the path integral optimal control framework and its iterative version based on the aforemetioned dualities. The resulting formulation of iterative path integral control is valid for general feedback policies and in contrast to previous work, it does not rely on pre-specified policy parameterizations. The derivation is based on successive applications of Girsanov's theorem and the use of Radon-Nikodým derivative as applied to diffusion processes due to the change of measure in the stochastic dynamics. We compare the PI control derived based on Dynamic Programming with PI based on the duality between free energy and relative entropy. Moreover we extend our analysis on the applicability of the relationship between free energy and relative entropy to optimal control of markov jump diffusions processes. Furthermore, we present the links between KL stochastic optimal control and the aforementioned dualities and discuss its generalizability.

Path integral methods [1], [2],[3] have recently been shown to be applicable to a very general class of optimal control problems. Here we examine the path integral formalism from a decision-theoretic point of view, since an optimal controller can always be regarded as an instance of a perfectly rational decision-maker that chooses its actions so as to maximize its expected utility [4]. The problem with perfect rationality is, however, that finding optimal actions is often very difficult due to prohibitive computational resource costs that are not taken into account. In contrast, a bounded rational decision-maker has only limited resources and therefore needs to strike some compromise between the desired utility and the required resource costs [5]. In particular, we suggest an information-theoretic measure of resource costs that can be derived axiomatically [6]. As a consequence we obtain a variational principle for choice probabilities that trades off maximizing a given utility criterion and avoiding resource costs that arise due to deviating from initially given default choice probabilities. The resulting bounded rational policies are in general probabilistic. We show that the solutions found by the path integral formalism are such bounded rational policies. Furthermore, we show that the same formalism generalizes to discrete control problems, leading to linearly solvable bounded rational control policies in the case of Markov systems. Importantly, Bellman's optimality principle is not presupposed by this variational principle, but it can be derived as a limit case. This suggests that the information-theoretic formalization of bounded rationality might serve as a general principle in control design that unifies a number of recently reported approximate optimal control methods both in the continuous and discrete domain.

We present a new approach to motion planning using a stochastic trajectory optimization framework. The approach relies on generating noisy trajectories to explore the space around an initial (possibly infeasible) trajectory, which are then combined to produced an updated trajectory with lower cost. A cost function based on a combination of obstacle and smoothness cost is optimized in each iteration. No gradient information is required for the particular optimization algorithm that we use and so general costs for which derivatives may not be available (e.g. costs corresponding to constraints and motor torques) can be included in the cost function. We demonstrate the approach both in simulation and on a mobile manipulation system for unconstrained and constrained tasks. We experimentally show that the stochastic nature of STOMP allows it to overcome local minima that gradient-based methods like CHOMP can get stuck in.

Although there has been a significant amount of work in the area of stochastic optimal control theory towards the development of new algorithms, the problem of how to control a stochastic nonlinear system remains an open research topic. Recent iterative linear quadratic optimal control methods iLQG [1], [2] handle control and state multiplicative noise while they are derived based on first order approximation of dynamics. On the other hand, methods such as Differential Dynamic Programming expand the dynamics up to the second order but so far they can handle nonlinear systems with additive noise. In this work we present a generalization of the classic Differential Dynamic Programming algorithm. We assume the existence of state and control multiplicative process noise, and proceed to derive the second-order expansion of the cost-to-go. We find the correction terms that arise from the stochastic assumption. Despite having quartic and cubic terms in the initial expression, we show that these vanish, leaving us with the same quadratic structure as standard DDP.

Reinforcement learning (RL) is one of the most general approaches to learning control. Its applicability to complex motor systems, however, has been largely impossible so far due to the computational difficulties that reinforcement learning encounters in high dimensional continuous state-action spaces. In this paper, we derive a novel approach to RL for parameterized control policies based on the framework of stochastic optimal control with path integrals. While solidly grounded in optimal control theory and estimation theory, the update equations for learning are surprisingly simple and have no danger of numerical instabilities as neither matrix inversions nor gradient learning rates are required. Empirical evaluations demonstrate significant performance improvements over gradient-based policy learning and scalability to high-dimensional control problems. Finally, a learning experiment on a robot dog illustrates the functionality of our algorithm in a real-world scenario. We believe that our new algorithm, Policy Improvement with Path Integrals (PI2), offers currently one of the most efficient, numerically robust, and easy to implement algorithms for RL in robotics.