Search

We propose a new Q-learning variant, called 2RA Q-learning, that addresses some weaknesses of existing Q-learning methods in a principled manner. One such weakness is an underlying estimation bias which cannot be controlled and often results in poor performance. We propose a distributionally robust estimator for the maximum expected value term, which allows us to precisely control the level of estimation bias introduced. The distributionally robust estimator admits a closed-form solution such that the proposed algorithm has a computational cost per iteration comparable to Watkins' Q-learning. For the tabular case, we show that 2RA Q-learning converges to the optimal policy and analyze its asymptotic mean-squared error. Lastly, we conduct numerical experiments for various settings, which corroborate our theoretical findings and indicate that 2RA Q-learning often performs better than existing methods., Comment: 26 pages, 5 figures

This paper proposes a data-driven approach to design a feedforward Neural Network (NN) controller with a stability guarantee for systems with unknown dynamics. We first introduce data-driven representations of stability conditions for Neural Feedback Loops (NFLs) with linear plants. These conditions are then formulated into a semidefinite program (SDP). Subsequently, this SDP constraint is integrated into the NN training process resulting in a stable NN controller. We propose an iterative algorithm to solve this problem efficiently. Finally, we illustrate the effectiveness of the proposed method and its superiority compared to model-based methods via numerical examples.

The paper aims to investigate relevant computational issues of deep neural network architectures with an eye to the interaction between the optimization algorithm and the classification performance. In particular, we aim to analyze the behaviour of state-of-the-art optimization algorithms in relationship to their hyperparameters setting in order to detect robustness with respect to the choice of a certain starting point in ending on different local solutions. We conduct extensive computational experiments using nine open-source optimization algorithms to train deep Convolutional Neural Network architectures on an image multi-class classification task. Precisely, we consider several architectures by changing the number of layers and neurons per layer, in order to evaluate the impact of different width and depth structures on the computational optimization performance.

Robust Reinforcement Learning (RRL) is a promising Reinforcement Learning (RL) paradigm aimed at training robust to uncertainty or disturbances models, making them more efficient for real-world applications. Following this paradigm, uncertainty or disturbances are interpreted as actions of a second adversarial agent, and thus, the problem is reduced to seeking the agents' policies robust to any opponent's actions. This paper is the first to propose considering the RRL problems within the positional differential game theory, which helps us to obtain theoretically justified intuition to develop a centralized Q-learning approach. Namely, we prove that under Isaacs's condition (sufficiently general for real-world dynamical systems), the same Q-function can be utilized as an approximate solution of both minimax and maximin Bellman equations. Based on these results, we present the Isaacs Deep Q-Network algorithms and demonstrate their superiority compared to other baseline RRL and Multi-Agent RL algorithms in various environments.

Real-time environmental monitoring using a multi-agent system (MAS) has long been a focal point of cooperative control. It is still a challenging task to provide cost-effective services for potential emergencies in surface environments. This paper explores the transformation of a general surface into a two-dimensional (2D) disk through the construction of a conformal mapping. Multiple agents are strategically deployed within the mapped convex disk, followed by mapping back to the original surface environment. This approach circumvents the complexities associated with handling the difficulties and intricacies of path planning. Technical analysis encompasses the design of distributed control laws and the method to eliminate distortions introduced by the mapping. Moreover, the developed coverage algorithm is applied to a scenario of monitoring surface deformation. Finally, the effectiveness of the proposed algorithm is validated through numerical simulations.

This study experimentally optimizes the mixing of a turbulent jet at $Re=10000$ with the surrounding air by targeted shear layer actuation. The forcing is composed of superposed harmonic signals of different azimuthal wavenumber $m$ generated by eight loudspeakers circumferentially distributed around the nozzle lip. Amplitudes and frequencies of the individual harmonic contributions serve as optimization parameters and the time-averaged centerline velocity downstream of the potential core is used as a metric for mixing optimization. The actuation is optimized through Bayesian optimization. Three search spaces are explored - axisymmetric forcing, $m=0$, superposed axisymmetric and helical forcing, $m \in \{0,1\}$, and axisymmetric actuation combined with two counter-rotating helical modes, $m \in \{-1,0,1\}$. High-speed PIV is employed to analyze the jet response to the optimized forcing. The optimization processes are analyzed by persistent data topology. In the search space of axisymmetric excitation, the routine identifies an actuation at the natural frequency of the flow to be most efficient, with the centerline velocity being decreased by $15\%$. The optimal solutions in both the two-mode and three-mode search space converge to a similar forcing with one axial and one helical mode combined at a frequency ratio of around $2.3$. Spectral analysis of the PIV images reveals that for the identified optimal forcing frequencies, a non-linear interaction between forced and natural structures in the jet flow is triggered, leading to a reduction in centerline velocity of around $35\%$. The topology of the most complex search space from the discrete data reveals four basins of attractions, classified into three forcing patterns including axisymmetric, axisym.-helical, and axisym.-flapping. Two deep basins are related to the optimal axisym.-helical pattern, and the others are shallower.

This paper is concerned with collective variables, or reaction coordinates, that map a discrete-in-time Markov process $X_n$ in $\mathbb{R}^d$ to a (much) smaller dimension $k \ll d$. We define the effective dynamics under a given collective variable map $\xi$ as the best Markovian representation of $X_n$ under $\xi$. The novelty of the paper is that it gives strict criteria for selecting optimal collective variables via the properties of the effective dynamics. In particular, we show that the transition density of the effective dynamics of the optimal collective variable solves a relative entropy minimization problem from certain family of densities to the transition density of $X_n$. We also show that many transfer operator-based data-driven numerical approaches essentially learn quantities of the effective dynamics. Furthermore, we obtain various error estimates for the effective dynamics in approximating dominant timescales / eigenvalues and transition rates of the original process $X_n$ and how optimal collective variables minimize these errors. Our results contribute to the development of theoretical tools for the understanding of complex dynamical systems, e.g. molecular kinetics, on large timescales. These results shed light on the relations among existing data-driven numerical approaches for identifying good collective variables, and they also motivate the development of new methods.

We consider the problem of parameter estimation in dynamic systems described by ordinary differential equations. A review of the existing literature emphasizes the need for deterministic global optimization methods due to the nonconvex nature of these problems. Recent works have focused on expanding the capabilities of specialized deterministic global optimization algorithms to handle more complex problems. Despite advancements, current deterministic methods are limited to problems with a maximum of around five state and five decision variables, prompting ongoing efforts to enhance their applicability to practical problems. Our study seeks to assess the effectiveness of state-of-the-art general-purpose global and local solvers in handling realistic-sized problems efficiently, and evaluating their capabilities to cope with the nonconvex nature of the underlying estimation problems.

Traditional mathematical programming solvers require long computational times to solve constrained minimization problems of complex and large-scale physical systems. Therefore, these problems are often transformed into unconstrained ones, and solved with computationally efficient optimization approaches based on first-order information, such as the gradient descent method. However, for unconstrained problems, balancing the minimization of the objective function with the reduction of constraint violations is challenging. We consider the class of time-dependent minimization problems with increasing (possibly) nonlinear and non-convex objective function and non-decreasing (possibly) nonlinear and non-convex inequality constraints. To efficiently solve them, we propose a penalty-based guardrail algorithm (PGA). This algorithm adapts a standard penalty-based method by dynamically updating the right-hand side of the constraints with a guardrail variable which adds a margin to prevent violations. We evaluate PGA on two novel application domains: a simplified model of a district heating system and an optimization model derived from learned deep neural networks. Our method significantly outperforms mathematical programming solvers and the standard penalty-based method, and achieves better performance and faster convergence than a state-of-the-art algorithm (IPDD) within a specified time limit.

We consider geodesically convex optimization problems involving distances to a finite set of points $A$ in a CAT(0) cubical complex. Examples include the minimum enclosing ball problem, the weighted mean and median problems, and the feasibility and projection problems for intersecting balls with centers in $A$. We propose a decomposition approach relying on standard Euclidean cutting plane algorithms. The cutting planes are readily derivable from efficient algorithms for computing geodesics in the complex.

In this paper, we study a dynamic pickup and delivery problem with docking constraints. There is a homogeneous fleet of vehicles to serve pickup-and-delivery requests at given locations. The vehicles can be loaded up to their capacity, while unloading has to follow the last-in-first-out (LIFO) rule. The locations have a limited number of docking ports for loading and unloading, which may force the vehicles to wait. The problem is dynamic since the transportation requests arrive real-time, over the day. Accordingly, the routes of the vehicles are to be determined dynamically. The goal is to satisfy all the requests such that a combination of tardiness penalties and traveling costs is minimized. We propose a cost function approximation based solution method. In each decision epoch, we solve the respective optimization problem with a perturbed objective function to ensure the solutions remain adaptable to accommodate new requests. We penalize waiting times and idle vehicles. We propose a variable neighborhood search based method for solving the optimization problems, and we apply two existing local search operators, and we also introduce a new one. We evaluate our method using a widely adopted benchmark dataset, and the results demonstrate that our approach significantly surpasses the current state-of-the-art methods.

This paper presents optimization models for electric Mobility-as-a-Service systems, whereby electric vehicles not only provide on-demand mobility, but also perform charging and Vehicle-to-Grid (V2G) operations to enhance the fleet operator profitability. Specifically, we formulate the optimal fleet operation problem as a mixed-integer linear program, with the objective combining of operational costs and revenues generated from servicing requests and grid electricity sales. Our cost function explicitly captures battery price and degradation, reflecting their impact on the fleet total cost of ownership due to additional charging and discharging activities. Simulation results for Eindhoven, The Netherlands, show that integrating V2G activities does not compromise the number of travel requests being served. Moreover, we emphasize the significance of accounting for battery degradation, as the costs associated with it can potentially outweigh the revenues stemming from V2G operations.

In this paper, the moment matching techniques are adopted to obtain reduced-order closed-loop systems with reduced-order controllers that maintain the closed-loop stability and guarantee desired asymptotic performance, after revealing the relationship between the Internal Model Principle used in control design and the time-domain moment matching problem. As a result, the design of a low order controller can be done starting from considering the achieving of asymptotic performance as a moment matching problem, resulting in a reduced order closed-loop system., Comment: 7 pages. Preliminary resukts have been presented in CDC2013

Positive semidefinite (PSD) cone is the cone of positive semidefinite matrices, and is the object of interest in semidefinite programming (SDP). A computational efficient approximation of the PSD cone is the $k$-PSD closure, $1 \leq k < n$, cone of $n\times n$ real symmetric matrices such that all of their $k\times k$ principal submatrices are positive semidefinite. For $k=1$, one obtains a polyhedral approximation, while $k=2$ yields a second order conic (SOC) approximation of the PSD cone. These approximations of the PSD cone have been used extensively in real-world applications such as AC Optimal Power Flow (ACOPF) to address computational inefficiencies where SDP relaxations are utilized for convexification the non-convexities. However a theoretical discussion about the geometry of these conic approximations of the PSD cone is rather sparse. In this short communication, we attempt to provide a characterization of some family of generators of the aforementioned conic approximations.

In this paper, a backward map is introduced for the purposes of analysis of the nonlinear (stochastic) filter stability. The backward map is important because the filter-stability in the sense of $\chisq$-divergence follows from showing a certain variance decay property for the backward map. To show this property requires additional assumptions on the model properties of the hidden Markov model (HMM). The analysis in this paper is based on introducing a Poincar\'e Inequality (PI) for HMMs with white noise observations. In finite state-space settings, PI is related to both the ergodicity of the Markov process as well as the observability of the HMM. It is shown that the Poincar\'e constant is positive if and only if the HMM is detectable.

The present paper deals with the perturbation analysis of set-valued inclusion problems, a problem format whose relevance has recently emerged in such contexts as robust and vector optimization as well as in vector equilibrium theory. The set-valued inclusions here considered are parameterized by variables belonging to a topological space, with and without constraints. By proper techniques of variational analysis, some qualitative global implicit function theorems are established, which ensure global solvability of these problems and continuous dependence on the parameter of the related solutions. Applications to parametric vector optimization are discussed, aimed at deriving sufficient conditions for the existence of ideal efficient solutions that depend continuously on the parameter perturbations.

This paper studies a class of network games with linear-quadratic payoffs and externalities exerted through a strictly concave interaction function. This class of game is motivated by the diminishing marginal effects with peer influences. We analyze the optimal pricing strategy for this class of network game. First, we prove the existence of a unique Nash Equilibrium (NE). Second, we study the optimal pricing strategy of a monopolist selling a divisible good to agents. We show that the optimal pricing strategy, found by solving a bilevel optimization problem, is strictly better when the monopolist knows the network structure as opposed to the best strategy agnostic to network structure. Numerical experiments demonstrate that in most cases, the maximum revenue is achieved with an asymmetric network. These results contrast with the previously studied case of linear interaction function, where a network-independent price is proven optimal with symmetric networks. Lastly, we describe an efficient algorithm to find the optimal pricing strategy., Comment: 7 pages, 2 figures, revisions under IEEE Control Systems Letters

Decentralized learning is appealing as it enables the scalable usage of large amounts of distributed data and resources (without resorting to any central entity), while promoting privacy since every user minimizes the direct exposure of their data. Yet, without additional precautions, curious users can still leverage models obtained from their peers to violate privacy. In this paper, we propose Decor, a variant of decentralized SGD with differential privacy (DP) guarantees. Essentially, in Decor, users securely exchange randomness seeds in one communication round to generate pairwise-canceling correlated Gaussian noises, which are injected to protect local models at every communication round. We theoretically and empirically show that, for arbitrary connected graphs, Decor matches the central DP optimal privacy-utility trade-off. We do so under SecLDP, our new relaxation of local DP, which protects all user communications against an external eavesdropper and curious users, assuming that every pair of connected users shares a secret, i.e., an information hidden to all others. The main theoretical challenge is to control the accumulation of non-canceling correlated noise due to network sparsity. We also propose a companion SecLDP privacy accountant for public use., Comment: Accepted as conference paper at ICML 2024

We consider Cournot mean field games of controls, a model originally developed for the production of an exhaustible resource by a continuum of producers. We prove uniqueness of the solution under general assumptions on the price function. Then, we prove convergence of a learning algorithm which gives existence of a solution to the mean field games system. The learning algorithm is implemented with a suitable finite difference discretization to get a numerical method to the solution. We supplement our theoretical analysis with several numerical examples and illustrate the impacts of model parameters.

We propose a Markov Chain Monte Carlo (MCMC) algorithm based on Gibbs sampling with parallel tempering to solve nonlinear optimal control problems. The algorithm is applicable to nonlinear systems with dynamics that can be approximately represented by a finite dimensional Koopman model, potentially with high dimension. This algorithm exploits linearity of the Koopman representation to achieve significant computational saving for large lifted states. We use a video-game to illustrate the use of the method.