Search

\Ac{MPC} and \ac{RL} are two powerful control strategies with, arguably, complementary advantages. In this work, we show how actor-critic \ac{RL} techniques can be leveraged to improve the performance of \ac{MPC}. The \ac{RL} critic is used as an approximation of the optimal value function, and an actor roll-out provides an initial guess for primal variables of the \ac{MPC}. A parallel control architecture is proposed where each \ac{MPC} instance is solved twice for different initial guesses. Besides the actor roll-out initialization, a shifted initialization from the previous solution is used. Thereafter, the actor and the critic are again used to approximately evaluate the infinite horizon cost of these trajectories. The control actions from the lowest-cost trajectory are applied to the system at each time step. We establish that the proposed algorithm is guaranteed to outperform the original \ac{RL} policy plus an error term that depends on the accuracy of the critic and decays with the horizon length of the \ac{MPC} formulation. Moreover, we do not require globally optimal solutions for these guarantees to hold. The approach is demonstrated on an illustrative toy example and an \ac{AD} overtaking scenario.

For continuous decision spaces, nonlinear programs (NLPs) can be efficiently solved via sequential quadratic programming (SQP) and, more generally, sequential convex programming (SCP). These algorithms linearize only the nonlinear equality constraints and keep the outer convex structure of the problem intact. The aim of the presented sequential mixed-integer quadratic programming (MIQP) algorithm for mixed-integer nonlinear problems (MINLPs) is to extend the SQP/SCP methodology to MINLPs and leverage the availability of efficient MIQP solvers. The algorithm employs a three-step method in each iterate: First, the MINLP is linearized at a given iterate. Second, an MIQP with its feasible set restricted to a specific region around the current linearization point is formulated and solved. Third, the integer variables obtained from the MIQP solution are fixed, and only an NLP in the continuous variables is solved. The outcome of the third step is compared to previous iterates, and the best iterate so far is used as a linearization point in the next iterate. Crucially, the objective values and derivatives from all previous iterates are used to formulate the polyhedral region in the second step. The linear inequalities that define the region build on concepts from generalized Benders' decomposition for MINLPs. Although the presented MINLP algorithm is a heuristic method without any global optimality guarantee, it converges to the exact integer solution when applied to convex MINLP with a linear outer structure. The conducted numerical experiments demonstrate that the proposed algorithm is competitive with other open-source solvers for MINLP. Finally, we solve two mixed-integer optimal control problems (MIOCPs) transcribed into MINLPs via direct methods, showing that the presented algorithm can effectively deal with nonlinear equality constraints, a major hurdle for generic MINLP solvers., Comment: Andrea Ghezzi and Wim Van Roy contributed equally to this work. 34 pages, 11 figures, 3 tables

This work presents a novel loss function for learning nonlinear Model Predictive Control policies via Imitation Learning. Standard approaches to Imitation Learning neglect information about the expert and generally adopt a loss function based on the distance between expert and learned controls. In this work, we present a loss based on the Q-function directly embedding the performance objectives and constraint satisfaction of the associated Optimal Control Problem (OCP). However, training a Neural Network with the Q-loss requires solving the associated OCP for each new sample. To alleviate the computational burden, we derive a second Q-loss based on the Gauss-Newton approximation of the OCP resulting in a faster training time. We validate our losses against Behavioral Cloning, the standard approach to Imitation Learning, on the control of a nonlinear system with constraints. The final results show that the Q-function-based losses significantly reduce the amount of constraint violations while achieving comparable or better closed-loop costs., Comment: Submitted to Conference on Decision and Control (CDC) 2023. The paper contains 6 pages

We present an optimization-based method for the joint estimation of system parameters and noise covariances of linear time-variant systems. Given measured data, this method maximizes the likelihood of the parameters. We solve the optimization problem of interest via a novel structure-exploiting solver. We present the advantages of the proposed approach over commonly used methods in the framework of Moving Horizon Estimation. Finally, we show the performance of the method through numerical simulations on a realistic example of a thermal system. In this example, the method can successfully estimate the model parameters in a short computational time., Comment: Submitted to IEEE Control Systems Letters (L-CSS) in March 2023. Contains 6 pages including 5 figures

We present a formulation for both implicit and explicit dual model predictive control for a steel recycling process. The process consists in the production of new steel by choosing a combination of several different steel scraps with unknown pollutant content. The pollutant content can only be measured after a scrap combination is molten, allowing for inference on the pollutants in the different scrap heaps. The production cost should be minimized while ensuring high quality of the product through constraining the maximum amount of pollutant. The dual control formulation allows to achieve the optimal explore-exploit trade-off between uncertainty reduction and cost minimization for the examined problem. Specifically, the dual effect is obtained by considering the dependence of the future pollutant uncertainties on the scrap selection in the predictions. The implicit formulation promotes uncertainty reduction indirectly via the impact of active constraints on the objective, while the explicit formulation adds a heuristic cost on uncertainty to encourage active exploration. We compare the formulations by numerical simulations of a simplified but representative industrial steel recycling process. The results demonstrate the superiority of the two dual formulations with respect to a robustified but non-dual formulation., Comment: Rejected by IEEE Conference on Decision and Control 2022 (CDC 22) Submitted after major changes based on reviewers indications to IEEE European Control Conference 2023 (ECC23)