Search

Direct data-driven design methods for the linear quadratic regulator (LQR) mainly use offline or episodic data batches, and their online adaptation has been acknowledged as an open problem. In this paper, we propose a direct adaptive method to learn the LQR from online closed-loop data. First, we propose a new policy parameterization based on the sample covariance to formulate a direct data-driven LQR problem, which is shown to be equivalent to the certainty-equivalence LQR with optimal non-asymptotic guarantees. Second, we design a novel data-enabled policy optimization (DeePO) method to directly update the policy, where the gradient is explicitly computed using only a batch of persistently exciting (PE) data. Third, we establish its global convergence via a projected gradient dominance property. Importantly, we efficiently use DeePO to adaptively learn the LQR by performing only one-step projected gradient descent per sample of the closed-loop system, which also leads to an explicit recursive update of the policy. Under PE inputs and for bounded noise, we show that the average regret of the LQR cost is upper-bounded by two terms signifying a sublinear decrease in time $\mathcal{O}(1/\sqrt{T})$ plus a bias scaling inversely with signal-to-noise ratio (SNR), which are independent of the noise statistics. Finally, we perform simulations to validate the theoretical results and demonstrate the computational and sample efficiency of our method.

Model Predictive Control (MPC) is a powerful method for complex system regulation, but its reliance on an accurate model poses many limitations in real-world applications. Data-driven predictive control (DDPC) aims at overcoming this limitation, by relying on historical data to provide information on the plant to be controlled. In this work, we present a unified stochastic framework for direct DDPC, where control actions are obtained by optimizing the Final Control Error (FCE), which is directly computed from available data only and automatically weighs the impact of uncertainty on the control objective. Our framework allows us to establish a separation principle for Predictive Control, elucidating the role that predictive models and their uncertainty play in DDPC. Moreover, it generalizes existing DDPC methods, like regularized Data-enabled Predictive Control (DeePC) and $\gamma$-DDPC, providing a path toward noise-tolerant data-based control with rigorous optimality guarantees. The theoretical investigation is complemented by a series of numerical case studies, revealing that the proposed method consistently outperforms or, at worst, matches existing techniques without requiring tuning regularization parameters as other methods do., Comment: 17 pages, 2 figures, 1 table, submitted to Automatica (first submission: December 22nd, 2023)

In this paper, we consider stable stochastic linear systems modeling whole-brain resting-state dynamics. We parametrize the state matrix of the system (effective connectivity) in terms of its steady-state covariance matrix (functional connectivity) and a skew-symmetric matrix $S$. We examine how the matrix $S$ influences some relevant dynamic properties of the system. Specifically, we show that a large $S$ enhances the degree of stability and excitability of the system, and makes the latter more responsive to high-frequency inputs., Comment: 6 pages, 5 figures, to appear in the proceedings of IEEE CDC 2023

In this paper, we study connections between the classical model-based approach to nonlinear system theory, where systems are represented by equations, and the nonlinear behavioral approach, where systems are defined as sets of trajectories. In particular, we focus on equivalent representations of the systems in the two frameworks for the problem of simulating a future nonlinear system trajectory starting from a given set of noisy data. The goal also includes extending some existing results from the deterministic to the stochastic setting., Comment: Accepted to CoDiT 2024

Model predictive control (MPC) is a control strategy widely used in industrial applications. However, its implementation typically requires a mathematical model of the system being controlled, which can be a time-consuming and expensive task. Data-driven predictive control (DDPC) methods offer an alternative approach that does not require an explicit mathematical model, but instead optimize the control policy directly from data. In this paper, we study the impact of two different regularization penalties on the closed-loop performance of a recently introduced data-driven method called $\gamma$-DDPC. Moreover, we discuss the tuning of the related coefficients in different data and noise scenarios, to provide some guidelines for the end user., Comment: 7 pages, 3 figures

Data-Driven Predictive Control (DDPC) has been recently proposed as an effective alternative to traditional Model Predictive Control (MPC), in that the same constrained optimization problem can be addressed without the need to explicitly identify a full model of the plant. However, DDPC is built upon input/output trajectories. Therefore, the finite sample effect of stochastic data, due to, e.g., measurement noise, may have a detrimental impact on closed-loop performance. Exploiting a formal statistical analysis of the prediction error, in this paper we propose the first systematic approach to deal with uncertainty due to finite sample effects. To this end, we introduce two regularization strategies for which, differently from existing regularization-based DDPC techniques, we propose a tuning rationale allowing us to select the regularization hyper-parameters before closing the loop and without additional experiments. Simulation results confirm the potential of the proposed strategy when closing the loop., Comment: 6 pages, 1 figure, this work has been submitted and accepted for publication at the IFAC World Congress 2023, Yokohama, Japan

Data-driven predictive control (DDPC) has been recently proposed as an effective alternative to traditional model-predictive control (MPC) for its unique features of being time-efficient and unbiased with respect to the oracle solution. Nonetheless, it has also been observed that noise may strongly jeopardize the final closed-loop performance since it affects both the data-based system representation and the control update computed from the online measurements. Recent studies have shown that regularization is potentially a successful tool to counteract the effect of noise. At the same time, regularization requires the tuning of a set of penalty terms, whose choice might be practically difficult without closed-loop experiments. In this paper, by means of subspace identification tools, we pursue a three-fold goal: $(i)$ we set up a unified framework for the existing regularized data-driven predictive control schemes for stochastic systems; $(ii)$ we introduce $\gamma$-DDPC, an efficient two-stage scheme that splits the optimization problem into two parts: fitting the initial conditions and optimizing the future performance, while guaranteeing constraint satisfaction; $(iii)$ we discuss the role of regularization for data-driven predictive control, providing new insight on $when$ and $how$ it should be applied. A benchmark numerical case study finally illustrates the performance of $\gamma$-DDPC, showing how controller design can be simplified in terms of tuning effort and computational complexity when benefiting from the insights coming from the subspace identification realm., Comment: 17 pages, 12 figures

We present a novel Deep Neural Network (DNN) architecture for non-linear system identification. We foster generalization by constraining DNN representational power. To do so, inspired by fading memory systems, we introduce inductive bias (on the architecture) and regularization (on the loss function). This architecture allows for automatic complexity selection based solely on available data, in this way the number of hyper-parameters that must be chosen by the user is reduced. Exploiting the highly parallelizable DNN framework (based on Stochastic optimization methods) we successfully apply our method to large scale datasets.

The Koopman operator is a mathematical tool that allows for a linear description of non-linear systems, but working in infinite dimensional spaces. Dynamic Mode Decomposition and Extended Dynamic Mode Decomposition are amongst the most popular finite dimensional approximation. In this paper we capture their core essence as a dual version of the same framework, incorporating them into the Kernel framework. To do so, we leverage the RKHS as a suitable space for learning the Koopman dynamics, thanks to its intrinsic finite-dimensional nature, shaped by data. We finally establish a strong link between kernel methods and Koopman operators, leading to the estimation of the latter through Kernel functions. We provide also simulations for comparison with standard procedures., Comment: Pre-print submitted for 19th IFAC Symposium, System Identification: learning models for decision and control

A different route to identification of time-invariant linear systems has been recently proposed which does not require committing to a specific parametric model structure. Impulse responses are described in a nonparametric Bayesian framework as zero-mean Gaussian processes. Their covariances are given by the so-called stable spline kernels encoding information on regularity and BIBO stability. In this paper, we demonstrate that these kernels also lead to a new family of radial basis functions kernels suitable to model system components subject to disturbances given by filtered white noise. This novel class, in cooperation with the stable spline kernels, paves the way to a new approach to solve missing data problems in both discrete and continuous-time settings. Numerical experiments show that the new technique may return models more predictive than those obtained by standard parametric Prediction Error Methods, also when these latter exploit the full data set.

This paper discusses online algorithms for inverse dynamics modelling in robotics. Several model classes including rigid body dynamics (RBD) models, data-driven models and semiparametric models (which are a combination of the previous two classes) are placed in a common framework. While model classes used in the literature typically exploit joint velocities and accelerations, which need to be approximated resorting to numerical differentiation schemes, in this paper a new `derivative-free' framework is proposed that does not require this preprocessing step. An extensive experimental study with real data from the right arm of the iCub robot is presented, comparing different model classes and estimation procedures, showing that the proposed `derivative-free' methods outperform existing methodologies., Comment: 14 pages, 11 figures

Causal relations among neuronal populations of the brain are studied through the so-called effective connectivity (EC) network. The latter is estimated from EEG or fMRI measurements, by inverting a generative model of the corresponding data. It is clear that the goodness of the estimated network heavily depends on the underlying modeling assumptions. In this present paper we consider the EC estimation problem using fMRI data in resting-state condition. Specifically, we investigate on how to model endogenous fluctuations driving the neuronal activity.

Contemporary neuroscience has embraced network science to study the complex and self-organized structure of the human brain; one of the main outstanding issues is that of inferring from measure data, chiefly functional Magnetic Resonance Imaging (fMRI), the so-called effective connectivity in brain networks, that is the existing interactions among neuronal populations. This inverse problem is complicated by the fact that the BOLD (Blood Oxygenation Level Dependent) signal measured by fMRI represent a dynamic and nonlinear transformation (the hemodynamic response) of neuronal activity. In this paper, we consider resting state (rs) fMRI data; building upon a linear population model of the BOLD signal and a stochastic linear DCM model, the model parameters are estimated through an EM-type iterative procedure, which alternately estimates the neuronal activity by means of the Rauch-Tung-Striebel (RTS) smoother, updates the connections among neuronal states and refines the parameters of the hemodynamic model; sparsity in the interconnection structure is favoured using an iteratively reweighting scheme. Experimental results using rs-fMRI data are shown demonstrating the effectiveness of our approach and comparison with state of the art routines (SPM12 toolbox) is provided.

Kernel-based methods have been recently introduced for linear system identification as an alternative to parametric prediction error methods. Adopting the Bayesian perspective, the impulse response is modeled as a non-stationary Gaussian process with zero mean and with a certain kernel (i.e. covariance) function. Choosing the kernel is one of the most challenging and important issues. In the present paper we introduce the harmonic analysis of this non-stationary process, and argue that this is an important tool which helps in designing such kernel. Furthermore, this analysis suggests also an effective way to approximate the kernel, which allows to reduce the computational burden of the identification procedure.

We extend the recently introduced regularization/Bayesian System Identification procedures to the estimation of time-varying systems. Specifically, we consider an online setting, in which new data become available at given time steps. The real-time estimation requirements imposed by this setting are met by estimating the hyper-parameters through just one gradient step in the marginal likelihood maximization and by exploiting the closed-form availability of the impulse response estimate (when Gaussian prior and Gaussian measurement noise are postulated). By relying on the use of a forgetting factor, we propose two methods to tackle the tracking of time-varying systems. In one of them, the forgetting factor is estimated by treating it as a hyper-parameter of the Bayesian inference procedure.

This paper presents a semi-parametric algorithm for online learning of a robot inverse dynamics model. It combines the strength of the parametric and non-parametric modeling. The former exploits the rigid body dynamics equa- tion, while the latter exploits a suitable kernel function. We provide an extensive comparison with other methods from the literature using real data from the iCub humanoid robot. In doing so we also compare two different techniques, namely cross validation and marginal likelihood optimization, for estimating the hyperparameters of the kernel function.

We consider an on-line system identification setting, in which new data become available at given time steps. In order to meet real-time estimation requirements, we propose a tailored Bayesian system identification procedure, in which the hyper-parameters are still updated through Marginal Likelihood maximization, but after only one iteration of a suitable iterative optimization algorithm. Both gradient methods and the EM algorithm are considered for the Marginal Likelihood optimization. We compare this "1-step" procedure with the standard one, in which the optimization method is run until convergence to a local minimum. The experiments we perform confirm the effectiveness of the approach we propose.

Modeling and identification of high-dimensional stochastic processes is ubiquitous in many fields. In particular, there is a growing interest in modeling stochastic processes with simple and interpretable structures. In many applications, such as econometrics and biomedical sciences, it seems natural to describe each component of that stochastic process in terms of few factor variables, which are not accessible for observation, and possibly of few other components of the stochastic process. These relations can be encoded in graphical way via a structured dynamic network, referred to as "sparse plus low-rank (S+L) network" hereafter. The problem of finding the S+L network as well as the dynamic model can be posed as a system identification problem. In this paper, we introduce two new nonparametric methods to identify dynamic models for stochastic processes described by a S+L network. These methods take inspiration from regularized estimators based on recently introduced kernels (e.g. "stable spline", "tuned-correlated" etc.). Numerical examples show the benefit to introduce the S+L structure in the identification procedure.

Recent contributions have framed linear system identification as a nonparametric regularized inverse problem. Relying on $\ell_2$-type regularization which accounts for the stability and smoothness of the impulse response to be estimated, these approaches have been shown to be competitive w.r.t classical parametric methods. In this paper, adopting Maximum Entropy arguments, we derive a new $\ell_2$ penalty deriving from a vector-valued kernel; to do so we exploit the structure of the Hankel matrix, thus controlling at the same time complexity, measured by the McMillan degree, stability and smoothness of the identified models. As a special case we recover the nuclear norm penalty on the squared block Hankel matrix. In contrast with previous literature on reweighted nuclear norm penalties, our kernel is described by a small number of hyper-parameters, which are iteratively updated through marginal likelihood maximization; constraining the structure of the kernel acts as a (hyper)regularizer which helps controlling the effective degrees of freedom of our estimator. To optimize the marginal likelihood we adapt a Scaled Gradient Projection (SGP) algorithm which is proved to be significantly computationally cheaper than other first and second order off-the-shelf optimization methods. The paper also contains an extensive comparison with many state-of-the-art methods on several Monte-Carlo studies, which confirms the effectiveness of our procedure.

Inspired by ideas taken from the machine learning literature, new regularization techniques have been recently introduced in linear system identification. In particular, all the adopted estimators solve a regularized least squares problem, differing in the nature of the penalty term assigned to the impulse response. Popular choices include atomic and nuclear norms (applied to Hankel matrices) as well as norms induced by the so called stable spline kernels. In this paper, a comparative study of estimators based on these different types of regularizers is reported. Our findings reveal that stable spline kernels outperform approaches based on atomic and nuclear norms since they suitably embed information on impulse response stability and smoothness. This point is illustrated using the Bayesian interpretation of regularization. We also design a new class of regularizers defined by "integral" versions of stable spline/TC kernels. Under quite realistic experimental conditions, the new estimators outperform classical prediction error methods also when the latter are equipped with an oracle for model order selection.

A new Bayesian approach to linear system identification has been proposed in a series of recent papers. The main idea is to frame linear system identification as predictor estimation in an infinite dimensional space, with the aid of regularization/Bayesian techniques. This approach guarantees the identification of stable predictors based on the prediction error minimization. Unluckily, the stability of the predictors does not guarantee the stability of the impulse response of the system. In this paper we propose and compare various techniques to address this issue. Simulations results comparing these techniques will be provided., Comment: number of pages = 6, number of figures = 3

The first order stable spline (SS-1) kernel is used extensively in regularized system identification. In particular, the stable spline estimator models the impulse response as a zero-mean Gaussian process whose covariance is given by the SS-1 kernel. In this paper, we discuss the maximum entropy properties of this prior. In particular, we formulate the exact maximum entropy problem solved by the SS-1 kernel without Gaussian and uniform sampling assumptions. Under general sampling schemes, we also explicitly derive the special structure underlying the SS-1 kernel (e.g. characterizing the tridiagonal nature of its inverse), also giving to it a maximum entropy covariance completion interpretation. Along the way similar maximum entropy properties of the Wiener kernel are also given.

We consider the problem of modeling multivariate time series with parsimonious dynamical models which can be represented as sparse dynamic Bayesian networks with few latent nodes. This structure translates into a sparse plus low rank model. In this paper, we propose a Gaussian regression approach to identify such a model.

Inference of three-dimensional motion from the fusion of inertial and visual sensory data has to contend with the preponderance of outliers in the latter. Robust filtering deals with the joint inference and classification task of selecting which data fits the model, and estimating its state. We derive the optimal discriminant and propose several approximations, some used in the literature, others new. We compare them analytically, by pointing to the assumptions underlying their approximations, and empirically. We show that the best performing method improves the performance of state-of-the-art visual-inertial sensor fusion systems, while retaining the same computational complexity., Comment: Submitted to ICRA 2015, Manuscript #2912. Video results available at: http://youtu.be/5JSF0-DbIRc

Visual representations are defined in terms of minimal sufficient statistics of visual data, for a class of tasks, that are also invariant to nuisance variability. Minimal sufficiency guarantees that we can store a representation in lieu of raw data with smallest complexity and no performance loss on the task at hand. Invariance guarantees that the statistic is constant with respect to uninformative transformations of the data. We derive analytical expressions for such representations and show they are related to feature descriptors commonly used in computer vision, as well as to convolutional neural networks. This link highlights the assumptions and approximations tacitly assumed by these methods and explains empirical practices such as clamping, pooling and joint normalization., Comment: UCLA CSD TR140023, Nov. 12, 2014, revised April 13, 2015, November 13, 2015, February 28, 2016