Search

This article introduces new multiplicative updates for nonnegative matrix factorization with the $\beta$-divergence and sparse regularization of one of the two factors (say, the activation matrix). It is well known that the norm of the other factor (the dictionary matrix) needs to be controlled in order to avoid an ill-posed formulation. Standard practice consists in constraining the columns of the dictionary to have unit norm, which leads to a nontrivial optimization problem. Our approach leverages a reparametrization of the original problem into the optimization of an equivalent scale-invariant objective function. From there, we derive block-descent majorization-minimization algorithms that result in simple multiplicative updates for either $\ell_{1}$-regularization or the more "aggressive" log-regularization. In contrast with other state-of-the-art methods, our algorithms are universal in the sense that they can be applied to any $\beta$-divergence (i.e., any value of $\beta$) and that they come with convergence guarantees. We report numerical comparisons with existing heuristic and Lagrangian methods using various datasets: face images, an audio spectrogram, hyperspectral data, and song play counts. We show that our methods obtain solutions of similar quality at convergence (similar objective values) but with significantly reduced CPU times.

We propose a new distributed optimization algorithm for solving a class of constrained optimization problems in which (a) the objective function is separable (i.e., the sum of local objective functions of agents), (b) the optimization variables of distributed agents, which are subject to nontrivial local constraints, are coupled by global constraints, and (c) only noisy observations are available to estimate (the gradients of) local objective functions. In many practical scenarios, agents may not be willing to share their optimization variables with others. For this reason, we propose a distributed algorithm that does not require the agents to share their optimization variables with each other; instead, each agent maintains a local estimate of the global constraint functions and share the estimate only with its neighbors. These local estimates of constraint functions are updated using a consensus-type algorithm, while the local optimization variables of each agent are updated using a first-order method based on noisy estimates of gradient. We prove that, when the agents adopt the proposed algorithm, their optimization variables converge with probability 1 to an optimal point of an approximated problem based on the penalty method., 8 pages

We propose two distributionally robust optimization (DRO) models for a mobile facility (MF) fleet-sizing, routing, and scheduling problem (MFRSP) with time-dependent and random demand as well as methodologies for solving these models. Specifically, given a set of MFs, a planning horizon, and a service region, our models aim to find the number of MFs to use (i.e., fleet size) within the planning horizon and a route and time schedule for each MF in the fleet. The objective is to minimize the fixed cost of establishing the MF fleet plus a risk measure (expectation or mean conditional value at risk) of the operational cost over all demand distributions defined by an ambiguity set. In the first model, we use an ambiguity set based on the demand’s mean, support, and mean absolute deviation. In the second model, we use an ambiguity set that incorporates all distributions within a 1-Wasserstein distance from a reference distribution. To solve the proposed DRO models, we propose a decomposition-based algorithm. In addition, we derive valid lower bound inequalities that efficiently strengthen the master problem in the decomposition algorithm, thus improving convergence. We also derive two families of symmetry-breaking constraints that improve the solvability of the proposed models. Finally, we present extensive computational experiments comparing the operational and computational performance of the proposed models and a stochastic programming model, demonstrating when significant performance improvements could be gained, and derive insights into the MFRSP. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.1153 .

Standard evolutionary optimization algorithms assume that the evaluation of the objective and constraint functions is straightforward and computationally cheap. However, in many real-world optimization problems, these evaluations involve computationally expensive numerical simulations or physical experiments. Surrogate-assisted evolutionary algorithms (SAEAs) have recently gained increased attention for their performance in solving these types of problems. The main idea of SAEAs is the integration of an evolutionary algorithm with a selected surrogate model that approximates the computationally expensive function. In this paper, we propose a surrogate model based on a Lipschitz underestimation and use it to develop a differential evolution-based algorithm. The algorithm, called Lipschitz Surrogate-assisted Differential Evolution (LSADE), utilizes the Lipschitz-based surrogate model, along with a standard radial basis function surrogate model and a local search procedure. The experimental results on seven benchmark functions of dimensions 30, 50, 100, and 200 show that the proposed LSADE algorithm is competitive compared with the state-of-the-art algorithms under a limited computational budget, being especially effective for the very complicated benchmark functions in high dimensions.

In the paper, we study a class of useful minimax problems on Riemanian manifolds and propose a class of effective Riemanian gradient-based methods to solve these minimax problems. Specifically, we propose an effective Riemannian gradient descent ascent (RGDA) algorithm for the deterministic minimax optimization. Moreover, we prove that our RGDA has a sample complexity of $O(\kappa^2\epsilon^{-2})$ for finding an $\epsilon$-stationary solution of the Geodesically-Nonconvex Strongly-Concave (GNSC) minimax problems, where $\kappa$ denotes the condition number. At the same time, we present an effective Riemannian stochastic gradient descent ascent (RSGDA) algorithm for the stochastic minimax optimization, which has a sample complexity of $O(\kappa^4\epsilon^{-4})$ for finding an $\epsilon$-stationary solution. To further reduce the sample complexity, we propose an accelerated Riemannian stochastic gradient descent ascent (Acc-RSGDA) algorithm based on the momentum-based variance-reduced technique. We prove that our Acc-RSGDA algorithm achieves a lower sample complexity of $\tilde{O}(\kappa^{4}\epsilon^{-3})$ in searching for an $\epsilon$-stationary solution of the GNSC minimax problems. Extensive experimental results on the robust distributional optimization and robust Deep Neural Networks (DNNs) training over Stiefel manifold demonstrate efficiency of our algorithms., Comment: This paper was accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence

Federated learning (FL) is an emerging machine learning paradigm that allows to accomplish model training without aggregating data at a central server. Most studies on FL consider a centralized framework, in which a single server is endowed with a central authority to coordinate a number of devices to perform model training in an iterative manner. Due to stringent communication and bandwidth constraints, such a centralized framework has limited scalability as the number of devices grows. To address this issue, in this paper, we propose a ConFederated Learning (CFL) framework. The proposed CFL consists of multiple servers, in which each server is connected with an individual set of devices as in the conventional FL framework, and decentralized collaboration is leveraged among servers to make full use of the data dispersed throughout the network. We develop an alternating direction method of multipliers (ADMM) algorithm for CFL. The proposed algorithm employs a random scheduling policy which randomly selects a subset of devices to access their respective servers at each iteration, thus alleviating the need of uploading a huge amount of information from devices to servers. Theoretical analysis is presented to justify the proposed method. Numerical results show that the proposed method can converge to a decent solution significantly faster than gradient-based FL algorithms, thus boasting a substantial advantage in terms of communication efficiency., 13 pages, 5 figures

We consider the problem of controlling the group behavior of a large number of dynamic systems that are constantly interacting with each other. These systems are assumed to have identical dynamics (e.g., birds flock, robot swarm) and their group behavior can be modeled by a distribution. Thus, this problem can be viewed as an optimal control problem over the space of distributions. We propose a novel algorithm to compute a feedback control strategy so that, when adopted by the agents, the distribution of them would be transformed from an initial one to a target one over a finite time window. Our method is built on optimal transport theory but differs significantly from existing work in this area in that our method models the interactions among agents explicitly. From an algorithmic point of view, our algorithm is based on a generalized version of the proximal gradient descent algorithm and has a convergence guarantee with a sublinear rate. We further extend our framework to account for the scenarios where the agents are from multiple species. In the linear quadratic setting, the solution is characterized by coupled Riccati equations which can be solved in closed-form. Finally, several numerical examples are presented to illustrate our framework., Comment: 14 pages, 10 figures

This paper studies a variational obstacle avoidance problem on complete Riemannian manifolds. That is, we minimize an action functional, among a set of admissible curves, which depends on an artificial potential function used to avoid obstacles. In particular, we generalize the theory of bi-Jacobi fields and biconjugate points and present necessary and sufficient conditions for optimality. Local minimizers of the action functional are divided into two categories and subsequently classified, with local uniqueness results obtained in both cases., 11 pages

The fundamental lemma by Jan C. Willems and co-workers, which is deeply rooted in behavioral systems theory, has become one of the supporting pillars of the recent progress on data-driven control and system analysis. This tutorial-style paper combines recent insights into stochastic and descriptor-system formulations of the lemma to further extend and broaden the formal basis for behavioral theory of stochastic linear systems. We show that series expansions -- in particular Polynomial Chaos Expansions (PCE) of $L^2$-random variables, which date back to Norbert Wiener's seminal work -- enable equivalent behavioral characterizations of linear stochastic systems. Specifically, we prove that under mild assumptions the behavior of the dynamics of the $L^2$-random variables is equivalent to the behavior of the dynamics of the series expansion coefficients and that it entails the behavior composed of sampled realization trajectories. We also illustrate the short-comings of the behavior associated to the time-evolution of the statistical moments. The paper culminates in the formulation of the stochastic fundamental lemma for linear (descriptor) systems, which in turn enables numerically tractable formulations of data-driven stochastic optimal control combining Hankel matrices in realization data (i.e. in measurements) with PCE concepts., 31 pages, 7 figures

We study privacy-utility trade-offs where users share privacy-correlated useful information with a service provider to obtain some utility. The service provider is adversarial in the sense that it can infer the users' private information based on the shared useful information. To minimize the privacy leakage while maintaining a desired level of utility, the users carefully perturb the useful information via a probabilistic privacy mapping before sharing it. We focus on the setting in which the adversary attempting an inference attack on the users' privacy has potentially biased information about the statistical correlation between the private and useful variables. This information asymmetry between the users and the limited adversary leads to better privacy guarantees than the case of the omniscient adversary under the same utility requirement. We first identify assumptions on the adversary's information so that the inference costs are well-defined and finite. Then, we characterize the impact of the information asymmetry and show that it increases the inference costs for the adversary. We further formulate the design of the privacy mapping against a limited adversary using a difference of convex functions program and solve it via the concave-convex procedure. When the adversary's information is not precisely available, we adopt a Bayesian view and represent the adversary's information by a probability distribution. In this case, the expected cost for the adversary does not admit a closed-form expression, and we establish and maximize a lower bound of the expected cost. We provide a numerical example regarding a census data set to illustrate the theoretical results.

We present a quantum annealing-based solution method for topology optimization (TO). In particular, we consider TO in a more general setting, i.e., applied to structures of continuum domains where designs are represented as distributed functions, referred to as continuum TO problems. According to the problem's properties and structure, we formulate appropriate sub-problems that can be solved on an annealing-based quantum computer. The methodology established can effectively tackle continuum TO problems formulated as mixed-integer nonlinear programs. To maintain the resulting sub-problems small enough to be solved on quantum computers currently accessible with small numbers of quits and limited connectivity, we further develop a splitting approach that splits the problem into two parts: the first part can be efficiently solved on classical computers, and the second part with a reduced number of variables is solved on a quantum computer. By such, a practical continuum TO problem of varying scales can be handled on the D-Wave quantum annealer. More specifically, we concern the minimum compliance, a canonical TO problem that seeks an optimal distribution of materials to minimize the compliance with desired material usage. The superior performance of the developed methodology is assessed and compared with the state-of-the-art heuristic classical methods, in terms of both solution quality and computational efficiency. The present work hence provides a promising new avenue of applying quantum computing to practical designs of topology for various applications.

In recent years, a variety of gradient-based methods have been developed to solve Bi-Level Optimization (BLO) problems in machine learning and computer vision areas. However, the theoretical correctness and practical effectiveness of these existing approaches always rely on some restrictive conditions (e.g., Lower-Level Singleton, LLS), which could hardly be satisfied in real-world applications. Moreover, previous literature only proves theoretical results based on their specific iteration strategies, thus lack a general recipe to uniformly analyze the convergence behaviors of different gradient-based BLOs. In this work, we formulate BLOs from an optimistic bi-level viewpoint and establish a new gradient-based algorithmic framework, named Bi-level Descent Aggregation (BDA), to partially address the above issues. Specifically, BDA provides a modularized structure to hierarchically aggregate both the upper- and lower-level subproblems to generate our bi-level iterative dynamics. Theoretically, we establish a general convergence analysis template and derive a new proof recipe to investigate the essential theoretical properties of gradient-based BLO methods. Furthermore, this work systematically explores the convergence behavior of BDA in different optimization scenarios, i.e., considering various solution qualities (i.e., global/local/stationary solution) returned from solving approximation subproblems. Extensive experiments justify our theoretical results and demonstrate the superiority of the proposed algorithm for hyper-parameter optimization and meta-learning tasks. Source code is available at https://github.com/vis-opt-group/BDA., 18 pages, accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

We study the distribution and uncertainty of nonconvex optimization for noisy tensor completion -- the problem of estimating a low-rank tensor given incomplete and corrupted observations of its entries. Focusing on a two-stage estimation algorithm proposed by Cai et al. (2019), we characterize the distribution of this nonconvex estimator down to fine scales. This distributional theory in turn allows one to construct valid and short confidence intervals for both the unseen tensor entries and the unknown tensor factors. The proposed inferential procedure enjoys several important features: (1) it is fully adaptive to noise heteroscedasticity, and (2) it is data-driven and automatically adapts to unknown noise distributions. Furthermore, our findings unveil the statistical optimality of nonconvex tensor completion: it attains un-improvable $\ell_{2}$ accuracy -- including both the rates and the pre-constants -- when estimating both the unknown tensor and the underlying tensor factors., Comment: Accepted in part to ICML 2020

We consider the static output feedback control for Linear Quadratic Regulator problems with structured constraints under the assumption that system parameters are unknown. To solve the problem in the model free setting, we propose the policy gradient algorithm based on the gradient projection method and show its global convergence to $\varepsilon$-stationary points. In addition, we introduce a variance reduction technique and show both theoretically and numerically that it significantly reduces the variance in the gradient estimation. We also show in the numerical experiments that the model free approach efficiently solves the problem.

This contribution features an accelerated computation of the Sinkhorn's algorithm, which approximates the Wasserstein transportation distance, by employing nonequispaced fast Fourier transforms (NFFT). The algorithm proposed allows approximations of the Wasserstein distance by involving not more than $\mathcal O(n\log n)$ operations for probability measures supported by~$n$ points. Furthermore, the proposed method avoids expensive allocations of the characterizing matrices. With this numerical acceleration, the transportation distance is accessible to probability measures out of reach so far. Numerical experiments using synthetic and real data affirm the computational advantage and superiority., Comment: 32 pages, 6 figures

This work focuses on the fulfillment of the Persistent Excitation (PE) condition for signals which result from transformations by means of polynomials. This is essential e.g. for the convergence of Adaptive Dynamic Programming algorithms due to commonly used polynomial function approximators. As theoretical statements are scarce regarding the nonlinear transformation of PE signals, we propose conditions on the system state such that its transformation by polynomials is PE. To validate our theoretical statements, we develop an exemplary excitation procedure based on our conditions using a feedforward control approach and demonstrate the effectiveness of our method in a nonzero-sum differential game. In this setting, our approach outperforms commonly used probing noise in terms of convergence time and the degree of PE, shown by a numerical example.

In this technical note we analyse the performance improvement and optimality properties of the Learning Model Predictive Control (LMPC) strategy for linear deterministic systems. The LMPC framework is a policy iteration scheme where closed-loop trajectories are used to update the control policy for the next execution of the control task. We show that, when a Linear Independence Constraint Qualification (LICQ) condition holds, the LMPC scheme guarantees strict iterative performance improvement and optimality, meaning that the closed-loop cost evaluated over the entire task converges asymptotically to the optimal cost of the infinite-horizon control problem. Compared to previous works this sufficient LICQ condition can be easily checked, it holds for a larger class of systems and it can be used to adaptively select the prediction horizon of the controller, as demonstrated by a numerical example., Comment: technical note

Information compression is essential to reduce communication cost in distributed optimization over peer-to-peer networks. This paper proposes a communication-efficient linearly convergent distributed (COLD) algorithm to solve strongly convex optimization problems. By compressing innovation vectors, which are the differences between decision vectors and their estimates, COLD is able to achieve linear convergence for a class of $\delta$-contracted compressors. We explicitly quantify how the compression affects the convergence rate and show that COLD matches the same rate of its uncompressed version. To accommodate a wider class of compressors that includes the binary quantizer, we further design a novel dynamical scaling mechanism and obtain the linearly convergent Dyna-COLD. Importantly, our results strictly improve existing results for the quantized consensus problem. Numerical experiments demonstrate the advantages of both algorithms under different compressors., Comment: 14 pages

Distributed Optimization is an increasingly important subject area with the rise of multi-agent control and optimization. We consider a decentralized stochastic optimization problem where the agents on a graph aim to asynchronously optimize a collective (additive) objective function consisting of agents' individual (possibly non-convex) local objective functions. Each agent only has access to a noisy estimate of the gradient of its own function (one component of the sum of objective functions). We proposed an asynchronous distributed algorithm for such a class of problems. The algorithm combines stochastic gradients with tracking in an asynchronous push-sum framework and obtain the standard sublinear convergence rate for general non-convex functions, matching the rate of centralized stochastic gradient descent SGD. Our experiments on a non-convex image classification task using convolutional neural network validate the convergence of our proposed algorithm across different number of nodes and graph connectivity percentages.

In this paper, we study the null controllability of a nonlinear age, space and two-sex structured population dynamics model. This model is such that the nonlinearity and the couplage are at birth level. We consider a population with males and females and we are dealing with two cases of null controllability problems. The first problem is related to the total extinction, which means that, we estimate a time \begin{document}$ T $\end{document} to bring the male and female subpopulation density to zero. The second case concerns null controllability of male or female subpopulation. Since the absence of males or females in the population stops births; so, if we have the total extinction of the females at time \begin{document}$ T, $\end{document} and if \begin{document}$ A $\end{document} is the life span of the individuals, at time \begin{document}$ T+A $\end{document} one will get certainly the total extinction of the population. Our method uses first an observability inequality related to the adjoint of an auxiliary system, a null controllability of the linear auxiliary system, and after the Schauder's fixed point theorem.

We study a seemingly unexpected and relatively less understood overfitting aspect of a fundamental tool in sparse linear modeling - best subset selection, which minimizes the residual sum of squares subject to a constraint on the number of nonzero coefficients. While the best subset selection procedure is often perceived as the "gold standard" in sparse learning when the signal to noise ratio (SNR) is high, its predictive performance deteriorates when the SNR is low. In particular, it is outperformed by continuous shrinkage methods, such as ridge regression and the Lasso. We investigate the behavior of best subset selection in the high-noise regimes and propose an alternative approach based on a regularized version of the least-squares criterion. Our proposed estimators (a) mitigate, to a large extent, the poor predictive performance of best subset selection in the high-noise regimes; and (b) perform favorably, while generally delivering substantially sparser models, relative to the best predictive models available via ridge regression and the Lasso. We conduct an extensive theoretical analysis of the predictive properties of the proposed approach and provide justification for its superior predictive performance relative to best subset selection when the noise-level is high. Our estimators can be expressed as solutions to mixed integer second order conic optimization problems and, hence, are amenable to modern computational tools from mathematical optimization.

In the absence of randomized controlled and natural experiments, it is necessary to balance the distributions of (observable) covariates of the treated and control groups in order to obtain an unbiased estimate of a causal effect of interest; otherwise, a different effect size may be estimated, and incorrect recommendations may be given. To achieve this balance, there exist a wide variety of methods. In particular, several methods based on optimization models have been recently proposed in the causal inference literature. While these optimization-based methods empirically showed an improvement over a limited number of other causal inference methods in their relative ability to balance the distributions of covariates and to estimate causal effects, they have not been thoroughly compared to each other and to other noteworthy causal inference methods. In addition, we believe that there exist several unaddressed opportunities that operational researchers could contribute with their advanced knowledge of optimization, for the benefits of the applied researchers that use causal inference tools. In this review paper, we present an overview of the causal inference literature and describe in more detail the optimization-based causal inference methods, provide a comparative analysis of the prevailing optimization-based methods, and discuss opportunities for new methods., Comment: In Press, Corrected Proof

We consider a multi-agent optimal resource sharing problem that is represented by a linear program. The amount of resource to be shared is fixed, and agents belong to a population that is characterized probabilistically so as to allow heterogeneity among the agents. In this paper, we provide a characterization of the probability that the arrival of a new agent affects the resource share of other agents, which means that accommodating the new agent request at the detriment of the other agents allocation provides some payoff. This probability represents a sensitivity index for the optimal solution of a linear programming resource sharing problem when a new agent shows up, and it is of fundamental importance for a correct and profitable operation of the multi-agent system. Our developments build on the equivalence between the resource sharing problem and certain dual reformulations which can be interpreted as scenario programs with the number of scenarios corresponding to the number of agents in the primal problem. The recent "wait-and-judge" scenario approach is then used to obtain the sought sensitivity index. Our theoretical findings are demonstrated through a numerical example on optimal cargo aircraft loading., Comment: 14 pages, 3 figures

Distribution grid reliability and resilience has become a major topic of concern for utilities and their regulators. In particular, with the increase in severity of extreme events, utilities are considering major investments in distribution grid assets to mitigate the damage of highly impactful outages. Communicating the overall economic and risk-mitigation benefits of these investments to regulators is an important element of the approval process. Today, industry reliability and resilience planning practices are based largely on methods that do not take explicit account of risk. This paper proposes a practical method for identifying optimal combinations of investments in new line segments and storage devices while considering the balance between the risk associated with high impact low probability events and the reliability related to routine failures. We show that this method can be scaled to address large scale networks and demonstrate its benefits using a Target Feeder from the Commonwealth Edison Reliability Program., Comment: in IEEE Transactions on Power Systems, 2023

Conic optimization is the minimization of a convex quadratic function subject to conic constraints. We introduce a novel first-order method for conic optimization, named \emph{extrapolated proportional-integral projected gradient method (xPIPG)}, that automatically detects infeasibility. The iterates of xPIPG either asymptotically satisfy a set of primal-dual optimality conditions, or generate a proof of primal or dual infeasibility. We demonstrate the application of xPIPG using benchmark problems in model predictive control. xPIPG outperforms many state-of-the-art conic optimization solvers, especially when solving large-scale problems.

This work presents a convex optimization framework for the planning and tracking of quadcopter trajectories with continuous-time safety guarantees. Using B-spline basis functions and the differential flatness property of quadcopters, a second-order cone program is formulated to generate optimal trajectories that respect safe state and input constraints in the continuous-time sense. A quadratic program (QP) based on control barrier functions is proposed to guarantee bounded trajectory tracking in continuous time by filtering a nominal controller, where the QP is shown to be always feasible. Furthermore, conditions that ensure the safe tracking controller respects thrust, roll angle, and pitch angle constraints are also proposed. The effectiveness of the proposed framework is demonstrated by real-world experiments using a Crazyflie2.1 nano quadcopter., Comment: 16 pages, 13 figures

In this work, we propose a tube-based MPC scheme for state- and input-constrained linear systems subject to dynamic uncertainties characterized by dynamic integral quadratic constraints (IQCs). In particular, we extend the framework of $\rho$-hard IQCs for exponential stability analysis to external inputs. This result yields that the error between the true uncertain system and the nominal prediction model is bounded by an exponentially stable scalar system. In the proposed tube-based MPC scheme, the state of this error bounding system is predicted along with the nominal model and used as a scaling parameter for the tube size. We prove that this method achieves robust constraint satisfaction and input-to-state stability despite dynamic uncertainties and additive bounded disturbances. A numerical example demonstrates the reduced conservatism of this IQC approach compared to state-of-the-art robust MPC approaches for dynamic uncertainties.

In this work, we characterize a class of nonlinear monotone dynamical systems that have a certain translation invariance property which goes by the name of plus-homogeneity; usually called "topical" systems. Such systems need not be asymptotically stable, since they are merely nonexpansive but not contractive. Thus, we introduce a stricter version of monotonicity, termed "type-K" in honor of Kamke, and we prove the asymptotic stability of the equilibrium points, as well as the convergence of all trajectories to such equilibria for type-K monotone and plus-homogeneous systems: we call them "K-topical". Since topical maps are the natural nonlinear counterpart of linear maps defined by row-stochastic matrices, which are a cornerstone in the convergence analysis of linear multi-agent systems (MASs), we exploit our results for solving the consensus problem over nonlinear K-topical MASs. We first provide necessary and sufficient conditions on the local interaction rules of the agents ensuring the K-topicality of a MAS. Then, we prove that the agents achieve consensus asymptotically if the graph describing their interactions contains a globally reachable node. Finally, several examples for continuous-time and discrete-time systems are discussed to corroborate the enforceability of our results in different applications.

In this paper, we examine the influence of communication latency on performance of networked control systems. Even though distributed control architectures offer advantages in terms of communication, maintenance costs, and scalability, it is an open question how communication latency that varies with network topology influences closed-loop performance. For networks in which delays increase with the number of links, we establish the existence of a fundamental performance trade-off that arises from control architecture. In particular, we utilize consensus dynamics with single- and double-integrator agents to show that, if delays increase fast enough, a sparse controller with nearest neighbor interactions can outperform the centralized one with all-to-all communication topology., Comment: 12 pages, 13 figures; accepted for publication on IEEE Transactions on Control of Network Systems; final accepted version

This paper presents converse theorems for safety in terms of barrier functions for unconstrained continuous-time systems modeled as differential inclusions. Via a counterexample, we show the lack of existence of autonomous and continuous barrier functions certifying safety for a nonlinear system that is not only safe but also has a smooth right-hand side. Guided by converse Lyapunov theorems for (non-asymptotic) stability,time-varying barrier functions and appropriate infinitesimal conditions are shown to be both necessary as well as sufficient under mild regularity conditions on the right-hand side of the system. More precisely, we propose a general construction of a time-varying barrier function in terms of a marginal function involving the finite-horizon reachable set. Using techniques from set-valued and nonsmooth analysis, we show that such a function guarantees safety when the system is safe. Furthermore, we show that the proposed barrier function construction inherits the regularity properties of the proposed reachable set. In addition, when the system is safe and smooth, we build upon the constructed barrier function to show the existence of a smooth barrier function guaranteeing safety. Comparisons and relationships to results in the literature are also presented.

In the context of structured nonconvex optimization, we estimate the increase in minimum value for a decision that is robust to parameter perturbations as compared to the value of a nominal problem. The estimates rely on detailed expressions for subgradients and local Lipschitz moduli of min-value functions in nonconvex robust optimization and require only the solution of the nominal problem. The theoretical results are illustrated by examples from military operations research involving mixed-integer optimization models. Across 54 cases examined, the median error in estimating the increase in minimum value is 12%. Therefore, the derived expressions for subgradients and local Lipschitz moduli may accurately inform analysts about the possibility of obtaining cost-effective, parameter-robust decisions in nonconvex optimization.

We provide a stability and performance analysis for nonlinear model predictive control (NMPC) schemes subject to input constraints. Given an exponential stabilizability and detectability condition w.r.t. the employed state cost, we provide a sufficiently long prediction horizon to ensure asymptotic stability and a desired performance bound w.r.t. the infinite-horizon optimal controller. Compared to existing results, the provided analysis is applicable to positive semi-definite (detectable) cost functions, provides tight bounds using a linear programming analysis, and allows for a seamless integration of general positive-definite terminal cost functions in the analysis. The practical applicability of the derived theoretical results are demonstrated with numerical examples., Comment: This is the accepted version of the paper in IEEE Transaction on Automatic Control, 2023. This version contains additionally the proof of Theorem 7 in the appendix

In this paper we propose novel optimization-based methods for verifying reach-avoid (or, eventuality) properties of continuous-time systems modelled by ordinary differential equations. Given a system, an initial set, a safe set and a target set of states, we say that the reach-avoid property holds if for all initial conditions in the initial set, any trajectory of the system starting at them will eventually, i.e.\ in unbounded yet finite time, enter the target set while remaining inside the safe set until that first target hit. Based on a discount value function, two sets of quantified constraints are derived for verifying the reach-avoid property via the computation of exponential/asymptotic guidance-barrier functions (they form a barrier escorting the system to the target set safely at an exponential or asymptotic rate). It is interesting to find that one set of constraints whose solution is termed exponential guidance-barrier functions is just a simplified version of the existing one derived from the moment based method, while the other one whose solution is termed asymptotic guidance-barrier functions is completely new. Furthermore, built upon this new set of constraints, we derive a set of more expressive constraints, which includes the aforementioned two sets of constraints as special instances, providing more chances for verifying the reach-avoid properties successfully. When the involved datum are polynomials, i.e., the initial set, safe set and target set are semi-algebraic, and the system has polynomial dynamics, the problem of solving these sets of constraints can be framed as a semi-definite optimization problem using sum-of-squares decomposition techniques and thus can be efficiently solved in polynomial time via interior point methods. Finally, several examples demonstrate the theoretical developments and performance of proposed methods.

Control of multistable dynamical system has important applications, from physics to biology. Here, we attack this problem from the perspective of local sensitivity analysis. We develop sensitivity rules to control properties of continuous-time multistable dynamics with simple attractors, namely, the relative size and depth of their basins of attraction. Our parameter control signal is computationally cheap and provides counter-intuitive information about the sensitive parameters to be manipulated in an experimental setting., Comment: Gave an in-depth discussion of and added references to the theory in Section III. Slight change of structure in Control Strategies. Slight changes to figures and discussion of example systems

We model two systems of two conservation laws defined on complementary spatial intervals and coupled by a moving interface as a single non-autonomous port-Hamiltonian system, and provide sufficient conditions for its Kato-stability. An example shows that these conditions are quite restrictive. The more general question under which conditions an evolution family is generated remains open., arXiv admin note: text overlap with arXiv:2301.07344, arXiv:2301.08967

Joint chance-constrained optimal power flow (JCC-OPF) is a promising tool to manage uncertainties from distributed renewable generation. However, most existing works are based on power flow equations, which require accurate network parameters that may be unobservable in many distribution systems. To address this issue, this paper proposes a learning-based surrogate model for JCC-OPF with renewable generation. This model equivalently converts joint chance constraints in quantile-based forms and introduces deep quantile regression to replicate them, in which a multi-layer perceptron (MLP) is trained with a special loss function to predict the quantile of constraint violations. Another MLP is trained to predict the expected power loss. Then, the JCC-OPF can be formulated without network parameters by reformulating these two MLPs into mixed-integer linear constraints. To further improve its performance, two pre-processing steps, i.e., data augmentation and calibration, are developed. The former trains a simulator to generate more training samples for enhancing the prediction accuracy of MLPs. The latter designs a positive parameter to calibrate the predictions of MLPs so that the feasibility of solutions can be guaranteed. Numerical experiments based on the IEEE 33- and 123-bus systems validate that the proposed model can achieve desirable feasibility and optimality simultaneously with no need for network parameters., 10 pages

We investigate the problem of recovering a partially observed high-rank matrix whose columns obey a nonlinear structure such as a union of subspaces, an algebraic variety or grouped in clusters. The recovery problem is formulated as the rank minimization of a nonlinear feature map applied to the original matrix, which is then further approximated by a constrained non-convex optimization problem involving the Grassmann manifold. We propose two sets of algorithms, one arising from Riemannian optimization and the other as an alternating minimization scheme, both of which include first- and second-order variants. Both sets of algorithms have theoretical guarantees. In particular, for the alternating minimization, we establish global convergence and worst-case complexity bounds. Additionally, using the Kurdyka-Lojasiewicz property, we show that the alternating minimization converges to a unique limit point. We provide extensive numerical results for the recovery of union of subspaces and clustering under entry sampling and dense Gaussian sampling. Our methods are competitive with existing approaches and, in particular, high accuracy is achieved in the recovery using Riemannian second-order methods., Comment: Fixed some typos in version 2

Optimization of vaccine allocations among different segments of a heterogeneous population is important for enhancing the effectiveness of vaccination campaigns in reducing the burden of epidemics. Intuitively, it would seem that allocations designed to minimize infections should prioritize those with the highest risk of being infected and infecting others. This prescription is well supported by vaccination theory, e.g., when the vaccination campaign aims to reach herd immunity. In this work, we show, however, that for vaccines providing partial protection (leaky vaccines) and for sufficiently high values of the basic reproduction number, intuition is overturned: the optimal allocation for minimizing the number of infections prioritizes the vaccination of those who are least likely to be infected. Furthermore, we show that this phenomenon occurs at a range of basic reproduction numbers relevant for the currently circulating strains of SARS-CoV-19. The work combines numerical investigations, asymptotic analysis for a general model, and complete mathematical analysis in a simple two-group model. The results point to important considerations in managing vaccination campaigns for infections with high transmissibility.

We study the acceleration of the Local Polynomial Interpolation-based Gradient Descent method (LPI-GD) recently proposed for the approximate solution of empirical risk minimization problems (ERM). We focus on loss functions that are strongly convex and smooth with condition number $\sigma$. We additionally assume the loss function is $\eta$-H\"older continuous with respect to the data. The oracle complexity of LPI-GD is $\tilde{O}\left(\sigma m^d \log(1/\varepsilon)\right)$ for a desired accuracy $\varepsilon$, where $d$ is the dimension of the parameter space, and $m$ is the cardinality of an approximation grid. The factor $m^d$ can be shown to scale as $O((1/\varepsilon)^{d/2\eta})$. LPI-GD has been shown to have better oracle complexity than gradient descent (GD) and stochastic gradient descent (SGD) for certain parameter regimes. We propose two accelerated methods for the ERM problem based on LPI-GD and show an oracle complexity of $\tilde{O}\left(\sqrt{\sigma} m^d \log(1/\varepsilon)\right)$. Moreover, we provide the first empirical study on local polynomial interpolation-based gradient methods and corroborate that LPI-GD has better performance than GD and SGD in some scenarios, and the proposed methods achieve acceleration.

We aim to compute lifted stationary points of a sparse optimization problem (P0) with complementarity constraints. We define a continuous relaxation problem (Rv) that has the same global minimizers and optimal value with problem (P0). Problem (Rv) is a mathematical program with complementarity constraints (MPCC) and a difference-of-convex (DC) objective function. We define MPCC lifted-stationarity of (Rv) and show that it is weaker than directional stationarity, but stronger than Clarke stationarity for local optimality. Moreover, we propose an approximation method to solve (Rv) and an augmented Lagrangian method to solve its subproblem, which relaxes the equality constraint in (Rv) with a tolerance. We prove the convergence of our algorithm to an MPCC lifted-stationary point of problem (Rv) and use a sparse optimization problem with vertical linear complementarity constraints to demonstrate the efficiency of our algorithm on finding sparse solutions in practice., Comment: 29 pages, 3 figures

The prescribed-time stabilization problem for a general class of nonlinear systems with unknown input gain and appended dynamics (with unmeasured state) is addressed. Unlike the asymptotic stabilization problem, the prescribed-time stabilization objective requires convergence of the state to the origin in a finite time that can be arbitrarily picked (i.e., prescribed) by the control system designer irrespective of the initial condition of the system. The class of systems considered is allowed to have general nonlinear uncertain terms throughout the system dynamics as well as uncertain appended dynamics (that effectively generate a time-varying non-vanishing disturbance signal input into the nominal system). The control design is based on a time scale transformation, dynamic high-gain scaling, and adaptation dynamics with temporal forcing terms.

We introduce a twice differentiable augmented Lagrangian for nonlinear optimization with general inequality constraints and show that a strict local minimizer of the original problem is an approximate strict local solution of the augmented Lagrangian. A novel augmented Lagrangian method of multipliers (ALM) is then presented. Our method is originated from a generalization of the Hetenes-Powell augmented Lagrangian, and is a combination of the augmented Lagrangian and the interior-point technique. It shares a similar algorithmic framework with existing ALMs for optimization with inequality constraints, but it can use the second derivatives and does not depend on projections on the set of inequality constraints. In each iteration, our method solves a twice continuously differentiable unconstrained optimization subproblem on primal variables. The dual iterates, penalty and smoothing parameters are updated adaptively. The global and local convergence are analyzed. Without assuming any constraint qualification, it is proved that the proposed method has strong global convergence. The method may converge to either a Kurash-Kuhn-Tucker (KKT) point or a singular stationary point when the converging point is a minimizer. It may also converge to an infeasible stationary point of nonlinear program when the problem is infeasible. Furthermore, our method is capable of rapidly detecting the possible infeasibility of the solved problem. Under suitable conditions, it is locally linearly convergent to the KKT point, which is consistent with ALMs for optimization with equality constraints. The preliminary numerical experiments on some small benchmark test problems demonstrate our theoretical results., Comment: 31 pages

We present an O(nm) algorithm for all-pairs shortest paths computations in a directed graph with n nodes, m arcs, and nonnegative integer arc costs. This matches the complexity bound attained by Thorup [ 31 ] for the all-pairs problems in undirected graphs. The main insight is that shortest paths problems with approximately balanced directed cost functions can be solved similarly to the undirected case. The algorithm finds an approximately balanced reduced cost function in an O(m √ n log n ) preprocessing step. Using these reduced costs, every shortest path query can be solved in O(m) time using an adaptation of Thorup’s component hierarchy method. The balancing result can also be applied to the ℓ ∞ -matrix balancing problem.

We prove that every semidefinite moment relaxation of a polynomial optimization problem (POP) with a ball constraint can be reformulated as a semidefinite program involving a matrix with constant trace property (CTP). As a result such moment relaxations can be solved efficiently by first-order methods that exploit CTP, e.g., the conditional gradient-based augmented Lagrangian method. We also extend this CTP-exploiting framework to large-scale POPs with different sparsity structures. The efficiency and scalability of our framework are illustrated on second-order moment relaxations for various randomly generated quadratically constrained quadratic programs., 43 pages, 6 algorithms, 23 tables

This paper generalizes recent results by the authors on noninvasive model-reference adaptive control designs for control-based continuation of periodic orbits in periodically excited linear systems with matched uncertainties to a larger class of periodically excited nonlinear systems with matched uncertainties and known structure. A candidate adaptive feedback design is also proposed in the case of scalar problems with unmodeled nonlinearities. In the former case, rigorous analysis shows guaranteed performance bounds for the associated prediction and estimation errors. Together with an assumption of persistent excitation, there follows asymptotic convergence to periodic responses determined uniquely by an a priori unknown periodic reference input and independent of initial conditions, as required by the control-based continuation paradigm. In particular, when the reference input equals the sought periodic response, the steady-state control input vanishes. Identical conclusions follow for the case of scalar dynamics with unmodeled nonlinearities, albeit with slow rates of convergence. Numerical simulations validate the theoretical predictions for individual parameter values. Integration with the software package COCO demonstrate successful continuation along families of stable and unstable periodic orbits with a minimum of parameter tuning. The results expand the envelope of known noninvasive feedback strategies for use in experimental model validation and engineering design.

International audience; This work focuses on a parameter-free joint piecewise smooth image denoising and contour detection. Formulated as the minimization of a discrete Mumford-Shah functional and estimated via a theoretically grounded alternating minimization scheme, the bottleneck of such a variational approach lies in the need to fine-tune their hyperparameters, while not having access to ground truth data.To that aim, a Stein-like strategy providing optimal hyperparameters is designed, based on the minimization of an unbiased estimate of the quadratic risk.Efficient and automated minimization of the estimate of the risk crucially relies on an unbiased estimate of the gradient of the risk with respect to hyperparameters. Its practical implementation is performed using a forward differentiation of the alternating scheme minimizing the Mumford-Shah functional, requiring exact differentiation of the proximity operators involved. Intensive numerical experiments are performed on synthetic images with different geometry and noise levels, assessing the accuracy and the robustness of the proposed procedure.The resulting parameter-free piecewise-smooth estimation and contour detection procedure, not requiring prior image processing expertise nor annotated data, can then be applied to real-world images.

This paper deals with the integration of input move-blocking into the framework of suboptimal model predictive control. The blocked input parameterization is explicitly considered as a source of suboptimality. A straightforward integration approach is to hold back a manually generated stabilizing fallback solution in some buffer for the case that the optimizer does not find a better input move-blocked solution. An extended approach superimposes the manually generated stabilizing warm-start by the move-blocked control sequence and enables a stepwise improvement of the control performance. In addition, this contribution provides a detailed review of the literature on input move-blocked model predictive control and combines important results with the findings of suboptimal model predictive control. A numerical example supports the theoretical results and shows the effectiveness of the proposed approach., Added missing condition to Eq. (11); Added reference to difference inclusion (12) in Prop. 7 and Prop. 10; Removed typos

The paper studies coincidence points of parameterized set-valued mappings (multifunctions), which provide an extended framework to cover several important topics in variational analysis and optimization that include the existence of solutions of parameterized generalized equations, implicit function and fixed-point theorems, optimal value functions in parametric optimization, etc. Using the advanced machinery of variational analysis and generalized differentiation that furnishes complete characterizations of well-posedness properties of multifunctions, we establish a general theorem ensuring the existence of parameter-dependent coincidence point mappings with explicit error bounds for parameterized multifunctions between infinite-dimensional spaces. The obtained major result yields a new implicit function theorem and allows us to derive efficient conditions for semicontinuity and continuity of optimal value functions associated with parametric minimization problems subject to constraints governed by parameterized generalized equations.

In the d-dimensional online bin packing problem, d-dimensional cubes of positive sizes no larger than 1 are presented one by one to be assigned to positions in d-dimensional unit cube bins. In this work, we provide improved upper bounds on the asymptotic competitive ratio for square and cube bin packing problems, where our bounds do not exceed 2.0885 and 2.5735 for square and cube packing, respectively. To achieve these results, we adapt and improve a previously designed harmonic-type algorithm, and apply a different method for defining weight functions. We detect deficiencies in the state-of-the-art results by providing counter-examples to the current best algorithms and the analysis, where the claimed bounds were 2.1187 for square packing and 2.6161 for cube packing., Comment: WADS 2021

In this paper, we extend a recently established subgradient method for the computation of Riemannian metrics that optimizes certain singular value functions associated with dynamical systems. This extension is threefold. First, we introduce a projected subgradient method which results in Riemannian metrics whose parameters are confined to a compact convex set and we can thus prove that a minimizer exists; second, we allow inexact subgradients and study the effect of the errors on the computed metrics; and third, we analyze the subgradient algorithm for three different choices of step sizes: constant, exogenous and Polyak. The new methods are illustrated by application to dimension and entropy estimation of the H\'enon map., Comment: 30 pages, 5 figures