Search

We propose a bilinear decomposition for the Burer-Monteiro method and combine it with the standard Alternating Direction Method of Multipliers algorithm for semidefinite programming. Bilinear decomposition reduces the degree of the augmented Lagrangian from four to two, which makes each of the subproblems a quadratic programming and hence computationally efficient. Our approach is able to solve a class of large-scale SDPs with diagonal constraints. We prove that our ADMM algorithm converges globally to a first-order stationary point, and show by exploiting the negative curvature that the algorithm converges to a point within $O(1-1/r)$ of the optimal objective value. Additionally, the proximal variant of the algorithm can solve block-diagonally constrained SDPs with global convergence to a first-order stationary point. Numerical results show that both our ADMM algorithm and the proximal variant outperform the state-of-art Riemannian manifold algorithms and can reach the global optimum empirically.

Bilevel optimization problems, which are problems where two optimization problems are nested, have more and more applications in machine learning. In many practical cases, the upper and the lower objectives correspond to empirical risk minimization problems and therefore have a sum structure. In this context, we propose a bilevel extension of the celebrated SARAH algorithm. We demonstrate that the algorithm requires $\mathcal{O}((n+m)^{\frac12}\varepsilon^{-1})$ gradient computations to achieve $\varepsilon$-stationarity with $n+m$ the total number of samples, which improves over all previous bilevel algorithms. Moreover, we provide a lower bound on the number of oracle calls required to get an approximate stationary point of the objective function of the bilevel problem. This lower bound is attained by our algorithm, which is therefore optimal in terms of sample complexity.

We consider a setting in which one swarm of agents is to service or track a second swarm, and formulate an optimal control problem which trades off between the competing objectives of servicing and motion costs. We consider the continuum limit where large-scale swarms are modeled in terms of their time-varying densities, and where the Wasserstein distance between two densities captures the servicing cost. We show how this non-linear infinite-dimensional optimal control problem is intimately related to the geometry of Wasserstein space, and provide new results in the case of absolutely continuous densities and constant-in-time references. Specifically, we show that optimal swarm trajectories follow Wasserstein geodesics, while the optimal control tradeoff determines the time-schedule of travel along these geodesics. We briefly describe how this solution provides a basis for a model-predictive control scheme for tracking time-varying and real-time reference trajectories as well., Comment: 7 pages

In this work, we consider the dynamic unsplittable flow problem. This variation of the unsplittable flow problem has received little attention so far. The unsplittable flow problem is an NP-hard extension of the multi-commodity flow problem where each commodity sends its flow on only one path. In its dynamic version, this problem features several time steps and a penalty is paid when a commodity changes its path from one time step to the next. We present several mixed-integer linear programming formulations for this problem and compare the strength of their linear relaxation. These formulations are embedded in several solvers which are extensively compared on small to large instances. One of these formulations must be solved through a column generation process whose pricing problem is more difficult than those used in classical flow problems. We present limitations of the pricing schemes proposed in earlier works and describe two new schemes with a better worst-case complexity. Overall, this work lays a strong algorithmic baseline for the resolution of the dynamic unsplittable flow problem, proposes original formulations, and discusses the compared advantages of each, thus hopefully contributing a step towards a better understanding of this problem for both OR researchers and practical applications.

A novel space-discretized Finite Differences-based model reduction, introduced in (Liu,Guo,2020) is extended to the partial differential equations (PDE) model of a multi-layer Mead-Marcus-type sandwich beam with clamped-free boundary conditions. The PDE model describes transverse vibrations for a sandwich beam whose alternating outer elastic layers constrain viscoelastic core layers, which allow transverse shear. The major goal of this project is to design a single tip velocity sensor to control the overall dynamics on the beam. Since the spectrum of the PDE can not be constructed analytically, the so-called multipliers approach is adopted to prove that the PDE model is exactly observable with sub-optimal observation time. Next, the PDE model is reduced by the ``order-reduced'' Finite-Differences technique. This method does not require any type of filtering though the exact observability as $h\to 0$ is achieved by a constraint on the material constants. The main challenge here is the strong coupling of the shear dynamics of the middle layer with overall bending dynamics. This complicates the absorption of coupling terms in the discrete energy estimates. This is sharply different from a single-layer (Euler-Bernoulli) beam., Comment: 42 pages with several figures

We present a potent computational method for the solution of inverse problems in fluid mechanics. We consider inverse problems formulated in terms of a deterministic loss function that can accommodate data and regularization terms. We introduce a multigrid decomposition technique that accelerates the convergence of gradient-based methods for optimization problems with parameters on a grid. We incorporate this multigrid technique to the ODIL (Optimizing a DIscrete Loss) framework. The multiresolution ODIL (mODIL) accelerates by an order of magnitude the original formalism and improves the avoidance of local minima. Moreover, mODIL accommodates the use of automatic differentiation for calculating the gradients of the loss function, thus facilitating the implementation of the framework. We demonstrate the capabilities of mODIL on a variety of inverse and flow reconstruction problems: solution reconstruction for the Burgers equation, inferring conductivity from temperature measurements, and inferring the body shape from wake velocity measurements in three dimensions. We also provide a comparative study with the related, popular Physics-Informed Neural Networks (PINNs) method. We demonstrate that mODIL has three to five orders of magnitude lower computational cost than PINNs in benchmark problems including simple PDEs and lid-driven cavity problems. Our results suggest that mODIL is a very potent, fast and consistent method for solving inverse problems in fluid mechanics., Comment: 16 pages, 9 figures

In this technical report, we compare multiple reformulation techniques and solvers that can be used with the Julia package BilevelJuMP. We focus on the special case of Hyperparameter Tuning for Support Vector Regression. We describe a bilevel model for the problem in question. Then we present code for generating data and models that solve the problem. Finally, we present results and a brief analysis., Comment: arXiv admin note: substantial text overlap with arXiv:2205.02307

We propose an algorithm which appears to be the first bridge between the fields of conditional gradient methods and abs-smooth optimization. Our problem setting is motivated by various applications that lead to nonsmoothness, such as $\ell_1$ regularization, phase retrieval problems, or ReLU activation in machine learning. To handle the nonsmoothness in our problem, we propose a generalization to the traditional Frank-Wolfe gap and prove that first-order minimality is achieved when it vanishes. We derive a convergence rate for our algorithm which is {\em identical} to the smooth case. Although our algorithm necessitates the solution of a subproblem which is more challenging than the smooth case, we provide an efficient numerical method for its partial solution, and we identify several applications where our approach fully solves the subproblem. Numerical and theoretical convergence is demonstrated, yielding several conjectures.

Computational optimal transport (OT) has recently emerged as a powerful framework with applications in various fields. In this paper we focus on a relaxation of the original OT problem, the entropic OT problem, which allows to implement efficient and practical algorithmic solutions, even in high dimensional settings. This formulation, also known as the Schr\"odinger Bridge problem, notably connects with Stochastic Optimal Control (SOC) and can be solved with the popular Sinkhorn algorithm. In the case of discrete-state spaces, this algorithm is known to have exponential convergence; however, achieving a similar rate of convergence in a more general setting is still an active area of research. In this work, we analyze the convergence of the Sinkhorn algorithm for probability measures defined on the $d$-dimensional torus $\mathbb{T}_L^d$, that admit densities with respect to the Haar measure of $\mathbb{T}_L^d$. In particular, we prove pointwise exponential convergence of Sinkhorn iterates and their gradient. Our proof relies on the connection between these iterates and the evolution along the Hamilton-Jacobi-Bellman equations of value functions obtained from SOC-problems. Our approach is novel in that it is purely probabilistic and relies on coupling by reflection techniques for controlled diffusions on the torus., Comment: version accepted to COLT 2023

Graph learning from signals is a core task in Graph Signal Processing (GSP). One of the most commonly used models to learn graphs from stationary signals is SpecT. However, its practical formulation rSpecT is known to be sensitive to hyperparameter selection and, even worse, to suffer from infeasibility. In this paper, we give the first condition that guarantees the infeasibility of rSpecT and design a novel model (LogSpecT) and its practical formulation (rLogSpecT) to overcome this issue. Contrary to rSpecT, the novel practical model rLogSpecT is always feasible. Furthermore, we provide recovery guarantees of rLogSpecT, which are derived from modern optimization tools related to epi-convergence. These tools could be of independent interest and significant for various learning problems. To demonstrate the advantages of rLogSpecT in practice, a highly efficient algorithm based on the linearized alternating direction method of multipliers (L-ADMM) is proposed. The subproblems of L-ADMM admit closed-form solutions and the convergence is guaranteed. Extensive numerical results on both synthetic and real networks corroborate the stability and superiority of our proposed methods, underscoring their potential for various graph learning applications.

In this paper, we present a novel concept of the Fenchel conjugate for set-valued mappings and investigate its properties in finite and infinite dimensions. After establishing the fundamental properties of the Fenchel conjugate for set-valued mappings, we derive its main calculus rules in various settings. Our approach is geometric and draws inspiration from the successful application of this method by B.~S.~Mordukhovich and coauthors in variational and convex analysis. Subsequently, we demonstrate that our new findings for the Fenchel conjugate of set-valued mappings can be utilized to obtain many established calculus rules of convex generalized differentiation in both finite and infinite dimensions.

We present an efficient implementation of interior point methods for a family of nonsymmetric cones, including generalized power cones, power mean cones and relative entropy cones, by exploiting underlying low-rank and sparse properties of Hessians of homogeneous self-concordant barrier functions. We prove that the augmented linear system in our interior point method is sparse and quasi-definite, enabling the use of sparse LDL factorization with a dual scaling strategy for nonsymmetric cones. Numerical results show that our proposed implementation for nonsymmetric cones performs much faster than the state-of-art solvers for spare problems and scales well for large problems.

This paper introduces state polynomials, i.e., polynomials in noncommuting variables and formal states of their products. A state analog of Artin's solution to Hilbert's 17th problem is proved showing that state polynomials, positive over all matrices and matricial states, are sums of squares with denominators. Somewhat surprisingly, it is also established that a Krivine-Stengle Positivstellensatz fails to hold in the state polynomial setting. Further, archimedean Positivstellens\"atze in the spirit of Putinar and Helton-McCullough are presented leading to a hierarchy of semidefinite relaxations converging monotonically to the optimum of a state polynomial subject to state constraints. This hierarchy can be seen as a state analog of the Lasserre hierarchy for optimization of polynomials, and the Navascu\'es-Pironio-Ac\'in scheme for optimization of noncommutative polynomials. The motivation behind this theory arises from the study of correlations in quantum networks. Determining the maximal quantum violation of a polynomial Bell inequality for an arbitrary network is reformulated as a state polynomial optimization problem. Several examples of quadratic Bell inequalities in the bipartite and the bilocal tripartite scenario are analyzed. To reduce the size of the constructed SDPs, sparsity, sign symmetry and conditional expectation of the observables' group structure are exploited. To obtain the above-mentioned results, techniques from noncommutative algebra, real algebraic geometry, operator theory, and convex optimization are employed.

In this paper, we will consider a class of continuous-time stochastic control systems with both unknown nonlinear structure and unknown disturbances, and investigate the capability of the classical proportional-integral-derivative(PID) controller in tracking time-varying reference signals. First, under some suitable conditions on system nonlinear functions, reference signals, and unknown disturbances, we will show that PID controllers can be designed to globally stabilize such systems and ensure the boundedness of the tracking error. Analytic design formulae for PID gain matrices are also provided, which only involve some prior knowledge of the partial derivatives of system structural nonlinear functions. Besides, it will be shown that the steady-state tracking error hinges on three critical factors: i) the change rate of reference signals and external disturbances; ii) the intensity of random noises; iii) the selection of PID gains, and can be made arbitrarily small by choosing PID gains suitably large. Finally, by introducing a desired transient process which is shaped from the reference signal, we will present a new PID tuning rule, which can guarantee both nice steady-state and superior transient control performances.

In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.

This paper considers the problem of distributed multi-agent learning, where the global aim is to minimize a sum of local objective (empirical loss) functions through local optimization and information exchange between neighbouring nodes. We introduce a Newton-type fully distributed optimization algorithm, Network-GIANT, which is based on GIANT, a Federated learning algorithm that relies on a centralized parameter server. The Network-GIANT algorithm is designed via a combination of gradient-tracking and a Newton-type iterative algorithm at each node with consensus based averaging of local gradient and Newton updates. We prove that our algorithm guarantees semi-global and exponential convergence to the exact solution over the network assuming strongly convex and smooth loss functions. We provide empirical evidence of the superior convergence performance of Network-GIANT over other state-of-art distributed learning algorithms such as Network-DANE and Newton-Raphson Consensus.

We design and compute a class of optimal control problems for reaction-diffusion systems. They form mean field control problems related to multi-density reaction-diffusion systems. To solve proposed optimal control problems numerically, we first apply high-order finite element methods to discretize the space-time domain and then solve the optimal control problem using augmented Lagrangian methods (ALG2). Numerical examples, including generalized optimal transport and mean field control problems between Gaussian distributions and image densities, demonstrate the effectiveness of the proposed modeling and computational methods for mean field control problems involving reaction-diffusion equations/systems., Comment: 40 pages, 12 figures

We study a class of quasi-variational inequality problems defined over infinite dimensional Banach space and deduce sufficient conditions for ensuring solutions to such problems under the upper semi-continuity and pseudomonotonicity assumptions on the map defining the inequalities. The special structure of the quasi-variational inequality enables us to show the occurrence of solutions for such inequalities based on the classical existence theorem for variational inequalities. This special type of quasi-variational inequalities is motivated by the pure exchange economic problems and Radner equilibrium problems for sequential trading game. Further, we study the solvability of the specific class of quasi-variational inequalities on Banach spaces in which the constraint map may admit unbounded values. Finally, we demonstrate the occurrence of dynamic competitive equilibrium for a time-dependent pure exchange economy as an application.

This paper establishes the iteration-complexity of proximal bundle methods for solving hybrid (i.e., a blend of smooth and nonsmooth) weakly convex composite optimization (HWC-CO) problems. This is done in a unified manner by considering a proximal bundle framework (PBF) based on a generic bundle update scheme which includes various well-known bundle update schemes. In contrast to other wellknown stationary conditions in the context of HWC-CO, PBF uses a new stationarity measure which is easily verifiable and, at the same time, implies any of the former ones., Comment: 24 pages

The reconfigurable intelligent surface (RIS) is an emerging technology that changes how wireless networks are perceived, therefore its potential benefits and applications are currently under intense research and investigation. In this letter, we focus on electromagnetically consistent models for RISs inheriting from a recently proposed model based on mutually coupled loaded wire dipoles. While existing related research focuses on free-space wireless channels thereby ignoring interactions between RIS and scattering objects present in the propagation environment, we introduce an RIS-aided channel model that is applicable to more realistic scenarios, where the scattering objects are modeled as loaded wire dipoles. By adjusting the parameters of the wire dipoles, the properties of general natural and engineered material objects can be modeled. Based on this model, we introduce a provably convergent and efficient iterative algorithm that jointly optimizes the RIS and transmitter configurations to maximize the system sum-rate. Extensive numerical results show the net performance improvement provided by the proposed method compared with existing optimization algorithms., Comment: Submitted for publication

Quadratic programs with box constraints involve minimizing a possibly nonconvex quadratic function subject to lower and upper bounds on each variable. This is a well-known NP-hard problem that frequently arises in various applications. We focus on two convex relaxations, namely the RLT (Reformulation-Linearization Technique) relaxation and the SDP-RLT relaxation obtained by adding semidefinite constraints to the RLT relaxation. Both relaxations yield lower bounds on the optimal value of a quadratic program with box constraints. We present complete algebraic descriptions of the set of instances that admit exact RLT relaxations as well as those that admit exact SDP-RLT relaxations. We show that our descriptions can be converted into algorithms for efficiently constructing instances with exact or inexact relaxations., Comment: Technical Report, School of Mathematics, The University of Edinburgh, Edinburgh, EH9 3FD, United Kingdom

Numerical and symbolic methods for optimization are used extensively in engineering, industry, and finance. Various methods are used to reduce problems of interest to ones that are amenable to solution by such software. We develop a framework for designing and applying such reductions, using the Lean programming language and interactive proof assistant. Formal verification makes the process more reliable, and the availability of an interactive framework and ambient mathematical library provides a robust environment for constructing the reductions and reasoning about them.

The primary objective of this paper is to propose and analyze the notion of dual cones associated with the metric projection and generalized projection in Banach spaces. We show that the dual cones, related to the metric projection and generalized metric projection, lose many important properties in transitioning from Hilbert spaces to Banach spaces. We also propose and analyze the notions of faces and visions in Banach spaces and relate them to the metric projection and generalized projection. We provide many illustrative examples to give insight into the given results.

General Matrix Multiplication (GEMM) is a fundamental operation widely used in scientific computations. Its performance and accuracy significantly impact the performance and accuracy of applications that depend on it. One such application is semidefinite programming (SDP), and it often requires binary128 or higher precision arithmetic to solve problems involving SDP stably. However, only some processors support binary128 arithmetic, which makes SDP solvers generally slow. In this study, we focused on accelerating GEMM with binary128 arithmetic on field-programmable gate arrays (FPGAs) to enable the flexible design of accelerators for the desired computations. Our binary128 GEMM designs on a recent high-performance FPGA achieved approximately 90GFlops, 147x faster than the computation executed on a recent CPU with 20 threads for large matrices. Using our binary128 GEMM design on the FPGA, we successfully accelerated two numerical applications: LU decomposition and SDP problems, for the first time., Comment: 12 pages, 8 figures

The closed-loop stability and infinite-horizon performance of receding-horizon approximations are studied for non-stationary linear-quadratic regulator (LQR) problems. The approach is based on a lifted reformulation of the optimal control problem, under assumed uniform controllability and observability, leading to a strict contraction property of the corresponding Riccati operator. This property is leveraged to construct a stabilizing linear time-varying state-feedback approximation of the infinite-horizon optimal control policy, that also meets a performance-loss specification. Its synthesis involves only finite preview of the non-stationary problem data at each time step, over a sufficiently long prediction horizon., Comment: Submitted to IEEE Conference on Decision and Control, 2023

The clustering of data is one of the most important and challenging topics in data science. The minimum sum-of-squares clustering (MSSC) problem asks to cluster the data points into $k$ clusters such that the sum of squared distances between the data points and their cluster centers (centroids) is minimized. This problem is NP-hard, but there exist exact solvers that can solve such problem to optimality for small or medium size instances. In this paper, we use a branch-and-bound solver based on semidefinite programming relaxations called SOS-SDP to compute the optimum solutions of the MSSC problem for various $k$ and for multiple datasets, with real and artificial data, for which the data provider has provided ground truth clustering. Next, we use several extrinsic and intrinsic measures to evaluate how the optimum clustering and ground truth clustering matches, and how well these clusterings perform with respect to the criteria underlying the intrinsic measures. Our calculations show that the ground truth clusterings are generally far from the optimum solution to the MSSC problem. Moreover, the intrinsic measures evaluated on the ground truth clusterings are generally significantly worse compared to the optimum clusterings. However, when the ground truth clustering is in the form of convex sets, e.g., ellipsoids, that are well separated from each other, the ground truth clustering comes very close to the optimum clustering., Comment: 23 pages; 2 figures, 5 tables

In this work, we propose a robust optimization approach to mitigate the impact of uncertainties in particle precipitation. Our model incorporates partial differential equations, more particular nonlinear and nonlocal population balance equations to describe particle synthesis. The goal of the optimization problem is to design products with desired size distributions. Recognizing the impact of uncertainties, we extend the model to hedge against them. We emphasize the importance of robust protection to ensure the production of high-quality particles. To solve the resulting robust problem, we enhance a novel adaptive bundle framework for nonlinear robust optimization that integrates the exact method of moments approach for solving the population balance equations. Computational experiments performed with the integrated algorithm focus on uncertainties in the total mass of the system as it greatly influence the quality of the resulting product. Using realistic parameter values for quantum dot synthesis, we demonstrate the efficiency of our integrated algorithm. Furthermore, we find that the unprotected process fails to achieve the desired particle characteristics, even for small uncertainties, which highlights the necessity of the robust process. The latter consistently outperforms the unprotected process in quality of the obtained product, in particular in perturbed scenarios.

The team orienteering problem (TOP) determines a set of routes, each within a time or distance budget, which collectively visit a set of points of interest (POIs) such that the total score collected at those visited points are maximized. This paper proposes an extension of the TOP (ETOP) by allowing the POIs to be visited multiple times to accumulate scores. Such an extension is necessary for application scenarios like urban sensing where each POI needs to be continuously monitored, or disaster relief where certain locations need to be repeatedly covered. We present two approaches to solve the ETOP, one based on the adaptive large neighborhood search (ALNS) algorithm and the other being a bi-level matheuristic method. Sensitivity analyses are performed to fine-tune the algorithm parameters. Test results on complete graphs with different problem sizes show that: (1) both algorithms significantly outperform a greedy heuristic, with improvements ranging from 9.43% to 27.68%; and (2) while the ALNS-based algorithm slightly outperform the matheuristic in terms of solution optimality, the latter is far more computationally efficient, by 11 to 385 times faster. Finally, a real-world case study of VOCs sensing is presented and formulated as ETOP on a road network (incomplete graph), where the ALNS is outperformed by matheuristic in terms of optimality as the destroy and repair operators yield limited perturbation of existing solutions when constrained by a road network., Comment: 23 pages, 9 figures, 4 tables

For time inconsistent optimal control problems, a quite popular approach is the equilibrium approach, taken by the sophisticated agents. In this short note we construct a deterministic continuous time example where the unique equilibrium is dominated by another control. Therefore, in this situation it may not be wise to take the equilibrium strategy.

We study the elective surgery planning problem in a hospital with operation rooms shared by elective and emergency patients. This problem can be split in two distinct phases. First, a subset of patients to be operated in the next planning period has to be selected, and the selected patients have to be assigned to a block and a tentative starting time. Then, in the online phase of the problem, a policy decides how to insert the emergency patients in the schedule and may cancel planned surgeries. The overall goal is to minimize the expectation of a cost function representing the assignment of patient to blocks, case cancellations, overtime, waiting time and idle time. We model the offline problem by a two-stage stochastic program, and show that the second-stage costs can be replaced by a convex piecewise linear surrogate model that can be computed in a preprocessing step. This results in a mixed integer program which can be solved in a short amount of time, even for very large instances of the problem. We also describe a greedy policy for the online phase of the problem, and analyze the performance of our approach by comparing it to either heuristic methods or approaches relying on sampling average approximation (SAA) on a large set of benchmarking instances. Our simulations indicate that our approach can reduce the expected costs by as much as 20% compared to heuristic methods and is able to solve problems with $1000$ patients in about one minute, while SAA-approaches fail to obtain near-optimal solutions within 30 minutes, already for $100$ patients.

This paper presents a novel framework for adaptive portfolio management that combines a dynamic Black-Litterman optimization with the general factor model and Elastic Net regression. This integrated approach allows us to systematically generate investors' views and mitigate potential estimation errors. Our empirical results demonstrate that this combined approach can lead to computational advantages as well as promising trading performances., Comment: 9 pages, 6 figures

In this paper, we study the predict-then-optimize problem where the output of a machine learning prediction task is used as the input of some downstream optimization problem, say, the objective coefficient vector of a linear program. The problem is also known as predictive analytics or contextual linear programming. The existing approaches largely suffer from either (i) optimization intractability (a non-convex objective function)/statistical inefficiency (a suboptimal generalization bound) or (ii) requiring strong condition(s) such as no constraint or loss calibration. We develop a new approach to the problem called \textit{maximum optimality margin} which designs the machine learning loss function by the optimality condition of the downstream optimization. The max-margin formulation enjoys both computational efficiency and good theoretical properties for the learning procedure. More importantly, our new approach only needs the observations of the optimal solution in the training data rather than the objective function, which makes it a new and natural approach to the inverse linear programming problem under both contextual and context-free settings; we also analyze the proposed method under both offline and online settings, and demonstrate its performance using numerical experiments., Comment: to be published in ICML 2023

We investigate pathwise turnpike behavior of discrete-time stochastic linear-quadratic optimal control problems. Our analysis is based on a novel strict dissipativity notion for such problems, in which a stationary stochastic process replaces the optimal steady state of the deterministic setting. The analytical findings are illustrated by a numerical example.

We consider the optimization problem of the form $\min_{x \in \mathbb{R}^d} f(x) \triangleq \mathbb{E}_{\xi} [F(x; \xi)]$, where the component $F(x;\xi)$ is $L$-mean-squared Lipschitz but possibly nonconvex and nonsmooth. The recently proposed gradient-free method requires at most $\mathcal{O}( L^4 d^{3/2} \epsilon^{-4} + \Delta L^3 d^{3/2} \delta^{-1} \epsilon^{-4})$ stochastic zeroth-order oracle complexity to find a $(\delta,\epsilon)$-Goldstein stationary point of objective function, where $\Delta = f(x_0) - \inf_{x \in \mathbb{R}^d} f(x)$ and $x_0$ is the initial point of the algorithm. This paper proposes a more efficient algorithm using stochastic recursive gradient estimators, which improves the complexity to $\mathcal{O}(L^3 d^{3/2} \epsilon^{-3}+ \Delta L^2 d^{3/2} \delta^{-1} \epsilon^{-3})$., Comment: ICML 2023

This paper investigates the stochastic linear-quadratic (LQ, for short) optimal control problems with non-Markovian regime switching in a finite time horizon where the state equation is multi-dimensional. Similar to the classical stochastic LQ problems, we establish the relationship between the stochastic LQ optimal control problems with non-Markovian regime switching and the related extended stochastic Riccati equations. To solve the extended stochastic Riccati equations, we construct a monotone Piccard iterative sequence and present the link bewteen this sequence and solutions of a family of forward-backward stochastic differential equations. Relying on $L^p$ estimates for FBSDEs, we show that the extended stochastic Riccati equation has a solution. This partially addresses one question left in Hu et al. (Ann. Appl. Probab. 32(1): 426-460, 2022). Finally, the stochastic LQ optimal control problems with non-Markovian regime switching is solved.

Over the past two decades, multiobejective gradient descent methods have received increasing attention due to the seminal work of Fliege and Svaiter. Recently, Chen et al. pointed out that imbalances among objective functions can lead to a small stepsize in Fliege and Svaiter's method, which significantly decelerates the convergence. To address the issue, Chen et al. propose the Barzilai-Borwein descent method for multiobjective optimization (BBDMO). Their work demonstrated that BBDMO achieves better stepsize and performance compared to Fliege and Svaiter's method. However, a theoretical explanation for the superiority of BBDMO over the previous method has been open. In this paper, we extend Chen et al.'s method to composite cases and propose two types of Barzilai-Borwein proximal gradient methods (BBPGMO). Moreover, we prove that the convergence rates of BBPGMO are $O(\frac{1}{\sqrt{k}})$, $O(\frac{1}{k})$, and $O(r^{k})(0

Tikhonov regularization involves minimizing the combination of a data discrepancy term and a regularizing term, and is the standard approach for solving inverse problems. The use of non-convex regularizers, such as those defined by trained neural networks, has been shown to be effective in many cases. However, finding global minimizers in non-convex situations can be challenging, making existing theory inapplicable. A recent development in regularization theory relaxes this requirement by providing convergence based on critical points instead of strict minimizers. This paper investigates convergence rates for the regularization with critical points using Bregman distances. Furthermore, we show that when implementing near-minimization through an iterative algorithm, a finite number of iterations is sufficient without affecting convergence rates.

Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + \rho \frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}$ of the current point $x_t$. Existing studies prove convergence of SAM for smooth functions, but they do so by assuming decaying perturbation size $\rho$ and/or no gradient normalization in $y_t$, which is detached from practice. To address this gap, we study deterministic/stochastic versions of SAM with practical configurations (i.e., constant $\rho$ and gradient normalization in $y_t$) and explore their convergence properties on smooth functions with (non)convexity assumptions. Perhaps surprisingly, in many scenarios, we find out that SAM has limited capability to converge to global minima or stationary points. For smooth strongly convex functions, we show that while deterministic SAM enjoys tight global convergence rates of $\tilde \Theta(\frac{1}{T^2})$, the convergence bound of stochastic SAM suffers an inevitable additive term $O(\rho^2)$, indicating convergence only up to neighborhoods of optima. In fact, such $O(\rho^2)$ factors arise for stochastic SAM in all the settings we consider, and also for deterministic SAM in nonconvex cases; importantly, we prove by examples that such terms are unavoidable. Our results highlight vastly different characteristics of SAM with vs. without decaying perturbation size or gradient normalization, and suggest that the intuitions gained from one version may not apply to the other., Comment: 39 pages. v2 adds/corrects a couple of citations

Sinkhorn algorithm has been used pervasively to approximate the solution to optimal transport (OT) and unbalanced optimal transport (UOT) problems. However, its practical application is limited due to the high computational complexity. To alleviate the computational burden, we propose a novel importance sparsification method, called Spar-Sink, to efficiently approximate entropy-regularized OT and UOT solutions. Specifically, our method employs natural upper bounds for unknown optimal transport plans to establish effective sampling probabilities, and constructs a sparse kernel matrix to accelerate Sinkhorn iterations, reducing the computational cost of each iteration from $O(n^2)$ to $\widetilde{O}(n)$ for a sample of size $n$. Theoretically, we show the proposed estimators for the regularized OT and UOT problems are consistent under mild regularity conditions. Experiments on various synthetic data demonstrate Spar-Sink outperforms mainstream competitors in terms of both estimation error and speed. A real-world echocardiogram data analysis shows Spar-Sink can effectively estimate and visualize cardiac cycles, from which one can identify heart failure and arrhythmia. To evaluate the numerical accuracy of cardiac cycle prediction, we consider the task of predicting the end-systole time point using the end-diastole one. Results show Spar-Sink performs as well as the classical Sinkhorn algorithm, requiring significantly less computational time., Comment: Accepted by Journal of Machine Learning Research

The control of manufacturing processes must satisfy high quality and efficiency requirements while meeting safety requirements. A broad spectrum of monitoring and control strategies, such as model- and optimization-based controllers, are utilized to address these issues. Driven by rising demand for flexible yet energy and resource-efficient operations existing approaches are challenged due to high uncertainties and changes. Machine learning algorithms are becoming increasingly important in tackling these challenges, especially due to the growing amount of available data. The ability for automatic adaptation and learning from human operators offer new opportunities to increase efficiency yet provide flexible operation. Combining machine learning algorithms with safe or robust controls offers novel reliable operation methods. This chapter highlights ways to fuse machine learning and control for the safe and improved operation of chemical and biochemical processes. We outline and summarize both - learning models for control and learning the control components. We offer a general overview, including a literature review, to provide a guideline for utilizing machine learning techniques in control structures.

In the past several decades, the world's economy has become increasingly globalized. On the other hand, there are also ideas advocating the practice of ``buy local'', by which people buy locally produced goods and services rather than those produced farther away. In this paper, we establish a mathematical theory of real price that determines the optimal global versus local spending of an agent which achieves the agent's optimal tradeoff between spending and obtained utility. Our theory of real price depends on the asymptotic analysis of a Markov chain transition probability matrix related to the network of producers and consumers. We show that the real price of a product or service can be determined from the involved Markov chain matrix, and can be dramatically different from the product's label price. In particular, we show that the label prices of products and services are often not ``real'' or directly ``useful'': given two products offering the same myopic utility, the one with lower label price may not necessarily offer better asymptotic utility. This theory shows that the globality or locality of the products and services does have different impacts on the spending-utility tradeoff of a customer. The established mathematical theory of real price can be used to determine whether to adopt or not to adopt certain artificial intelligence (AI) technologies from an economic perspective., Comment: 16 pages, 3 figures

The Noisy-SGD algorithm is widely used for privately training machine learning models. Traditional privacy analyses of this algorithm assume that the internal state is publicly revealed, resulting in privacy loss bounds that increase indefinitely with the number of iterations. However, recent findings have shown that if the internal state remains hidden, then the privacy loss might remain bounded. Nevertheless, this remarkable result heavily relies on the assumption of (strong) convexity of the loss function. It remains an important open problem to further relax this condition while proving similar convergent upper bounds on the privacy loss. In this work, we address this problem for DP-SGD, a popular variant of Noisy-SGD that incorporates gradient clipping to limit the impact of individual samples on the training process. Our findings demonstrate that the privacy loss of projected DP-SGD converges exponentially fast, without requiring convexity or smoothness assumptions on the loss function. In addition, we analyze the privacy loss of regularized (unprojected) DP-SGD. To obtain these results, we directly analyze the hockey-stick divergence between coupled stochastic processes by relying on non-linear data processing inequalities.

Gradient methods have become mainstream techniques for Bi-Level Optimization (BLO) in learning fields. The validity of existing works heavily rely on either a restrictive Lower-Level Strong Convexity (LLSC) condition or on solving a series of approximation subproblems with high accuracy or both. In this work, by averaging the upper and lower level objectives, we propose a single loop Bi-level Averaged Method of Multipliers (sl-BAMM) for BLO that is simple yet efficient for large-scale BLO and gets rid of the limited LLSC restriction. We further provide non-asymptotic convergence analysis of sl-BAMM towards KKT stationary points, and the comparative advantage of our analysis lies in the absence of strong gradient boundedness assumption, which is always required by others. Thus our theory safely captures a wider variety of applications in deep learning, especially where the upper-level objective is quadratic w.r.t. the lower-level variable. Experimental results demonstrate the superiority of our method., Comment: To be published in ICML 2023

For multi-view learning problems coming from reality, it is common that the data are incomplete due to lost files, poor data entry and so on. Thus, we consider the incomplete multi-view clustering problem. In the clustering process, the influence of each view of one object may be different and the data are often low rank. On the other hand, self-representation subspace method works well on multi-view clustering problem. Taking these factors into account, we propose an adaptive weighted self-representation subspace clustering (AWSR) method for solving the incomplete multi-view clustering problem. The AWSR method employs a weight matrix to measure the contribution of each missing view, and seeks for the low rank coefficient matrix to enhance the clustering effect. Moreover, the regularization terms are properly designed to eliminate the possible data noise and keep the convexity property of the objective function. The convergence analysis is provided for the alternate minimization algorithm of the AWSR model. Meanwhile, numerical experiments of AWSR method are implemented on five real datasets. The numerical performance shows that the AWSR method is effective and delivers superior results when compared to other six common approaches considering the clustering accuracy, normalized mutual information and purity.

The paper provides a new approach to the determination of a single state value for stochastic output feedback problems using paradigms from Model Predictive Control, particularly the distinction between open-loop and closed-loop control and between deterministic optimal control and stochastic optimal control. The State Selection Algorithm is presented and relies on given dynamics and constraints, a nominal deterministic state-feedback controller, and a sampling based method to select the best state value, based on optimizing a prescribed finite-horizon performance function, over the available candidates provided by a particle filter. The cost function is minimized over the horizon with controls determined by the nominal controller and the selected states. So, the minimization is performed not over the selection of the control other than through the choice of state value to use. The algorithm applies generally to nonlinear stochastic systems and relies on Monte Carlo sampling and averaging. However, in linear quadratic polyhedrally constrained cases the technique reduces to a quadratic program for the state value. The algorithm is evaluated in a set of computational examples, which illustrate its efficacy and limitations. Numerical aspects and the opportunity for parallelization are discussed. The examples demonstrate the algorithm operating, in closed-loop with its attendant particle filter, over the long horizon., Comment: 15 pages, 10 figures, 2 algorithms

In this paper, we consider a distributed Bayesian Nash equilibrium (BNE) seeking problem in incomplete-information aggregative games, which is a generalization of Bayesian games and deterministic aggregative games. We handle the aggregation function for distributed incomplete-information situations. Since the feasible strategies are infinite-dimensional functions and lie in a non-compact set, the continuity of types brings barriers to seeking equilibria. To this end, we discretize the continuous types and then prove that the equilibrium of the derived discretized model is an $\epsilon$-BNE. On this basis, we propose a distributed algorithm for an $\epsilon$-BNE and further prove its convergence.

Derivative-free optimization problems are optimization problems where derivative information is unavailable. The least Frobenius norm updating quadratic interpolation model function is one of the essential under-determined model functions for model-based derivative-free trust-region methods. This article proposes derivative-free optimization with transformed objective functions and gives a trust-region method with the least Frobenius norm model. The model updating formula is based on Powell's formula. The method shares the same framework with those for problems without transformations, and its query scheme is given. We propose the definitions related to optimality-preserving transformations to understand the interpolation model in our method. We prove the existence of model optimality-preserving transformations beyond translation transformation. The necessary and sufficient condition for such transformations is given. The affine transformation with a positive multiplication coefficient is not model optimality-preserving. We also analyze the corresponding least Frobenius norm updating model and its interpolation error when the objective function is affinely transformed. Convergence property of a provable algorithmic framework containing our model is given. Numerical results of solving test problems and a real-world problem with the implementation NEWUOA-Trans show that our method can successfully solve most problems with objective optimality-preserving transformations, even though such transformations will change the optimality of the model function. To our best knowledge, this is the first work providing the model-based derivative-free algorithm and analysis for transformed problems with the function evaluation oracle (not the function-value comparison oracle). This article also proposes the ``moving-target'' optimization problem., Comment: 38 pages, Derivative-free Optimization with Transformed Objective Functions (DFOTO) and the Algorithm Based on Least Frobenius Norm Updating Quadratic Model

Hybrid power plants comprising renewable power sources and electrolyzers are envisioned to play a key role in accelerating the transition towards decarbonization. It is common in the current literature to use simplified operational models for electrolyzers. It is still an open question whether this is a good practice, and if not, when a more detailed operational model is necessary. This paper answers it by assessing the impact of adding different levels of electrolyzer details, i.e., physics and operational constraints, to the optimal dispatch problem of a hybrid power plant in the day-ahead time stage. Our focus lies on the number of operating states (on, off, standby) as well as the number of linearization segments used for approximating the non-linear hydrogen production curve. For that, we develop several mixed-integer linear models, each representing a different level of operational details. We conduct a thorough comparative ex-post performance analysis under different price conditions, wind farm capacities, and minimum hydrogen demand requirements, and discuss under which operational circumstances a detailed model is necessary. In particular, we provide a case under which a simplified model, compared to a detailed one, results in a decrease in profit of 1.8% and hydrogen production of 13.5% over a year. The key lesson learned is that a detailed model potentially earns a higher profit in circumstances under which the electrolyzer operates with partial loading. This could be the case for a certain range of electricity and hydrogen prices, or limited wind power availability. The detailed model also provides a better estimation of true hydrogen production, facilitating the logistics required., Comment: Accepted for IEEE PES PowerTech 2023

Derivative-free optimization methods are numerical methods for optimization problems in which no derivative information is used. Such optimization problems are widely seen in many real applications. One particular class of derivative-free optimization algorithms is trust-region algorithms based on quadratic models given by under-determined interpolation. Different techniques in updating the quadratic model from iteration to iteration will give different interpolation models. In this paper, we propose a new way to update the quadratic model by minimizing the $H^2$ norm of the difference between neighbouring quadratic models. Motivation for applying the $H^2$ norm is given, and theoretical properties of our new updating technique are also presented. Projection in the sense of $H^2$ norm and interpolation error analysis of our model function are proposed. We obtain the coefficients of the quadratic model function by using the KKT conditions. Numerical results show advantages of our model, and the derivative-free algorithms based on our least $H^2$ norm updating quadratic model functions can solve test problems with fewer function evaluations than algorithms based on least Frobenius norm updating., Comment: 24 pages

In convex optimization, first-order optimization methods efficiently minimizing function values have been a central subject study since Nesterov's seminal work of 1983. Recently, however, Kim and Fessler's OGM-G and Lee et al.'s FISTA-G have been presented as alternatives that efficiently minimize the gradient magnitude instead. In this paper, we present H-duality, which represents a surprising one-to-one correspondence between methods efficiently minimizing function values and methods efficiently minimizing gradient magnitude. In continuous-time formulations, H-duality corresponds to reversing the time dependence of the dissipation/friction term. To the best of our knowledge, H-duality is different from Lagrange/Fenchel duality and is distinct from any previously known duality or symmetry relations. Using H-duality, we obtain a clearer understanding of the symmetry between Nesterov's method and OGM-G, derive a new class of methods efficiently reducing gradient magnitudes of smooth convex functions, and find a new composite minimization method that is simpler and faster than FISTA-G.