Search

The core of a good model is in its ability to focus only on important information that reflects the basic patterns and consistencies, thus pulling out a clear, noise-free signal from the dataset. This necessitates using a simplified model defined by fewer parameters. The importance of theoretical foundations becomes clear in this context, as this paper relies on established results from the domain of advanced sparse optimization, particularly those addressing nonlinear differentiable functions. The need for such theoretical foundations is further highlighted by the trend that as computational power for training NNs increases, so does the complexity of the models in terms of a higher number of parameters. In practical scenarios, these large models are often simplified to more manageable versions with fewer parameters. Understanding why these simplified models with less number of parameters remain effective raises a crucial question. Understanding why these simplified models with fewer parameters remain effective raises an important question. This leads to the broader question of whether there is a theoretical framework that can clearly explain these empirical observations. Recent developments, such as establishing necessary conditions for the convergence of iterative hard thresholding (IHT) to a sparse local minimum (a sparse method analogous to gradient descent) are promising. The remarkable capacity of the IHT algorithm to accurately identify and learn the locations of nonzero parameters underscores its practical effectiveness and utility. This paper aims to investigate whether the theoretical prerequisites for such convergence are applicable in the realm of neural network (NN) training by providing justification for all the necessary conditions for convergence. Then, these conditions are validated by experiments on a single-layer NN, using the IRIS dataset as a testbed.

The Iterative Hard Thresholding (IHT) algorithm has been considered extensively as an effective deterministic algorithm for solving sparse optimizations. The IHT algorithm benefits from the information of the batch (full) gradient at each point and this information is a crucial key for the convergence analysis of the generated sequence. However, this strength becomes a weakness when it comes to machine learning and high dimensional statistical applications because calculating the batch gradient at each iteration is computationally expensive or impractical. Fortunately, in these applications the objective function has a summation structure that can be taken advantage of to approximate the batch gradient by the stochastic mini-batch gradient. In this paper, we study the mini-batch Stochastic IHT (SIHT) algorithm for solving the sparse optimizations. As opposed to previous works where increasing and variable mini-batch size is necessary for derivation, we fix the mini-batch size according to a lower bound that we derive and show our work. To prove stochastic convergence of the objective value function we first establish a critical sparse stochastic gradient descent property. Using this stochastic gradient descent property we show that the sequence generated by the stochastic mini-batch SIHT is a supermartingale sequence and converges with probability one. Unlike previous work we do not assume the function to be a restricted strongly convex. To the best of our knowledge, in the regime of sparse optimization, this is the first time in the literature that it is shown that the sequence of the stochastic function values converges with probability one by fixing the mini-batch size for all steps.

Sparse optimization receives increasing attention in many applications such as compressed sensing, variable selection in regression problems, and recently neural network compression in machine learning. For example, the problem of compressing a neural network is a bi-level, stochastic, and nonconvex problem that can be cast into a sparse optimization problem. Hence, developing efficient methods for sparse optimization plays a critical role in applications. The goal of this paper is to develop analytical techniques for general, large size sparse optimization problems using the hard thresholding operator. To this end, we study the iterative hard thresholding (IHT) algorithm, which has been extensively studied in the literature because it is scalable, fast, and easily implementable. In spite of extensive research on the IHT scheme, we develop several new techniques that not only recover many known results but also lead to new results. Specifically, we first establish a new and critical gradient descent property of the hard thresholding (HT) operator. Our gradient descent result can be related to the distance between points that are sparse. Also, our gradient descent property allows one to study the IHT when the stepsize is less than or equal to 1/L, where L>0 is the Lipschitz constant of the gradient of an objective function. Note that the existing techniques in the literature can only handle the case when the stepsize is strictly less than 1/L. By exploiting this we introduce and study HT-stable and HT-unstable stationary points and show no matter how close an initialization is to a HT-unstable stationary point (saddle point in sparse sense), the IHT sequence leaves it. Finally, we show that no matter what sparse initial point is selected, the IHT sequence converges if the function values at HT-stable stationary points are distinct.

CAV platooning technology has received considerable attention in the past few years, driven by the next generation smart transportation systems. Unlike most of the existing platooning methods that focus on linear vehicle dynamics of CAVs, this paper considers nonlinear vehicle dynamics and develops fully distributed optimization based CAV platooning control schemes via the model predictive control (MPC) approach for a possibly heterogeneous CAV platoon. The nonlinear vehicle dynamics leads to several major difficulties in distributed algorithm development and control analysis and design. Specifically, the underlying MPC optimization problem is nonconvex and densely coupled. Further, the closed loop dynamics becomes a time-varying nonlinear system subject to external perturbations, making closed loop stability analysis rather complicated. To overcome these difficulties, we formulate the underlying MPC optimization problem as a locally coupled, albeit nonconvex, optimization problem and develop a sequential convex programming based fully distributed scheme for a general MPC horizon. Such a scheme can be effectively implemented for real-time computing using operator splitting methods. To analyze the closed loop stability, we apply various tools from global implicit function theorems, stability of linear time-varying systems, and Lyapunov theory for input-to-state stability to show that the closed loop system is locally input-to-state stable uniformly in all small coefficients pertaining to the nonlinear dynamics. Numerical tests on homogeneous and heterogeneous CAV platoons demonstrate the effectiveness of the proposed fully distributed schemes and CAV platooning control.

Mean-reverting portfolios with few assets, but high variance, are of great interest for investors in financial markets. Such portfolios are straightforwardly profitable because they include a small number of assets whose prices not only oscillate predictably around a long-term mean but also possess enough volatility. Roughly speaking, sparsity minimizes trading costs, volatility provides arbitrage opportunities, and mean-reversion property equips investors with ideal investment strategies. Finding such favorable portfolios can be formulated as a nonconvex quadratic optimization problem with an additional sparsity constraint. To the best of our knowledge, there is no method for solving this problem and enjoying favorable theoretical properties yet. In this paper, we develop an effective two-stage algorithm for this problem. In the first stage, we apply a tailored penalty decomposition method for finding a stationary point of this nonconvex problem. For a fixed penalty parameter, the block coordinate descent method is utilized to find a stationary point of the associated penalty subproblem. In the second stage, we improve the result from the first stage via a greedy scheme that solves restricted nonconvex quadratically constrained quadratic programs (QCQPs). We show that the optimal value of such a QCQP can be obtained by solving their semidefinite relaxations. Numerical experiments on S\&P 500 are conducted to demonstrate the effectiveness of the proposed algorithm.

This paper develops distributed optimization based, platoon centered CAV car following schemes, motivated by the recent interest in CAV platooning technologies. Various distributed optimization or control schemes have been developed for CAV platooning. However, most existing distributed schemes for platoon centered CAV control require either centralized data processing or centralized computation in at least one step of their schemes, referred to as partially distributed schemes. In this paper, we develop fully distributed optimization based, platoon centered CAV platooning control under the linear vehicle dynamics via the model predictive control approach with a general prediction horizon. These fully distributed schemes do not require centralized data processing or centralized computation through the entire schemes. To develop these schemes, we propose a new formulation of the objective function and a decomposition method that decomposes a densely coupled central objective function into the sum of several locally coupled functions whose coupling satisfies the network topology constraint. We then exploit the formulation of locally coupled optimization and operator splitting methods to develop fully distributed schemes. Control design and stability analysis is carried out to achieve desired traffic transient performance and asymptotic stability. Numerical tests demonstrate the effectiveness of the proposed fully distributed schemes and CAV platooning control.

This paper develops column partition based distributed schemes for a class of large-scale convex sparse optimization problems, e.g., basis pursuit (BP), LASSO, basis pursuit denosing (BPDN), and their extensions, e.g., fused LASSO. We are particularly interested in the cases where the number of (scalar) decision variables is much larger than the number of (scalar) measurements, and each agent has limited memory or computing capacity such that it only knows a small number of columns of a measurement matrix. These problems in consideration are densely coupled and cannot be formulated as separable convex programs using column partition. To overcome this difficulty, we consider their dual problems which are separable or locally coupled. Once a dual solution is attained, it is shown that a primal solution can be found from the dual of corresponding regularized BP-like problems under suitable exact regularization conditions. A wide range of existing distributed schemes can be exploited to solve the obtained dual problems. This yields two-stage column partition based distributed schemes for LASSO-like and BPDN-like problems; the overall convergence of these schemes is established using sensitivity analysis techniques. Numerical results illustrate the effectiveness of the proposed schemes.

Matching pursuit, especially its orthogonal version (OMP) and variations, is a greedy algorithm widely used in signal processing, compressed sensing, and sparse modeling. Inspired by constrained sparse signal recovery, this paper proposes a constrained matching pursuit algorithm and develops conditions for exact support and vector recovery on constraint sets via this algorithm. We show that exact recovery via constrained matching pursuit not only depends on a measurement matrix but also critically relies on a constraint set. We thus identify an important class of constraint sets, called coordinate projection admissible set, or simply CP admissible sets; analytic and geometric properties of these sets are established. We study exact vector recovery on convex, CP admissible cones for a fixed support. We provide sufficient exact recovery conditions for a general support as well as necessary and sufficient recovery conditions when a support has small size. As a byproduct, we construct a nontrivial counterexample to a renowned necessary condition of exact recovery via the OMP for a support of size three. Moreover, using the properties of convex CP admissible sets and convex optimization techniques, we establish sufficient conditions for uniform exact recovery on convex CP admissible sets in terms of the restricted isometry-like constant and the restricted orthogonality-like constant.

In this paper, we study the solution uniqueness of an individual feasible vector of a class of convex optimization problems involving convex piecewise affine functions and subject to general polyhedral constraints. This class of problems incorporates many important polyhedral constrained $\ell_1$ recovery problems arising from sparse optimization, such as basis pursuit, LASSO, and basis pursuit denoising, as well as polyhedral gauge recovery. By leveraging the max-formulation of convex piecewise affine functions and convex analysis tools, we develop dual variables based necessary and sufficient uniqueness conditions via simple and yet unifying approaches; these conditions are applied to a wide range of $\ell_1$ minimization problems under possible polyhedral constraints. An effective linear program based scheme is proposed to verify solution uniqueness conditions. The results obtained in this paper not only recover the known solution uniqueness conditions in the literature by removing restrictive assumptions but also yield new uniqueness conditions for much broader constrained $\ell_1$-minimization problems.

Motivated by $\ell_p$-optimization arising from sparse optimization, high dimensional data analytics and statistics, this paper studies sparse properties of a wide range of $p$-norm based optimization problems with $p > 1$, including generalized basis pursuit, basis pursuit denoising, ridge regression, and elastic net. It is well known that when $p > 1$, these optimization problems lead to less sparse solutions. However, the quantitative characterization of the adverse sparse properties is not available. In this paper, by exploiting optimization and matrix analysis techniques, we give a systematic treatment of a broad class of $p$-norm based optimization problems for a general $p > 1$ and show that optimal solutions to these problems attain full support, and thus have the least sparsity, for almost all measurement matrices and measurement vectors. Comparison to $\ell_p$-optimization with $0 < p \le 1$ and implications to robustness are also given. These results shed light on analysis and computation of general $p$-norm based optimization problems in various applications.

Inspired by shape constrained estimation under general nonnegative derivative constraints, this paper considers the B-spline approximation of constrained functions and studies the asymptotic performance of the constrained B-spline estimator. By invoking a deep result in B-spline theory (known as de Boor's conjecture) first proved by A. Shardin as well as other new analytic techniques, we establish a critical uniform Lipschitz property of the B-spline estimator subject to arbitrary nonnegative derivative constraints under the $\ell_\infty$-norm with possibly non-equally spaced design points and knots. This property leads to important asymptotic analysis results of the B-spline estimator, e.g., the uniform convergence and consistency on the entire interval under consideration. The results developed in this paper not only recover the well-studied monotone and convex approximation and estimation as special cases, but also treat general nonnegative derivative constraints in a unified framework and open the door for the constrained B-spline approximation and estimation subject to a broader class of shape constraints.

Estimation of convex functions finds broad applications in engineering and science, while convex shape constraint gives rise to numerous challenges in asymptotic performance analysis. This paper is devoted to minimax optimal estimation of univariate convex functions from the H\"older class in the framework of shape constrained nonparametric estimation. Particularly, the paper establishes the optimal rate of convergence in two steps for the minimax sup-norm risk of convex functions with the H\"older order between one and two. In the first step, by applying information theoretical results on probability measure distance, we establish the minimax lower bound under the supreme norm by constructing a novel family of piecewise quadratic convex functions in the H\"older class. In the second step, we develop a penalized convex spline estimator and establish the minimax upper bound under the supreme norm. Due to the convex shape constraint, the optimality conditions of penalized convex splines are characterized by nonsmooth complementarity conditions. By exploiting complementarity methods, a critical uniform Lipschitz property of optimal spline coefficients in the infinity norm is established. This property, along with asymptotic estimation techniques, leads to uniform bounds for bias and stochastic errors on the entire interval of interest. This further yields the optimal rate of convergence by choosing the suitable number of knots and penalty value. The present paper provides the first rigorous justification of the optimal minimax risk for convex estimation under the supreme norm.

This paper considers the development of spatially adaptive smoothing splines for the estimation of a regression function with non-homogeneous smoothness across the domain. Two challenging issues that arise in this context are the evaluation of the equivalent kernel and the determination of a local penalty. The roughness penalty is a function of the design points in order to accommodate local behavior of the regression function. It is shown that the spatially adaptive smoothing spline estimator is approximately a kernel estimator. The resulting equivalent kernel is spatially dependent. The equivalent kernels for traditional smoothing splines are a special case of this general solution. With the aid of the Green's function for a two-point boundary value problem, the explicit forms of the asymptotic mean and variance are obtained for any interior point. Thus, the optimal roughness penalty function is obtained by approximately minimizing the asymptotic integrated mean square error. Simulation results and an application illustrate the performance of the proposed estimator.

This paper addresses the problem of estimating a convex regression function under both the sup-norm risk and the pointwise risk using B-splines. The presence of the convex constraint complicates various issues in asymptotic analysis, particularly uniform convergence analysis. To overcome this difficulty, we establish the uniform Lipschitz property of optimal spline coefficients in the $\ell_\infty$-norm by exploiting piecewise linear and polyhedral theory. Based upon this property, it is shown that this estimator attains optimal rates of convergence on the entire interval of interest over the H\"older class under both the risks. In addition, adaptive estimates are constructed under both the sup-norm risk and the pointwise risk when the exponent of the H\"older class is between one and two. These estimates achieve a maximal risk within a constant factor of the minimax risk over the H\"older class.

This paper addresses asymptotic properties of general penalized spline estimators with an arbitrary B-spline degree and an arbitrary order difference penalty. The estimator is approximated by a solution of a linear differential equation subject to suitable boundary conditions. It is shown that, in certain sense, the penalized smoothing corresponds approximately to smoothing by the kernel method. The equivalent kernels for both inner points and boundary points are obtained with the help of Green's functions of the differential equation. Further, the asymptotic normality is established for the estimator at interior points. It is shown that the convergence rate is independent of the degree of the splines, and the number of knots does not affect the asymptotic distribution, provided that it tends to infinity fast enough.