Search

Model training algorithms which observe a small portion of the training set in each computational step are ubiquitous in practical machine learning, and include both stochastic and online optimization methods. In the vast majority of cases, such algorithms typically observe the training samples via the gradients of the cost functions the samples incur. Thus, these methods exploit are the slope of the cost functions via their first-order approximations. To address limitations of gradient-based methods, such as sensitivity to step-size choice in the stochastic setting, or inability to exploit small function variability in the online setting, several streams of research attempt to exploit more information about the cost functions than just their gradients via the well-known proximal operators. However, implementing such methods in practice poses a challenge, since each iteration step boils down to computing the proximal operator, which may not be as easy as computing a gradient. In this work we devise a novel algorithmic framework, which exploits convex duality theory to achieve both algorithmic efficiency and software modularity of proximal operator implementations, in order to make experimentation with incremental proximal optimization algorithms accessible to a larger audience of researchers and practitioners, by reducing the gap between their theoretical description in research papers and their use in practice. We provide a reference Python implementation for the framework developed in this paper as an open source library at on GitHub (https://github.com/alexshtf/inc%5fprox%5fpt/releases/tag/prox%5fpt%5fpaper) Shtoff (Efficient implementation of incremental proximal point methods arXiv:2205.01457, 2024), along with examples which demonstrate our implementation on a variety of problems, and reproduce the numerical experiments in this paper. The pure Python reference implementation is not necessarily the most efficient, but is a basis for creating efficient implementations by combining Python with a native backend. [ABSTRACT FROM AUTHOR]

A class of proximal gradient-type algorithm for bilevel nonlinear nondifferentiable programming problems with smooth substructure is developed in this paper. The original problem is approximately reformulated by explicit slow control technique to a parameterized family function which makes full use of the information of smoothness. At each iteration, we only need to calculate one proximal point analytically or with low computational cost. We prove that the accumulation iterations generated by the algorithms are solutions of the original problem. Moreover, some results of complexity of the algorithms are presented in convergence analysis. Numerical experiments are implemented to verify the efficiency of the proximal gradient algorithms for solving this kind of bilevel programming problems. [ABSTRACT FROM AUTHOR]

We introduce a geometrically transparent strict saddle property for nonsmooth functions. This property guarantees that simple proximal algorithms on weakly convex problems converge only to local minimizers, when randomly initialized. We argue that the strict saddle property may be a realistic assumption in applications, since it provably holds for generic semi-algebraic optimization problems. [ABSTRACT FROM AUTHOR]