Search

Many researchers witness the efficiency of decomposition-based evolutionary multi- and many-objective optimization algorithms in finding quality solutions on problems with regular Pareto front. These decomposition-based evolutionary approaches profoundly depend on aggregation methods and weight vectors. Such algorithms fail to guarantee the diversity in the population, especially when dealing with irregular Pareto front, because some of the weight vectors become ineffective during simulation. Efforts are being made over time to frequently adopt the position of weight vectors at the rate of increased computation burden, in order to achieve improved solutions. Hence an alternative mechanism has been suggested in this paper. Instead of changing the weight vector, a better treatment on environmental selection is proposed that could effectively balance both convergence and diversity in solving both multi- and many-objective optimization problems. The higher rate of performance of the proposed algorithm has been validated over the comparative permanence with six popular state-of-the-art algorithms in terms of hypervolume, complemented with winning ratio using t-test. It is concluded in the latter part of this paper that the proposed method performs better while solving test suite problems with irregular Pareto fronts. Moreover, it is more prone towards better solutions with an increase in the number of objective functions in MaOPs. [ABSTRACT FROM AUTHOR]

This paper studies the convergence properties of the well-known message-passing algorithm for convex optimisation. Under the assumption of pairwise separability and scaled diagonal dominance, asymptotic convergence is established and a simple bound for the convergence rate is provided for message-passing. In comparison with previous results, our results do not require the given convex program to have known convex pairwise components and that our bound for the convergence rate is tighter and simpler. When specialised to quadratic optimisation, we generalise known results by providing a very simple bound for the convergence rate. [ABSTRACT FROM AUTHOR]

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

With the increase in the number of optimization objectives, balancing the convergence and diversity in evolutionary multiobjective optimization becomes more intractable. So far, a variety of evolutionary algorithms have been proposed to solve many-objective optimization problems (MaOPs) with more than three objectives. Most of the existing algorithms, however, find difficulties in simultaneously counterpoising convergence and diversity during the whole evolutionary process. To address the issue, this paper proposes to solve MaOPs via multistage evolutionary search. To be specific, a two-stage evolutionary algorithm is developed, where the convergence and diversity are highlighted during different search stages to avoid the interferences between them. The first stage pushes multiple subpopulations with different weight vectors to converge to different areas of the Pareto front. After that, the nondominated solutions coming from each subpopulation are selected for generating a new population for the second stage. Moreover, a new environmental selection strategy is designed for the second stage to balance the convergence and diversity close to the Pareto front. This selection strategy evenly divides each objective dimension into a number of intervals, and then one solution having the best convergence in each interval will be retained. To assess the performance of the proposed algorithm, 48 benchmark functions with 7, 10, and 15 objectives are used to make comparisons with five representative many-objective optimization algorithms. [ABSTRACT FROM AUTHOR]

Convergence and diversity are interdependently handled during the evolutionary process by most existing many-objective evolutionary algorithms (MaOEAs). In such a design, the degraded performance of one would deteriorate the other, and only solutions with both are able to improve the performance of MaOEAs. Unfortunately, it is not easy to constantly maintain a population of solutions with both convergence and diversity. In this paper, an MaOEA based on two independent stages is proposed for effectively solving many-objective optimization problems (MaOPs), where the convergence and diversity are addressed in two independent and sequential stages. To achieve this, we first propose a nondominated dynamic weight aggregation method by using a genetic algorithm, which is capable of finding the Pareto-optimal solutions for MaOPs with concave, convex, linear and even mixed Pareto front shapes, and then these solutions are employed to learn the Pareto-optimal subspace for the convergence. Afterward, the diversity is addressed by solving a set of single-objective optimization problems with reference lines within the learned Pareto-optimal subspace. To evaluate the performance of the proposed algorithm, a series of experiments are conducted against six state-of-the-art MaOEAs on benchmark test problems. The results show the significantly improved performance of the proposed algorithm over the peer competitors. In addition, the proposed algorithm can focus directly on a chosen part of the objective space if the preference area is known beforehand. Furthermore, the proposed algorithm can also be used to effectively find the nadir points. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Copyright of Application Research of Computers / Jisuanji Yingyong Yanjiu is the property of Application Research of Computers Edition and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Chongqing University of Posts & Telecommunications (Natural Science Edition) is the property of Chongqing University of Posts & Telecommunications and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This paper proposes a general framework for determining the effect of communication delays on the convergence of certain distributed frequency regulation (DFR) protocols for prosumer-based energy systems, where prosumers are serving as balancing areas. DFR relies on iterative and distributed optimization algorithms to obtain an optimal feedback law for frequency regulation. But, it is, in general, hard to know beforehand how many iterations suffice to ensure stability. This paper develops a framework to determine a lower bound on the number of iterations required for two distributed optimization protocols. This allows prosumers to determine whether they can compute stabilizing control strategies within an acceptable time frame by taking communication delays into account. The efficacy of the method is demonstrated on two realistic power systems. [ABSTRACT FROM PUBLISHER]

In this paper, a novel algorithm “firework optimization” (FO) is introduced, as well as its hybridization with generating set search (GSS). FO is an exploration algorithm, which intends to explore a vast region within the search space in order to determine regions that may contain a regional optimum. As the regions with potential are defined, GSS, the exploitation algorithm is applied to determine whether regions determined by FO contain a local or a global optimum. The validity of FO and GSS hybridization algorithm is verified using Branin and Six-Hump Camel test functions, and the hybridization algorithm is applied to an interior permanent magnet synchronous motor, in order to minimize torque ripple. [ABSTRACT FROM AUTHOR]

Multiobjective evolutionary algorithm based on decomposition (MOEA/D) decomposes a multiobjective optimization problem into a set of scalar optimization subproblems and optimizes them in a collaborative manner. Subproblems and solutions are two sets of agents that naturally exist in MOEA/D. The selection of promising solutions for subproblems can be regarded as a matching between subproblems and solutions. Stable matching, proposed in economics, can effectively resolve conflicts of interests among selfish agents in the market. In this paper, we advocate the use of a simple and effective stable matching (STM) model to coordinate the selection process in MOEA/D. In this model, subproblem agents can express their preferences over the solution agents, and vice versa. The stable outcome produced by the STM model matches each subproblem with one single solution, and it tradeoffs convergence and diversity of the evolutionary search. Comprehensive experiments have shown the effectiveness and competitiveness of our MOEA/D algorithm with the STM model. We have also demonstrated that user-preference information can be readily used in our proposed algorithm to find a region that decision makers are interested in. [ABSTRACT FROM AUTHOR]

Many automation or manufacturing systems are too complex to be modeled by analytical approaches and can only resort to fast-running simulation. Stochastic Nelder–Mead simplex method (SNM) is a newly developed methodology for simulation optimization with expected-value-based objective functions. Quantile, as an important alternative to the usual expected value, provides additional information about the distribution of system performance. In particular, it is useful in describing the tail behavior of the distribution. In this paper, we exploit the structure of SNM and extend it to solve simulation optimization problems with quantile-based objective functions and inequality constraints. The proposed method, called SNM-QC, utilizes the same search strategy as SNM but further incorporates effective quantile estimation techniques and penalty function approaches to solve the problem. We prove that SNM-QC has the desirable global convergence guarantee, i.e., the algorithm is guaranteed to converge to the true optima with probability one. One advantage of SNM-QC is that it is a direct search method that determines the moving direction simply by comparing a set of solutions rather than estimating gradient, thus it can handle many practical problems where gradient does not exist or is difficult to estimate. An extensive numerical study shows that the performance of SNM-QC is promising compared to the existing heuristics. Two illustrative applications are provided in the end to demonstrate the viability of SNM-QC in practical settings. [ABSTRACT FROM AUTHOR]

We design an efficient robust multi-group multicast beamforming scheme for massive multiple-input multiple-output (MIMO) systems. Assuming only estimates of the channel covariance matrices are available at the base station with a bounded error, we formulate the robust quality-of-service (QoS) problem, which is to minimize the transmit power subject to the worst-case minimum signal-to-interference-plus-noise-ratio (SINR) guarantee. We directly solve the worst-case SINR problem and convert the robust QoS constraint into a number of non-convex constraints. Based on the recent convergence result of the alternating direction method of multipliers (ADMM) for non-convex problems, we develop an ADMM-based fast algorithm to directly tackle the reformulated non-convex problem with a convergence guarantee. The algorithm contains two layers of ADMM procedures. We design the outer-layer ADMM to decompose the problem into three convex subproblems and solve them alternatingly. We further develop an inner-layer consensus-ADMM-based algorithm to efficiently solve one subproblem. By exploring each subproblem structure and developing the special optimization techniques, we obtain closed-form or semi-closed-form solutions to each subproblem. These results lead to a fast iterative algorithm, which is guaranteed to converge to a stationary point of the original robust QoS problem. Simulation shows that our proposed algorithm provides a favorable performance compared with existing alternative methods with magnitudes of computational complexity reduction. [ABSTRACT FROM AUTHOR]

When solving multi-objective optimization problems, an important issue is how to promote convergence and distribution of solution set simultaneously. To address the above issue, a novel optimization algorithm, named as multi-objective modified teaching-learning-based optimization (MOMTLBO), is proposed. Firstly, a grouping teaching strategy based on pareto dominance relationship is proposed to strengthen the convergence efficiency. Afterward, a diversified learning strategy is presented to enhance the distribution. Meanwhile, differential operations are incorporated to the proposed algorithm. By the above process, the search ability of the algorithm can be encouraged. Additionally, a set of well-known benchmark test functions including ten complex problems proposed for CEC2009 is used to verify the performance of the proposed algorithm. The results show that MOMTLBO exhibits competitive performance against other comparison algorithms. Finally, the proposed algorithm is applied to the aerodynamic optimization of airfoils. [ABSTRACT FROM AUTHOR]

This paper proposes a performance assessment-based system identification of different practically useful open-loop and closed-loop Wiener systems using an evolutionary computational algorithm named as brain storm optimization (BSO) algorithm. Different performance measures of the estimation process in practical scenario, i.e., accuracy; precision; consistency; and computational time, are measured with a properly selected fitness function which is the output mean square error (MSE) between the desired and the estimated outputs. Bias and variance of the output MSE have been found negligible for each plant model to show the accuracy and consistency limits, and the corresponding statistical test results have been shown to establish the consistency of the BSO-based identification approach. Efficient identification of each plant under a noisy environment ensures the robustness and the stability of the proposed evolutionary-based identification approach with BSO. The BSO-based optimum MSE values, corresponding estimated parameter values, computational times and the other statistical information are all compared and are found to be superior to those of the other approaches reported earlier. [ABSTRACT FROM AUTHOR]

Phase retrieval is an inverse problem which consists in recovering an unknown signal from a set of absolute squared projections. Recently, gradient descent algorithms have been developed to solve this problem. However, their optimization cost functions are non-convex and non-smooth. To address the non-smoothness of the cost function, some of these methods use truncation thresholds to calculate a truncated step gradient direction. But, the truncation requires designing parameters to obtain a desired performance in the phase recovery, which drastically modifies the search direction update, increasing the sampling complexity. Therefore, this paper develops the Phase Retrieval Smoothing Conjugate Gradient method (PR-SCG) which uses a smoothing function to retrieve the signal. PR-SCG is based on the smooth-ing projected gradient method which is useful for non-convex optimization problems. PR-SCG uses a nonlinear conjugate gradient of the smoothing function as the search direction to accelerate the convergence. Furthermore, the incremental Stochastic Smoothing Phase Retrieval algorithm (SSPR) is developed. SSPR involves a single equation per iteration which results in a simple, scalable, and fast approach useful when the size of the signal is large. Also, it is shown that SSPR converges linearly to the true signal, up to a global unimodular constant. Additionally, the proposed methods do not require truncation parameters. Simulation results are provided to validate the efficiency of PR-SCG and SSPR compared to existing phase retrieval algorithms. It is shown that PR-SCG and SSPR are able to reduce the number of measurements and iterations to recover the phase, compared with recently developed algorithms. [ABSTRACT FROM AUTHOR]

We present a hybrid metaheuristic optimization algorithm for solving economic dispatch problems in power systems. The proposed algorithm, based on bat algorithm, combines chaotic map and random black hole model together. Chaotic map is used to prevent premature convergence, and the random black hole model is helpful not only in avoiding premature convergence, but also in increasing the global search ability, enlarging exploitation area and accelerating convergence speed. The pseudocode and related parameters of the proposed algorithm are also given in this paper. Different from other related works, the costs of conventional thermal generators and random wind power are both included in the cost function because of the increasing penetration of wind power. The proposed algorithm has no requirement on the convexity or continuous differentiability of the cost function, although the effect on fuel cost, caused by the underestimation and overestimation of wind power, is included. This makes it feasible to take more practical nonlinear constraints into account, such as prohibited operating zones and ramp rate limits. Three test cases are given to illustrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, based on a new smoothing function, the general box constrained variational inequalities are solved by a smoothing Newton algorithm with a nonmonotone line search. The proposed algorithm is proved to be globally and locally superlinearly convergent under suitable assumptions. The preliminary numerical results are reported. [ABSTRACT FROM AUTHOR]

In this paper, a convergence proof for the recently proposed cost function optimization sparse possibilistic c-means (SPCM) algorithm is provided. Specifically, it is shown that the algorithm will converge to one of the local minima of its associated cost function. It is also shown that similar convergence results can be derived for the well-known possibilistic c-means (PCM) algorithm proposed by Krishnapuram and Keller, 1996, if we view it as a special case of SPCM. Note that the convergence results for PCM are stronger than those established in previous works. [ABSTRACT FROM PUBLISHER]

In this paper, we propose an iteratively reweightedminimization algorithm (IRL1 algorithm) for solving the elasticminimization problem. We prove that any sequence generated by the IRL1 algorithm is bounded and asymptotically regular. We also prove that the sequence is convergent for any rationaland the limit is a stationary point of the elasticminimization problem. Moreover, under certain conditions, we present an error bound between the limit point of convergent sequence and the sparse solution of underdetermined linear system. Numerical experiments on sparse vector recovery are presented to demonstrate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In this paper we analyze the method of centers for generalized fractional programs, with further insights. The introduced method is based on a different parametric auxiliary problem than Dinkelbach's type. With the help of this auxiliary parametric problem, we present a new dual for convex generalized fractional programs. We then propose an algorithm to solve this problem, and subsequently the original primal program. The proposed algorithm generates a sequence of dual values that converges from below to the optimal value. The method also generates a bounded sequence of dual solutions whose every accumulation point is a solution of the dual problem. The rate of convergence is shown to be at least linear. In the penultimate section, we specialize the results obtained for the linear case. The computational results show that the different variants of our algorithms, primal as well as dual, are competitive. [ABSTRACT FROM AUTHOR]

This article is concerned with the design of a super-twisting algorithm (STA) based sliding mode controller for permanent magnet synchronous motor (PMSM) speed regulation system under the self-triggered mechanism. By using the strict Lyapunov function approach, it is shown that the tracking error converges to an ultimate domain within the finite-time sense under the proposed self-triggered STA. A feasible self-triggered strategy is designed for both cases with and without external perturbation. Moreover, a nonlinear optimization problem is formulated in terms of the tradeoff between the ultimate domain and the communication burden. The optimized STA gains are obtained by solving the above-formulated optimization problem via a particle swarm optimization algorithm. Finally, the applicability of the proposed self-triggered STA for PMSM is verified by simulation and experiment results. [ABSTRACT FROM AUTHOR]

In this paper, brain storm optimization (BSO)-based efficient identification approach has been applied to different types of stable and practically useful Nonlinear Auto Regressive Moving Average with exogenous noise (NARMAX) Hammerstein models with various performance criteria-based assessments. Different performance measures of the estimation process like accuracy, precision and consistency have been established to ensure the general applicability and practical usefulness of the proposed approach. The accuracy and the precision of the parameter estimation are established with the corresponding bias and variance information, while the consistency has been justified with the help of hypothesis test results. BSO-based optimum values of the output mean square errors and the parameters and their corresponding convergences ensure the stability and robustness of the proposed identification scheme. The comparative studies of the performance of the BSO algorithm with the other basic evolutionary algorithms have been reported with optimum values of the mean square errors, estimated values of the parameters, corresponding computational times and hypothesis test outcomes. Copyright © 2016 John Wiley & Sons, Ltd. [ABSTRACT FROM AUTHOR]

We study distributed composite optimization over networks: agents minimize a sum of smooth (strongly) convex functions–the agents’ sum-utility–plus a nonsmooth (extended-valued) convex one. We propose a general unified algorithmic framework for such a class of problems and provide a convergence analysis leveraging the theory of operator splitting. Distinguishing features of our scheme are: (i) When each of the agent’s functions is strongly convex, the algorithm converges at a linear rate, whose dependence on the agents’ functions and network topology is decoupled; (ii) When the objective function is convex (but not strongly convex), similar decoupling as in (i) is established for the coefficient of the proved sublinear rate. This also reveals the role of function heterogeneity on the convergence rate. (iii) The algorithm can adjust the ratio between the number of communications and computations to achieve a rate (in terms of computations) independent on the network connectivity; and (iv) A by-product of our analysis is a tuning recommendation for several existing (non-accelerated) distributed algorithms yielding provably faster (worst-case) convergence rate for the class of problems under consideration. [ABSTRACT FROM AUTHOR]

Optimization is an important research field, especially in engineering, physical sciences, and economics. The main purpose of optimization is to maximize the profit and minimize the cost of production as well as the loss of the system. Evolutionary computation algorithms, such as the genetic algorithm and the particle swarm optimization (PSO) algorithm have been successfully employed in solving various optimization problems. Owing to its application potential and promising performance in discovering the optimal solution, the PSO algorithm has been recognized as a powerful optimization technique and attracted an ever-increasing interest in the evolutionary computation community. In this article, a novel $N$ -state Markovian jumping PSO (NS-MJPSO) algorithm is presented where the velocity updating equation is adjusted based on the state evolution governed by a Markov chain. The performance of the proposed NS-MJPSO algorithm is evaluated via some widely used mathematical benchmark functions. The experimental results demonstrate that the developed NS-MJPSO algorithm outperforms some currently popular PSO algorithms on the widely used benchmark functions. [ABSTRACT FROM AUTHOR]

Based on our previous work, this paper presents a dynamic hybrid framework, called DyHF, for solving constrained optimization problems. This framework consists of two major steps: global search model and local search model. In the global and local search models, differential evolution serves as the search engine, and Pareto dominance used in multiobjective optimization is employed to compare the individuals in the population. Unlike other existing methods, the above two steps are executed dynamically according to the feasibility proportion of the current population in this paper, with the purpose of reasonably distributing the computational resource for the global and local search during the evolution. The performance of DyHF is tested on 22 benchmark test functions. The experimental results clearly show that the overall performance of DyHF is highly competitive with that of a number of state-of-the-art approaches from the literature. [ABSTRACT FROM AUTHOR]

Differential evolution (DE) algorithms compose an efficient type of evolutionary algorithm (EA) for the global optimization domain. Although it is well known that the population structure has a major influence on the behavior of EAs, there are few works studying its effect in DE algorithms. In this paper, we propose and analyze several DE variants using different panmictic and decentralized population schemes. As it happens for other EAs, we demonstrate that the population scheme has a marked influence on the behavior of DE algorithms too. Additionally, a new operator for generating the mutant vector is proposed and compared versus a classical one on all the proposed population models. After that, a new heterogeneous decentralized DE algorithm combining the two studied operators in the best performing studied population structure has been designed and evaluated. In total, 13 new DE algorithms are presented and evaluated in this paper. Summarizing our results, all the studied algorithms are highly competitive compared to the state-of-the-art DE algorithms taken from the literature for most considered problems, and the best ones implement a decentralized population. With respect to the population structure, the proposed decentralized versions clearly provide a better performance compared to the panmictic ones. The new mutation operator demonstrates a faster convergence on most of the studied problems versus a classical operator taken from the DE literature. Finally, the new heterogeneous decentralized DE is shown to improve the previously obtained results, and outperform the compared state-of-the-art DEs. [ABSTRACT FROM PUBLISHER]

This paper proposes the design of robust repetitive control with time-varying sampling periods. First, it develops a new frequency domain method to design a low-order, stable, robust, and causal IIR repetitive compensator using an optimization method to achieve fast convergence and high tracking accuracy. As such, a new stable and causal repetitive controller can be implemented independently to reduce the design complexity. The comprehensive analysis and comparison study are presented. Then, this paper extends the method to design a robust repetitive controller, which compensates time-varying periodic signals in a known range. A complete series of experiments is successfully carried out to demonstrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Stochastic approximation techniques play an important role in solving many problems encountered in machine learning or adaptive signal processing. In these contexts, the statistics of the data are often unknown a priori or their direct computation is too intensive, and they have thus to be estimated online from the observed signals. For batch optimization of an objective function being the sum of a data fidelity term and a penalization (e.g., a sparsity promoting function), Majorize-Minimize (MM) methods have recently attracted much interest since they are fast, highly flexible, and effective in ensuring convergence. The goal of this paper is to show how these methods can be successfully extended to the case when the data fidelity term corresponds to a least squares criterion and the cost function is replaced by a sequence of stochastic approximations of it. In this context, we propose an online version of an MM subspace algorithm and we study its convergence by using suitable probabilistic tools. Simulation results illustrate the good practical performance of the proposed algorithm associated with a memory gradient subspace, when applied to both nonadaptive and adaptive filter identification problems. [ABSTRACT FROM PUBLISHER]

In this paper, we study Stochastic Composite Optimization (SCO) for sparse learning that aims to learn a sparse solution from a composite function. Most of the recent SCO algorithms have already reached the optimal expected convergence rate \mathcal O(1/\lambda T) , but they often fail to deliver sparse solutions at the end either due to the limited sparsity regularization during stochastic optimization (SO) or due to the limitation in online-to-batch conversion. Even when the objective function is strongly convex, their high probability bounds can only attain \mathcal O(\sqrt\log (1/\delta)/T) with $\delta$ is the failure probability, which is much worse than the expected convergence rate. To address these limitations, we propose a simple yet effective two-phase Stochastic Composite Optimization scheme by adding a novel powerful sparse online-to-batch conversion to the general Stochastic Optimization algorithms. We further develop three concrete algorithms, OptimalSL, LastSL and AverageSL, directly under our scheme to prove the effectiveness of the proposed scheme. Both the theoretical analysis and the experiment results show that our methods can really outperform the existing methods at the ability of sparse learning and at the meantime we can improve the high probability bound to approximately \mathcal {O}(\log (\log (T)/\delta)/\lambda T) . [ABSTRACT FROM AUTHOR]

Nemirovski and Yudin proposed the mirror descent algorithm at the late 1970s to solve convex optimization problems. This method is suitable to solve huge-scale optimization problems. In the paper, we describe a new version of the mirror descent method to solve variational inequalities with pseudomonotone operators. The method can be interpreted as a modification of Popov's two-step algorithm with the use of Bregman projections on the feasible set. We prove the convergence of the sequences generated by the proposed method. [ABSTRACT FROM AUTHOR]

Trust region methods are a class of numerical methods for optimization. Unlike line search type methods where a line search is carried out in each iteration, trust region methods compute a trial step by solving a trust region subproblem where a model function is minimized within a trust region. Due to the trust region constraint, nonconvex models can be used in trust region subproblems, and trust region algorithms can be applied to nonconvex and ill-conditioned problems. Normally it is easier to establish the global convergence of a trust region algorithm than that of its line search counterpart. In the paper, we review recent results on trust region methods for unconstrained optimization, constrained optimization, nonlinear equations and nonlinear least squares, nonsmooth optimization and optimization without derivatives. Results on trust region subproblems and regularization methods are also discussed. [ABSTRACT FROM AUTHOR]

In Part II of this paper, also in this issue, we carried out a detailed mean-square-error analysis of the performance of asynchronous adaptation and learning over networks under a fairly general model for asynchronous events including random topologies, random link failures, random data arrival times, and agents turning on and off randomly. In this Part III, we compare the performance of synchronous and asynchronous networks. We also compare the performance of decentralized adaptation against centralized stochastic-gradient (batch) solutions. Two interesting conclusions stand out. First, the results establish that the performance of adaptive networks is largely immune to the effect of asynchronous events: the mean and mean-square convergence rates and the asymptotic bias values are not degraded relative to synchronous or centralized implementations. Only the steady-state mean-square-deviation suffers a degradation in the order of \nu, which represents the small step-size parameters used for adaptation. Second, the results show that the adaptive distributed network matches the performance of the centralized solution. These conclusions highlight another critical benefit of cooperation by networked agents: cooperation does not only enhance performance in comparison to stand-alone single-agent processing, but it also endows the network with remarkable resilience to various forms of random failure events and is able to deliver performance that is as powerful as batch solutions. [ABSTRACT FROM PUBLISHER]

The discrete cross-entropy optimization algorithm iteratively samples solutions according to a probability density on the solution space. The density is adapted to the good solutions observed in the present sample before producing the next sample. The adaptation is controlled by a so-called smoothing parameter. We generalize this model by introducing a flexible concept of feasibility and desirability into the sampling process. In this way, our model covers several other optimization procedures, in particular the ant-based algorithms. The focus of this paper is on some theoretical properties of these algorithms. We examine the first hitting time \( \tau \) of an optimal solution and give conditions on the smoothing parameter for \( \tau \) to be finite with probability one. For a simple test case we show that runtime can be polynomially bounded in the problem size with a probability converging to 1. We then investigate the convergence of the underlying density and of the sampling process. We show, in particular, that a constant smoothing parameter, as it is often used, makes the sample process converge in finite time, freezing the optimization at a single solution that need not be optimal. Moreover, we define a smoothing sequence that makes the density converge without freezing the sample process and that still guarantees the reachability of optimal solutions in finite time. This settles an open question from the literature. [ABSTRACT FROM AUTHOR]

This paper presents an efficient algorithm for solving a balanced regularization problem in the frame-based image restoration. The balanced regularization is usually formulated as a minimization problem, involving an \ell2 data-fidelity term, an \ell1 regularizer on sparsity of frame coefficients, and a penalty on distance of sparse frame coefficients to the range of the frame operator. In image restoration, the balanced regularization approach bridges the synthesis-based and analysis-based approaches, and balances the fidelity, sparsity, and smoothness of the solution. Our proposed algorithm for solving the balanced optimal problem is based on a variable splitting strategy and the classical alternating direction method. This paper shows that the proposed algorithm is fast and efficient in solving the standard image restoration with balanced regularization. More precisely, a regularized version of the Hessian matrix of the \ell2 data-fidelity term is involved, and by exploiting the related fast tight Parseval frame and the special structures of the observation matrices, the regularized Hessian matrix can perform quite efficiently for the frame-based standard image restoration applications, such as circular deconvolution in image deblurring and missing samples in image inpainting. Numerical simulations illustrate the efficiency of our proposed algorithm in the frame-based image restoration with balanced regularization. [ABSTRACT FROM AUTHOR]

Multi-objective evolutionary algorithms (MOEAs) have proven their effectiveness in solving two or three objective problems. However, recent research shows that Pareto-based MOEAs encounter selection difficulties facing many similar non-dominated solutions in dealing with many-objective problems. In order to reduce the selection pressure and improve the diversity, we propose achievement scalarizing function sorting strategy to make strength Pareto evolutionary algorithm suitable for many-objective optimization. In the proposed algorithm, we adopt density estimation strategy to redefine a new fitness value of a solution, which can select solution with good convergence and distribution. In addition, a clustering method is used to classify the non-dominated solutions, and then, an achievement scalarizing function ranking method is designed to layer different frontiers and eliminate redundant solutions in the environment selection stage, thus ensuring the convergence and diversity of non-dominant solutions. The performance of the proposed algorithm is validated and compared with some state-of-the-art algorithms on a number of test problems with 3, 5, 8, 10 objectives. Experimental studies demonstrate that the proposed algorithm shows very competitive performance. [ABSTRACT FROM AUTHOR]

This paper introduces a set of skeleton operators for characterizing topologies evolving in a bit-array represented structural topology optimization problem. It is shown that the design generally converges to a stable skeleton fairly early in the optimization process. It is observed that further optimization is more about finding optimal gross shape for the various branches of the converged skeleton and the bit-array representation is not appropriate. A two-phase approach to topology optimization is proposed in which the first phase, where bit-array is used to represent the topology, ends with the detection of stabilization of skeleton, and the second phase proceeds further with the geometry based representation that directly addresses gross variation in shape of the branches of the converged skeleton. Genetic Algorithm has been used for optimization in both the phases. The efficiency and effectiveness of the use of skeleton operators and geometric variables for identification of convergence in the first phase and optimization in the second phase respectively is demonstrated. [ABSTRACT FROM AUTHOR]

The constrained global optimization problem being considered, a modified integral-level set method was illustrated based on Chew-Zheng’s paper on Integral Global Optimization and Wu’s paper on Implementable Algorithm Convergence of Modified Integral-Level Set Method for Global Optimization Problem. It has two characters: 1) Each phase must construct a new function which has the same global optimal value as that of primitive objective function; 2) Comparing it with Zheng’s method, solving level set procedure is avoided. An implementable algorithm also is given and it is proved that this algorithm is convergent. [ABSTRACT FROM AUTHOR]

This paper describes a technique for constructing block SOR methods for the solution of the large and sparse indefinite least squares problem which involves minimizing a certain type of indefinite quadratic form. Two block SOR-based algorithms and convergence results are presented. The optimum parameters for the methods are also given. It has been shown both theoretically and numerically that the optimum block SOR methods have a faster convergence than block Jacobi and Gauss-Seidel methods. [ABSTRACT FROM AUTHOR]

Ray shooting is a well-studied problem in computer graphics and also has applications in robotics such as collision detection and contact force optimization. Unfortunately, most ray-shooting algorithms developed for graphics applications only allow 3-dimensional (3-D) objects represented as triangle meshes, and therefore are not suited for objects with parametric surfaces or general convex sets in high-dimensional space which robotics applications often require. In contact force optimization, for example, the problem is in the 6-dimensional (6-D) wrench space and it is desirable to consider the nonlinear friction cone without approximating it by a pyramid. This paper discusses existing and novel geometry-based ray-shooting algorithms applicable to general convex sets, and compares their performances in two robotics applications: computing the distance between two convex objects and optimizing contact forces in grasping. [ABSTRACT FROM AUTHOR]

In this paper, a cooperative fuzzy model-predictive control (CFMPC) is presented. The overall nonlinear plant is assumed to consist of several parallel input-coupled Takagi–Sugeno (T–S) fuzzy models. Each such T–S fuzzy subsystem is represented in the form of a local linear model network (LLMN). The control of each local linear model in each LLMN is realized by model-predictive control (MPC). For each LLMN, the outputs of the associated MPCs are blended by the fuzzy membership functions, which leads to a fuzzy model-predictive controller (FMPC). The resulting structure is one FMPC for each LLMN subsystem. Overall, a parallel combination of FMPCs results, which mutually affects all LLMN subsystems by their respective manipulated variables. To compensate detrimental cross-couplings in this setup, a cooperation between the FMPCs is introduced. For this cooperation, convergence is proven, and for the closed-loop system, a stability proof is given. It is demonstrated in a simulation example that the proposed input-constraint CFMPC algorithm achieves convergence of the fuzzy LLMNs within few cooperative iteration steps. Simulations are given to demonstrate the effectiveness of the theoretical results. [ABSTRACT FROM PUBLISHER]

Consensus optimization has received considerable attention in recent years. A number of decentralized algorithms have been proposed for convex consensus optimization. However, to the behaviors or consensus nonconvex optimization, our understanding is more limited. When we lose convexity, we cannot hope that our algorithms always return global solutions though they sometimes still do. Somewhat surprisingly, the decentralized consensus algorithms, DGD and Prox-DGD, retain most other properties that are known in the convex setting. In particular, when diminishing (or constant) step sizes are used, we can prove convergence to a (or a neighborhood of) consensus stationary solution under some regular assumptions. It is worth noting that Prox-DGD can handle nonconvex nonsmooth functions if their proximal operators can be computed. Such functions include SCAD, MCP, and $\ell _q$ quasi-norms, $q\in [0,1)$. Similarly, Prox-DGD can take the constraint to a nonconvex set with an easy projection. To establish these properties, we have to introduce a completely different line of analysis, as well as modify existing proofs that were used in the convex setting. [ABSTRACT FROM AUTHOR]

In decentralized optimization, nodes cooperate to minimize an overall objective function that is the sum (or average) of per-node private objective functions. Algorithms interleave local computations with communication among all or a subset of the nodes. Motivated by a variety of applications..decentralized estimation in sensor networks, fitting models to massive data sets, and decentralized control of multirobot systems, to name a few..significant advances have been made toward the development of robust, practical algorithms with theoretical performance guarantees. This paper presents an overview of recent work in this area. In general, rates of convergence depend not only on the number of nodes involved and the desired level of accuracy, but also on the structure and nature of the network over which nodes communicate (e.g., whether links are directed or undirected, static or time varying). We survey the state-of-theart algorithms and their analyses tailored to these different scenarios, highlighting the role of the network topology. [ABSTRACT FROM AUTHOR]