Search

This paper investigates the stochastic optimization problem focusing on developing scalable parallel algorithms for deep learning tasks. Our solution involves a reformation of the objective function for stochastic optimization in neural network models, along with a novel parallel computing strategy, coined the weighted aggregating stochastic gradient descent (WASGD). Following a theoretical analysis on the characteristics of the new objective function, WASGD introduces a decentralized weighted aggregating scheme based on the performance of local workers. Without any center variable, the new method automatically gauges the importance of local workers and accepts them by their contributions. Furthermore, we have developed an enhanced version of the method, WASGD+, by (1) implementing a designed sample order and (2) upgrading the weight evaluation function. To validate the new method, we benchmark our pipeline against several popular algorithms including the state-of-the-art deep neural network classifier training techniques (e.g., elastic averaging SGD). Comprehensive validation studies have been conducted on four classic datasets: CIFAR-100, CIFAR-10, Fashion-MNIST, and MNIST. Subsequent results have firmly validated the superiority of the WASGD scheme in accelerating the training of deep architecture. Better still, the enhanced version, WASGD+, is shown to be a significant improvement over its prototype. [ABSTRACT FROM AUTHOR]

As an algorithmic framework, message passing is extremely powerful and has wide applications in the context of different disciplines including communications, coding theory, statistics, signal processing, artificial intelligence and combinatorial optimization. In this paper, we investigate the performance of a message-passing algorithm called min-sum belief propagation (BP) for the vertex-disjoint shortest $k$ -path problem ($k$ -VDSP) on weighted directed graphs, and derive the iterative message-passing update rules. As the main result of this paper, we prove that for a weighted directed graph $G$ of order $n$ , BP algorithm converges to the unique optimal solution of $k$ -VDSP on $G$ within $O(n^{2}w_{max})$ iterations, provided that the weight $w_{e}$ is nonnegative integral for each arc $e\in E(G)$ , where $w_{max}=\max \{w_{e}: e\in E(G)\}$. To the best of our knowledge, this is the first instance where BP algorithm is proved correct for NP-hard problems. Additionally, we establish the extensions of $k$ -VDSP to the case of multiple sources or sinks. [ABSTRACT FROM AUTHOR]

This paper studies distributed algorithms for (strongly convex) composite optimization problems over mesh networks, subject to quantized communications. Instead of focusing on a specific algorithmic design, a black-box model is proposed, casting linearly convergent distributed algorithms in the form of fixed-point iterates. The algorithmic model is equipped with a novel random or deterministic Biased Compression (BC) rule on the quantizer design, and a new Adaptive encoding Non-uniform Quantizer (ANQ) coupled with a communication-efficient encoding scheme, which implements the BC-rule using a finite number of bits (below machine precision). This fills a gap existing in most state-of-the-art quantization schemes, such as those based on the popular compression rule, which rely on communication of some scalar signals with negligible quantization error (in practice quantized at the machine precision). A unified communication complexity analysis is developed for the black-box model, determining the average number of bits required to reach a solution of the optimization problem within a target accuracy. It is shown that the proposed BC-rule preserves linear convergence of the unquantized algorithms, and a trade-off between convergence rate and communication cost under ANQ-based quantization is characterized. Numerical results validate our theoretical findings and show that distributed algorithms equipped with the proposed ANQ have more favorable communication cost than algorithms using state-of-the-art quantization rules. [ABSTRACT FROM AUTHOR]

In real-world applications, it is important for machine learning algorithms to be robust against data outliers or corruptions. In this paper, we focus on improving the robustness of a large class of learning algorithms that are formulated as low-rank semi-definite programming (SDP) problems. Traditional formulations use the square loss, which is notorious for being sensitive to outliers. We propose to replace this with more robust noise models, including the $\ell _1$ ℓ 1 -loss and other nonconvex losses. However, the resultant optimization problem becomes difficult as the objective is no longer convex or smooth. To alleviate this problem, we design an efficient algorithm based on majorization-minimization. The crux is on constructing a good optimization surrogate, and we show that this surrogate can be efficiently obtained by the alternating direction method of multipliers (ADMM). By properly monitoring ADMM's convergence, the proposed algorithm is empirically efficient and also theoretically guaranteed to converge to a critical point. Extensive experiments are performed on four machine learning applications using both synthetic and real-world data sets. Results show that the proposed algorithm is not only fast but also has better performance than the state-of-the-arts. [ABSTRACT FROM AUTHOR]

We consider a class of nonconvex regularized optimization problems, which appear frequently in machine learning and data processing. Due to the structure of the problems, the iteratively reweighted algorithm was developed and applied to the consensus optimization. In this paper, we propose the acceleration of this scheme by adding an inertial term in each iteration. The proposed algorithms inherit the advantages of classical decentralized algorithms: they can be implemented over a connected network, in which the agents communicate with their neighbors and perform local computations. We also employ the diminishing stepsizes technique for the iteratively reweighted algorithm and consider its acceleration. In specific cases, our algorithms reduce to existing decentralized schemes and also indicate novel ones. Mathematically, we prove the convergence for both algorithms with several assumptions on the objective functions. With Kurdyka-Łojasiewicz property, convergence rates can be derived for constant stepsize case. Numerical results demonstrate the efficiency of the algorithms. [ABSTRACT FROM AUTHOR]

This paper studies the convergence properties of the well-known message-passing algorithm for convex optimisation. Under the assumption of pairwise separability and scaled diagonal dominance, asymptotic convergence is established and a simple bound for the convergence rate is provided for message-passing. In comparison with previous results, our results do not require the given convex program to have known convex pairwise components and that our bound for the convergence rate is tighter and simpler. When specialised to quadratic optimisation, we generalise known results by providing a very simple bound for the convergence rate. [ABSTRACT FROM AUTHOR]

We consider the problem of minimizing the $\ell _1$ norm of a linear map over the sphere, which arises in various machine learning applications such as orthogonal dictionary learning (ODL) and robust subspace recovery (RSR). The problem is numerically challenging due to its nonsmooth objective and nonconvex constraint, and its algorithmic aspects have not been well explored. In this paper, we show how the manifold structure of the sphere can be exploited to design fast algorithms with provable guarantees for tackling this problem. Specifically, our contribution is fourfold. First, we present a manifold proximal point algorithm (ManPPA) for the problem and show that it converges at a global sublinear rate. Furthermore, we show that ManPPA can achieve a local quadratic convergence rate when applied to sharp instances of the problem. Second, we develop a semismooth Newton-based inexact augmented Lagrangian method for computing the search direction in each iteration of ManPPA and show that it has an asymptotic superlinear convergence rate. Third, we propose a stochastic variant of ManPPA called StManPPA, which is well suited for large-scale computation, and establish its sublinear convergence rate. Both ManPPA and StManPPA have provably faster convergence rates than existing subgradient-type methods. Fourth, using ManPPA as a building block, we propose a new heuristic method for solving a matrix analog of the problem, in which the sphere is replaced by the Stiefel manifold. The results from our extensive numerical experiments on the ODL and RSR problems demonstrate the efficiency and efficacy of our proposed methods. [ABSTRACT FROM AUTHOR]

This paper considers stochastic convex optimization problems with two sets of constraints: (a) deterministic constraints on the domain of the optimization variable, which are difficult to project onto; and (b) deterministic or stochastic constraints that admit efficient projection. Problems of this form arise frequently in the context of semidefinite programming as well as when various NP-hard problems are solved approximately via semidefinite relaxation. Since projection onto the first set of constraints is difficult, it becomes necessary to explore projection-free algorithms, such as the stochastic Frank-Wolfe (FW) algorithm. On the other hand, the second set of constraints cannot be handled in the same way, and must be incorporated as an indicator function within the objective function, thereby complicating the application of FW methods. Similar problems have been studied before; however, they suffer from slow convergence rates. This work, equipped with momentum based gradient tracking technique, guarantees fast convergence rates on par with the best-known rates for problems without the second set of constraints. Zeroth-order variants of the proposed algorithms are also developed and again improve upon the state-of-the-art rate results. We further propose the novel trimmed FW variants that enjoy the same convergence rates as their classical counterparts, but are empirically shown to require significantly fewer calls to the linear minimization oracle speeding up the overall algorithm. The efficacy of the proposed algorithms is tested on relevant applications of sparse matrix estimation, clustering via semidefinite relaxation, and uniform sparsest cut problem. [ABSTRACT FROM AUTHOR]

Dual Coordinate Descent (DCD) is one of the most popular optimization methods. The parallelization of DCD is difficult, as DCD is sequential in nature. As such, simultaneously running multiple DCD threads on batches of data elements causes result inaccuracy and slow convergence, due to the concurrent updates of multiple coordinates. Some parallelization methods adopt separable approximate functions that depend on the degree of parallelism. Such dependencies result in both poor scalability and slow-convergence. To address these challenges, in this paper we present a new parallel primal-dual algorithm for DCD, called PPD. In PPD, the block data distribution is utilized to obtain a new approximate function that is independent from the parallelism. Moreover, PPD is designed with a novel primal-dual acceleration scheme to approach the optimal solution closely and quickly. We demonstrate the advantages of PPD in terms of scalability and efficiency through experiments. [ABSTRACT FROM AUTHOR]

In this paper, we investigate resource allocation for IRS-assisted green multiuser multiple-input single-output (MISO) systems. To minimize the total transmit power, both the beamforming vectors at the access point (AP) and the phase shifts at multiple IRSs are jointly optimized, while taking into account the minimum required quality-of-service (QoS) of multiple users. First, two novel algorithms, namely a penalty-based alternating minimization (AltMin) algorithm and an inner approximation (IA) algorithm, are developed to tackle the non-convexity of the formulated optimization problem when perfect channel state information (CSI) is available. Existing designs employ semidefinite relaxation in AltMin-based algorithms, which, however, cannot ensure convergence. In contrast, the proposed penalty-based AltMin and IA algorithms are guaranteed to converge to a stationary point and a Karush-Kuhn-Tucker (KKT) solution of the design problem, respectively. Second, the impact of imperfect knowledge of the CSI of the channels between the AP and the users is investigated. To this end, a non-convex robust optimization problem is formulated and the penalty-based AltMin algorithm is extended to obtain a stationary solution. Simulation results reveal a key trade-off between the speed of convergence and the achievable total transmit power for the two proposed algorithms. In addition, we show that the proposed algorithms can significantly reduce the total transmit power at the AP compared to various baseline schemes and that the optimal numbers of transmit antennas and IRS reflecting elements, which maximize the system energy efficiency of the considered system, are finite. [ABSTRACT FROM AUTHOR]

Coupled Matrix Tensor Factorization (CMTF) facilitates the integration and analysis of multiple data sources and helps discover meaningful information. Nonnegative CMTF (N-CMTF) has been employed in many applications for identifying latent patterns, prediction, and recommendation. However, due to the added complexity with coupling between tensor and matrix data, existing N-CMTF algorithms exhibit poor computation efficiency. In this paper, a computationally efficient N-CMTF factorization algorithm is presented based on the column-wise element selection, preventing frequent gradient updates. Theoretical and empirical analyses show that the proposed N-CMTF factorization algorithm is not only more accurate but also more computationally efficient than existing algorithms in approximating the tensor as well as in identifying the underlying nature of factors. [ABSTRACT FROM AUTHOR]

The use of a large-scale antenna array (LSAA) has become an important characteristic of multi-antenna communication systems to achieve beamforming gains such as in designing millimeter-wave (mmWave) systems to combat severe propagation losses. In such applications, each antenna element has to be driven by a radio frequency (RF) chain for the implementation of fully-digital beamformers, significantly increasing the hardware cost, complexity, and power consumption. Therefore, constant-modulus analog beamforming (CMAB) becomes a viable solution. In this paper, we consider the scaled analog beamforming (SAB) or constant-modulus analog beamforming (CMAB) architecture and design the system parameters by solving two variants of beampattern matching problem. In the first case, both the magnitude and phase of the beampattern are matched to the given desired beampattern whereas in the second case, only the magnitude of the beampattern is matched. Both the beampattern matching problems are cast as a variant of the constant-modulus least-squares (CLS) problem. We provide efficient algorithms based on the alternating majorization-minimization (AMM) framework that combines the alternating minimization and the MM frameworks and the conventional-cyclic coordinate descent (C-CCD) algorithms to solve the problem in each case. We also propose algorithms based on a new modified-CCD (M-CCD) based approach. For all the developed algorithms we prove convergence to a Karush-Kuhn-Tucker (KKT) point (or a stationary point). Numerical results demonstrate that the proposed algorithms converge faster than the state-of-the-art solutions. Among all the algorithms, the M-CCD-based algorithms have faster convergence when evaluated in terms of the number of iterations and the AMM-based algorithms offer lower complexity. [ABSTRACT FROM AUTHOR]

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $\epsilon$ -stationary solution under a constant step size with $\mathcal {O}(1/\epsilon ^2)$ computation complexity and $\mathcal {O}(1/\epsilon)$ communication complexity when the epigraph of the non-smooth term is a polyhedral set. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $\mathcal {O}(1/\epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we consider the policy evaluation problem in reinforcement learning with agents on a decentralized and directed network. In order to evaluate the quality of a fixed policy in this decentralized setting, one option is for agents to run decentralized temporal-difference (TD) collaboratively. To account for the practical scenarios where the state and action spaces are large and malicious attacks emerge, we focus on the decentralized TD learning with linear function approximation in the presence of malicious agents (often termed as Byzantine agents). We propose a trimmed mean-based Byzantine-resilient decentralized TD algorithm to perform policy evaluation in this setting. We establish the finite-time convergence rate, as well as the asymptotic learning error that depends on the number of Byzantine agents. Numerical experiments corroborate the robustness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Recently, reconciling social networks, identifying the accounts belonging to the same individual across social networks, receives significant attention from both academic and industry. Most of the existing studies have limitations in the following three aspects: multiplicity, comprehensiveness and robustness. To address these limitations, we rethink this problem and, for the first time, robustly and comprehensively reconcile multiple social networks. In this paper, we propose two frameworks, MASTER and MASTER+, i.e., across Multiple social networks, integrate Attribute and STructure Embedding for Reconciliation. In MASTER, we first design a novel Constrained Dual Embedding model, simultaneously embedding and reconciling multiple social networks, to formulate this problem into a unified optimization. To address this optimization, we then design an effective NS-Alternating algorithm and prove it converges to KKT points. To further speed up MASTER, we propose a scalable framework, namely MASTER+. The core idea is to group accounts into clusters and then perform MASTER in each cluster in parallel. Specifically, we design an efficient Augmented Pre-Embedding model and Balance-aware Fuzzy Clustering algorithm for the high efficiency and the high accuracy. Extensive experiments demonstrate that both MASTER and MASTER+ outperform the state-of-the-art approaches. Moreover, MASTER+ inherits the effectiveness of MASTER and enjoys higher efficiency. [ABSTRACT FROM AUTHOR]

Periodic vehicle routing problem with time windows (PVRPTWs) is an important combinatorial optimization problem that can be applied in different fields. It is essentially a multiobjective optimization problem due to the problem nature. In this paper, a typical multiobjective PVRPTW with five objectives is first defined and new nonsymmetric real-world multiobjective PVRPTW instances are generated. Then, a hybrid multiobjective memetic algorithm is proposed for solving multiobjective PVRPTW. In the proposed algorithm, a two-phase strategy is devised to improve the comprehensive performance in terms of the convergence and diversity. In this strategy, several extreme solutions near an approximate Pareto front (PF) are identified at Phase I, and then the approximate PF is extended at Phase II. The proposed algorithm is extensively tested on both real-world instances and traditional instances. Experiment results show that the proposed algorithm outperforms two representative competitor algorithms on most of the instances. The effectiveness of the two-phase strategy is also confirmed. [ABSTRACT FROM AUTHOR]

Incomplete multi-view clustering (IMVC) optimally combines multiple pre-specified incomplete views to improve clustering performance. Among various excellent solutions, the recently proposed multiple kernel $k$ k -means with incomplete kernels (MKKM-IK) forms a benchmark, which redefines IMVC as a joint optimization problem where the clustering and kernel matrix imputation tasks are alternately performed until convergence. Though demonstrating promising performance in various applications, we observe that the manner of kernel matrix imputation in MKKM-IK would incur intensive computational and storage complexities, over-complicated optimization and limitedly improved clustering performance. In this paper, we first propose an Efficient and Effective Incomplete Multi-view Clustering (EE-IMVC) algorithm to address these issues. Instead of completing the incomplete kernel matrices, EE-IMVC proposes to impute each incomplete base matrix generated by incomplete views with a learned consensus clustering matrix. Moreover, we further improve this algorithm by incorporating prior knowledge to regularize the learned consensus clustering matrix. Two three-step iterative algorithms are carefully developed to solve the resultant optimization problems with linear computational complexity, and their convergence is theoretically proven. After that, we theoretically study the generalization bound of the proposed algorithms. Furthermore, we conduct comprehensive experiments to study the proposed algorithms in terms of clustering accuracy, evolution of the learned consensus clustering matrix and the convergence. As indicated, our algorithms deliver their effectiveness by significantly and consistently outperforming some state-of-the-art ones. [ABSTRACT FROM AUTHOR]

In this paper, an intelligent reflecting surface (IRS) assisted spectrum sharing underlay cognitive radio (CR) wiretap channel (WTC) is studied, and we aim at enhancing the secrecy rate of secondary user in this channel subject to total power constraint at secondary transmitter (ST), interference power constraint (IPC) at primary receiver (PR) as well as unit modulus constraint at IRS. Due to extra IPC and eavesdropper (Eve) are considered, all the existing solutions for enhancing secrecy rate of IRS-assisted non-CR WTC as well as enhancing transmission rate in IRS-assisted CR channel without eavesdropper fail in this work. Therefore, we propose new numerical solutions to optimize the secrecy rate of this channel under full primary, secondary users’ channel state information (CSI) and three different cases of Eve’s CSI: full CSI, imperfect CSI with bounded estimation error, and no CSI. To solve the difficult non-convex optimization problem, an efficient alternating optimization (AO) algorithm is proposed to jointly optimize the beamformer at ST and phase shift coefficients at IRS. In particular, when optimizing the phase shift coefficients during each iteration of AO, a Dinkelbach based solution in combination with successive approximation and penalty based solution is proposed under full CSI and a penalty convex-concave procedure solution is proposed under imperfect Eve’s CSI. For no Eve’s CSI case, artificial noise (AN) aided approach is adopted to help enhancing the secrecy rate. Simulation results show that our proposed solutions for the IRS-assisted design greatly enhance the secrecy performance compared with the existing numerical solutions with and without IRS under full and imperfect Eve’s CSI. And positive secrecy rate can be achieved by our proposed AN aided approach given most channel realizations under no Eve’s CSI case so that secure communication also can be guaranteed. All of the proposed AO algorithms are guaranteed to monotonic convergence. [ABSTRACT FROM AUTHOR]

Under consensus-based economic dispatch (ED) algorithm, multiple agents, which control local generation units, cooperatively minimize the total generation cost subject to the balance of the generation and expected demand in smart grids. As ubiquitous time delays on communication links exist in communication networks, studying the effect of delays on the dispatch performance is of both theoretical merit and practical value for the efficient and stable operation of smart grids. In this paper, we consider a well-developed consensus-based ED protocol under constant time delays. We find that there always exists a sufficiently small learning gain parameter under finite constant delays such that the convergence of the consensus-based algorithm is guaranteed. Further, an analytical expression of the upper bound is established for the learning gain parameter, which is determined by the largest delay, the weight matrix and the parameters of generation cost functions. In order to guarantee the optimality of the final solution, we propose the updating rule for iterations when initial states are not received by their neighbors due to time delays. The optimality of the final solution under the proposed updating rule is analyzed. We validate our theoretical results through extensive simulation studies. [ABSTRACT FROM AUTHOR]

Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers’ choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms. [ABSTRACT FROM AUTHOR]

This paper investigates the resource allocation problem for a group of agents communicating over a strongly connected directed graph, where the total objective function of the problem is composted of the sum of the local objective functions incurred by the agents. With local convex sets, we first design a continuous-time projection algorithm over a strongly connected and weight-balanced directed graph. Our convergence analysis indicates that when the local objective functions are strongly convex, the output state of the projection algorithm could asymptotically converge to the optimal solution of the resource allocation problem. In particular, when the projection operation is not involved, we show the exponential convergence at the equilibrium point of the algorithm. Second, we propose an adaptive continuous-time gradient algorithm over a strongly connected and weight-unbalanced directed graph for the reduced case without local convex sets. In this case, we prove that the adaptive algorithm converges exponentially to the optimal solution of the considered problem, where the local objective functions and their gradients satisfy strong convexity and Lipachitz conditions, respectively. Numerical simulations illustrate the performance of our algorithms. [ABSTRACT FROM AUTHOR]

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have been proposed to reduce the communication load at the price of a loss in solution accuracy. Recently, it has been shown how compression errors can be compensated for in the optimization algorithm to improve the solution accuracy. Even though convergence guarantees for error-compensated algorithms have been established, there is very limited theoretical support for quantifying the observed improvements in solution accuracy. In this paper, we show that Hessian-aided error compensation, unlike other existing schemes, avoids accumulation of compression errors on quadratic problems. We also present strong convergence guarantees of Hessian-based error compensation for stochastic gradient descent. Our numerical experiments highlight the benefits of Hessian-based error compensation, and demonstrate that similar convergence improvements are attained when only a diagonal Hessian approximation is used. [ABSTRACT FROM AUTHOR]

In sensor networks (SNs), how to allocate the resources so as to optimize data gathering and network utility is an important and challenging task. This paper studies the distributed optimization problem in SNs. A distributed hybrid-driven algorithm based on the coordinate descent method is presented for the optimization purpose. The proposed optimization algorithm differs from the existing ones since the hybrid driven scheme allows more choices of actuation time, resulting a tradeoff between communications and computation performance. Applying the proposed algorithm, each sensor node is driven in a hybrid event time manner, which removes the requirement of strict time synchronization. The convergence and optimality of the proposed algorithm are analyzed, and then verified by simulation examples. The developed results also show the tradeoff between communications and computation performance. [ABSTRACT FROM AUTHOR]

The matrix decomposing into a sum of low-rank and sparse components has found extensive applications in many areas including video surveillance, computer vision, and medical imaging. In this paper, we propose a new algorithm for recovery of low rank and sparse components of a given matrix. We have also proved the convergence of the proposed algorithm. The simulation results with synthetic and real signals such as image and video signals indicate that the proposed algorithm has a better performance with lower run-time than the conventional methods. [ABSTRACT FROM AUTHOR]

Optimal power flow (OPF) is a fundamental problem in power system operating and planning. The need for distributed solutions is rapidly growing in distribution systems with radial networks, discrete equipments and many distributed generation units. Typically the model is difficult to solve due to the highly non-convexity and even non-continuity of the arising OPF problem. In this paper, we address these challenges by developing a penalty dual decomposition (PDD) based algorithm to obtain distributed solutions. The proposed PDD based algorithm mainly consists of two loops: in the outer loop we update the dual variables and the penalty parameter according to the constraint violation, while in the inner loop we divide the primal variables into several blocks and employ the block successive upper-bound minimization (BSUM) method to iteratively optimize each block variables with fixed penalty parameter and dual variables. Each subproblem in the proposed PDD based algorithm can be solved either in closed-form or by the bisection method and every limit point generated by the proposed algorithm is guaranteed to be a stationary point of the OPF problem. Simulation results, and comparison with centralized PDD and concave-convex procedure (CCCP) based algorithms are presented to validate the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Matrix factorization has been widely applied to various applications. With the fast development of storage and internet technologies, we have been witnessing a rapid increase of data. In this paper, we propose new algorithms for matrix factorization with the emphasis on efficiency. In addition, most existing methods of matrix factorization only consider a general smooth least square loss. Differently, many real-world applications have distinctive characteristics. As a result, different losses should be used accordingly. Therefore, it is beneficial to design new matrix factorization algorithms that are able to deal with both smooth and non-smooth losses. To this end, one needs to analyze the characteristics of target data and use the most appropriate loss based on the analysis. We particularly study two representative cases of low-rank matrix recovery, i.e., collaborative filtering for recommendation and high dynamic range imaging. To solve these two problems, we respectively propose a stage-wise matrix factorization algorithm by exploiting manifold optimization techniques. From our theoretical analysis, they are both are provably guaranteed to converge to a stationary point. Extensive experiments on recommender systems and high dynamic range imaging demonstrate the satisfactory performance and efficiency of our proposed method on large-scale real data. [ABSTRACT FROM AUTHOR]

This work addresses the robust principal component analysis (PCA) problem using generalized nonoconvex regularization for low-rank and sparsity promotion. While the popular nuclear and $\ell _1$ -norm penalties have a bias problem, nonconvex regularization can alleviate the bias problem and can be expected to achieve better performance. In this paper, a proximal block coordinate descent (BCD) algorithm is used to efficiently solve the nonconvex regularized robust PCA problem. It is globally convergent under weak conditions. Further, for a popular class of penalties having discontinuous threshoding functions, we establish the convergence to a restricted strictly local minimizer and, also, a local linear convergence rate for the proximal BCD algorithm. Moreover, convergence to a local minimizer has been derived for hard-thresholding. Our result is the first on nonconvex robust PCA with established convergence to strictly local minimizer with local linear convergence rate. Numerical experiments have been provided to demonstrate the performance of the new algorithm. [ABSTRACT FROM AUTHOR]

This paper proposes a decentralized algorithm for solving a consensus optimization problem defined in a static networked multi-agent system, where the local objective functions have the smooth+nonsmooth composite form. Examples of such problems include decentralized constrained quadratic programming and compressed sensing problems, as well as many regularization problems arising in inverse problems, signal processing, and machine learning, which have decentralized applications. This paper addresses the need for efficient decentralized algorithms that take advantages of proximal operations for the nonsmooth terms. We propose a proximal gradient exact first-order algorithm (PG-EXTRA) that utilizes the composite structure and has the best known convergence rate. It is a nontrivial extension to the recent algorithm EXTRA. At each iteration, each agent locally computes a gradient of the smooth part of its objective and a proximal map of the nonsmooth part, as well as exchanges information with its neighbors. The algorithm is “exact” in the sense that an exact consensus minimizer can be obtained with a fixed step size, whereas most previous methods must use diminishing step sizes. When the smooth part has Lipschitz gradients, PG-EXTRA has an ergodic convergence rate of O\left(1\over k\right) in terms of the first-order optimality residual. When the smooth part vanishes, PG-EXTRA reduces to P-EXTRA, an algorithm without the gradients (so no “G” in the name), which has a slightly improved convergence rate at o\left(1\over k\right) in a standard (non-ergodic) sense. Numerical experiments demonstrate effectiveness of PG-EXTRA and validate our convergence results [ABSTRACT FROM PUBLISHER]

In this article, we consider a class of decentralized nonsmooth composite optimization problems over undirected graphs. The global optimization problem is to minimize the sum of local objective functions consisting of a Lipschitz-differentiable convex function and two possibly nonsmooth convex functions, one of which contains a bounded linear operator. The goal is to solve the global optimization problem through decentralized computation and communication over a network of agents without a central coordinator. Through using triple proximal splitting operators to deal with the nonsmooth terms, we come up with a novel decentralized algorithm with uncoordinated stepsizes, where the stepsizes with independent upper bounds are also distributed for agents or edges over the communication network. Furthermore, we establish the sublinear convergence rate for the proposed algorithm in terms of the first-order optimality residual in a nonergodic sense. Simulation experiments on a constrained quadratic programming problem and an optimal load-sharing problem are carried out to verify the correctness of the theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, a novel algorithm “firework optimization” (FO) is introduced, as well as its hybridization with generating set search (GSS). FO is an exploration algorithm, which intends to explore a vast region within the search space in order to determine regions that may contain a regional optimum. As the regions with potential are defined, GSS, the exploitation algorithm is applied to determine whether regions determined by FO contain a local or a global optimum. The validity of FO and GSS hybridization algorithm is verified using Branin and Six-Hump Camel test functions, and the hybridization algorithm is applied to an interior permanent magnet synchronous motor, in order to minimize torque ripple. [ABSTRACT FROM AUTHOR]

Graph cuts are widely used in computer vision. To speed up the optimization process and improve the scalability for large graphs, Strandmark and Kahl introduced a splitting method to split a graph into multiple subgraphs for parallel computation in both shared and distributed memory models. However, this parallel algorithm (the parallel BK-algorithm) does not have a polynomial bound on the number of iterations and is found to be non-convergent in some cases due to the possible multiple optimal solutions of its sub-problems. To remedy this non-convergence problem, in this paper, we first introduce a merging method capable of merging any number of those adjacent sub-graphs that can hardly reach agreement on their overlapping regions in the parallel BK-algorithm. Based on the pseudo-boolean representations of graph cuts, our merging method is shown to be effectively reused all the computed flows in these sub-graphs. Through both splitting and merging, we further propose a dynamic parallel and distributed graph cuts algorithm with guaranteed convergence to the globally optimal solutions within a predefined number of iterations. In essence, this paper provides a general framework to allow more sophisticated splitting and merging strategies to be employed to further boost performance. Our dynamic parallel algorithm is validated with extensive experimental results. [ABSTRACT FROM PUBLISHER]

Distributed generators are integrated into microgrids or geographically distributed subsystems in active distribution networks and may belong to various entities. Centralized solutions cannot meet future requirements such as high resilience and privacy protection. Therefore, fully distributed economic dispatch (ED) methods are needed and most existing algorithms exhibit linear or sublinear convergence. This paper introduces an efficient fully distributed dynamic ED solution for minimizing photovoltaic power curtailment, based on a parallel primal-dual interior-point algorithm with a quasi-Newton technique. In this method, each area optimizes its own problem with limited information exchanged with its neighbors, and no central coordinator is needed. Based on a peer-to-peer communication paradigm, this method can achieve a global optimum while the privacy of each area is obliquely protected. Numerical tests demonstrated that the algorithm outperforms gradient-based methods such as ADMM and maintains stability under a partial communication failure. [ABSTRACT FROM AUTHOR]

In this article, the distributed finite-time and fixed-time optimization problems are investigated by adopting the zero-gradient-sum (ZGS) framework in multiagent systems. Specifically, when the local convex functions are nonquadratic, a basic optimization protocol is proposed to obtain a finite-time convergence, such that the networked system can cooperatively seek the optimal solution of the global objective, the sum of local objective, in a limited time. By utilizing the property of quadratic functions, a reduced algorithm can remove the dependence of initial conditions in the estimation of the upper bound of settling time and achieve a fixed-time result. Besides, the problem with time-varying topologies is studied by introducing a modified algorithm with an artificial potential function to preserve the network connectivity. Finally, the validity of the protocols is demonstrated via some example simulations. [ABSTRACT FROM AUTHOR]

A new ramped convergence particle swarm optimization (RCPSO) algorithm to design radio frequency identification (RFID) reader antenna for gain optimization is presented in this paper. The algorithm breaks down the optimization problem by considering only a subset of dimensions at a time, hence overcoming curse of dimensionality. It uses a multi-start approach that alleviates premature convergence. It applies heterogeneous boundary conditions to errant particles. In this paper, the effectiveness of using RCPSO for RFID reader antenna design will be investigated by a number of iterations required to achieve optimum solution along with the quality of solution. RCPSO is built specifically for antenna design. For the microstrip antenna, a significant bandwidth improvement is achieved and the overall antenna dimensions are kept practically the same. The circular gain improves by a significant 2 dBiC over super high frequency (SHF). [ABSTRACT FROM PUBLISHER]

In this paper, we propose a novel memetic algorithm, which is explorative particle swarm optimization (ePSO), combined with mesh adaptive direct search and apply it to the design of a permanent magnet synchronous machine (PMSM). The ePSO, which is modified from the PSO, drastically improves search time and iteration number at an exploration search stage. Unlike the existing methods, the proposed rule of start point selection takes an advantage of minimizing the search time. By applying the proposed algorithm to PMSM, we clarify the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In space communication environments, most studies in delay tolerant networks (DTNs) adopt experimental approaches and they have shown that the utilization efficiency of link bandwidth can be affected by changing packet sizes at the bundle layer and the convergence layer. However, the lack of theoretical work makes it difficult to find the optimal packet sizes in general scenarios. In this paper, we focus on this important issue by performing a theoretical analysis to solve the packet size optimization problems in both one-hop and multihop DTNs. As our analysis is based on the widely used DTN architecture with Bundle Protocol (BP) and Licklider Transmission Protocol (LTP), the theoretical results can be applied to any DTN scenario using this architecture. We formulate packet size optimization problems in DTNs as nonconvex optimization problems. Furthermore, a goodput enhancement algorithm (GEA) is also proposed to find the optimal packet sizes in order to maximize goodput in DTNs. We conduct numerical analysis as well as simulations to validate our theoretical analysis and study the factors that determine the optimal packet sizes for BP and LTP. Numerical results show that GEA achieves significant performance improvement compared with schemes where packet size optimization on BP and LTP is not jointly considered. [ABSTRACT FROM PUBLISHER]

Geo-localization services are an important functionality in cellular networks. Besides, the use of Ultra Dense Networks and small cells, in current and future cellular networks, greatly increases the complexity of centralized localization approaches. Consequently, we propose a Self-Synchronization Positioning Estimation (SSPE) algorithm that estimates the transmitter position in a distributed fashion. The proposed SSPE algorithm reaches consensus for the posterior distribution of the transmitter position rather than on the final estimates. Such consensus ensures that the proposed SSPE algorithm converges to the centralized Direct Positioning Estimation (DPE) approach, which has the best performance of all localization approaches. We show that the proposed algorithm is related to the Iterative Positioning Estimation (IPE) algorithm, since both exploit the self-synchronization mechanism. As a result, the improvements and extensions for IPE, previously studied in other works, can be directly applied to the proposed SSPE algorithm. In addition, the proposed algorithm is able to localize the transmitter even when it is not time synchronized with the network as it is usually the case. The performance of the algorithms is numerically assessed through Monte-Carlo simulations by the mean distance error and mean range offset error. Finally, we not only show that our approach gets close to the DPE performance after a few iterations, but also that it converges for different logical network configurations. [ABSTRACT FROM AUTHOR]

We consider a class of popular distributed non-convex optimization problems, in which agents connected by a network $\mathcal {G}$ collectively optimize a sum of smooth (possibly non-convex) local objective functions. We address the following question: if the agents can only access the gradients of local functions, what are the fastest rates that any distributed algorithms can achieve, and how to achieve those rates. First, we show that there exist difficult problem instances, such that it takes a class of distributed first-order methods at least $\mathcal {O}(1/\sqrt{\xi ({ \mathcal {G} })} \times \bar{L} /{\epsilon })$ communication rounds to achieve certain $\epsilon$ -solution [where $\xi ({ \mathcal {G} })$ denotes the spectral gap of the graph Laplacian matrix, and $\bar{L}$ is some Lipschitz constant]. Second, we propose (near) optimal methods whose rates match the developed lower rate bound (up to a ploylog factor). The key in the algorithm design is to properly embed the classical polynomial filtering techniques into modern first-order algorithms. To the best of our knowledge, this is the first time that lower rate bounds and optimal methods have been developed for distributed non-convex optimization problems. [ABSTRACT FROM AUTHOR]