Search

Setting retail electricity prices is one of the significant strategies for energy management of multi-microgrid (MMG) systems integrated with renewable energy. Nevertheless, the need of privacy preservation, the uncertainties of renewable energy and loads, as well as the time-varying scenarios, bring challenges for pricing problems. In this paper, a two-level pricing framework is proposed based on interval predictions and model-free reinforcement learning to address these challenges. In particular, at the higher level, the distribution system operator (DSO) is viewed as an agent, which sets retail electricity prices without detailed user information for privacy protection to maximize the total revenue from selling energy with reinforcement learning. For time-varying scenarios with intermittent photovoltaic power generation and diverse loads, a differentiable trust region layer is considered in reinforcement learning to improve the robustness of the policy updating process. While at the lower level, operators in microgrids solve three-phase unbalanced optimal power flow (OPF) problems to minimize generation cost and network power loss. Additionally, to deal with the challenges from the uncertainties of renewable power generation and user loads, interval predictions are chosen to quantify prediction errors and improve the flexibility of pricing policies. Finally, a set of experiments are conducted to validate the effectiveness of the proposed method for pricing problems in MMG systems. [ABSTRACT FROM AUTHOR]

The memristor-based neuromorphic computing system (NCS) with emerging storage and computing integration architecture has drawn extensive attention. Because of the unique nonvolatility and programmability, the memristor is an ideal nano-device to realize neural synapses in VLSI circuit implementation of neural networks. However, in the hardware implementation, the performance of the memristive neural network is always affected by quantization error, writing error, and conductance drift, which seriously hinders its applications in practice. In this paper, a novel weight optimization scheme combining quantization and Bayesian inference is proposed to alleviate this problem. Specifically, the weight deviation in the memristive neural network is transformed into the weight uncertainty in the Bayesian neural network, which can make the network insensitive to unexpected weight changes. A quantization regularization term is designed and utilized during the training process of the Bayesian neural network, reducing the quantization error and improving the robustness of the network. Furthermore, a partial training method is raised to extend the applicability of the proposed scheme in large-scale neural networks. Finally, the experiments on a Multilayer Perceptron and LeNet demonstrate that the proposed weight optimization scheme can significantly enhance the robustness of memristive neural networks. [ABSTRACT FROM AUTHOR]

Using reinforcement learning (RL) algorithm to optimize guidance law can address non-idealities in complex environment. However, the optimization is difficult due to huge state-action space, unstable training, and high requirements on expertise. In this paper, the constrained guidance policy of a neural guidance system is optimized using improved RL algorithm, which is motivated by the idea of traditional model-based guidance method. A novel optimization objective with minimum overload regularization is developed to restrain the guidance policy directly from generating redundant missile maneuver. Moreover, a bi-level curriculum learning is designed to facilitate the policy optimization. Experiment results show that the proposed minimum overload regularization can reduce the vertical overloads of missile significantly, and the bi-level curriculum learning can further accelerate the optimization of guidance policy. [ABSTRACT FROM AUTHOR]

Residual block is a very common component in recent state-of-the art CNNs such as EfficientNet/EfficientDet. Shortcut data accounts for nearly 40% of feature-maps access in ResNet152. Most of the previous DNN compilers/accelerators ignore the shortcut data optimization. This paper presents ShortcutFusion, an optimization tool for FPGA-based accelerator with a reuse-aware static memory allocation for shortcut data, to maximize on-chip data reuse given resource constraints. From TensorFlow DNN models, the proposed design generates instruction sets for a group of nodes which uses an optimized data reuse for each residual block. The accelerator design implemented on the Xilinx KCU1500 FPGA card $2.8\times $ faster and $9.9\times $ more power efficient than NVIDIA RTX 2080 Ti for $256\times 256$ input size. Compared to the result from baseline, in which the weights/inputs/outputs are accessed from the off-chip memory exactly once per each layer, ShortcutFusion reduces the DRAM access by 47.8-84.8% for RetinaNet, Yolov3, ResNet152, and EfficientNet. Given a similar buffer size to ShortcutMining, which also “mine” the shortcut data in hardware, the proposed work reduces off-chip access for feature-maps $5.27\times $ while accessing weight from off-chip memory exactly once. [ABSTRACT FROM AUTHOR]

For the design of ring oscillator-based ADCs, little has been reported on how to optimally co-design the readout scheme and the ring oscillator core towards optimal energy efficiency. This paper describes a methodology to find this optimum for a target SQNR and signal bandwidth, for the case of 1st-order quantization noise shaping. In short, starting from initial assumptions on the VCO, the readout optimization boils down to finding the best VCO frequency. From this, the number of readout phases, the sampling frequency of the readout and the number of bits in the quantizers are derived. Then, it is explained how to adjust the VCO core toward the corresponding optimal VCO configuration. This is discussed for the case of a current-controlled ring oscillator, indicating the main constraints that define the design space for such a VCO optimization. If the actual VCO configuration is bounded by practical constraints, this approach can be re-iterated where a new readout optimization is performed taking these constraints into account. [ABSTRACT FROM AUTHOR]

In this paper, we revisit Boole’s expansion theorem to propose a new synthesis method for implication logic circuits based on memristors. By rewriting the sum-of-products form of Boole’s expansion theorem in terms that are best suited for the implication logic, we develop a generalized rule to derive the sequence of operations needed to realize any logic function written in the classical AND-OR form. The proposed method leverages on multi-input operation, minimizing both the number of steps required to compute a given Boolean function and the number of memristors involved. Moreover, it allows using well-established methods of logic circuit optimization like binary decision diagrams, Karnaugh maps, and heuristic algorithms, that are already implemented in commercial CAD software. The proposed method allows a fair comparison between the performance of CMOS and implication logic implementations of the same logic function under the same degree of optimization, and is shown to outperform existing approaches. Possible device-circuit co-design strategies to optimize circuit performance are finally discussed. [ABSTRACT FROM AUTHOR]

This paper presents a transimpedance-to-noise optimization approach for design of a resistive shunt-feedback TIA. This optimization offers an enhancement in the transimpedance and a noise performance very close to the theoretical minimum noise of the TIA. In addition, the transimpedance-to-noise optimization approach results in a small front-end FET size which enables a further reduction in power and area. Moreover, this approach enables using a fewer number of stages in the receiver chain which makes a high PSRR feasible and obviates the necessity for using an offset cancellation circuitry. Building on this approach, a fully differential analog front-end including a resistive shunt-feedback TIA and a post amplifier (PA) for time-of-flight (ToF) Lidar receivers is designed and implemented, achieving 94dB Ω transimpedance gain, 71nA input-referred rms noise current, −3dB bandwidth of 340MHz, and power supply rejection ratio (PSRR) of more than 87dB in a 0.11 μm CMOS process. The associated DC power consumption is 19.4mW with V DD of 1.8V. Moreover, a push-pull buffer with 1V output swing is integrated for driving 50 Ω loads, such as off-chip time discriminators, which also additionally amplifies the signal with a gain of 5dB while consuming an extra 20.9mW of DC power. The whole chip (excluding pads) occupies 210 μm × 110 μm in area. [ABSTRACT FROM AUTHOR]

Dropout and DropConnect are two techniques to facilitate the regularization of neural network models, having achieved the state-of-the-art results in several benchmarks. In this paper, to improve the generalization capability of spiking neural networks (SNNs), the two drop techniques are first applied to the state-of-the-art SpikeProp learning algorithm resulting in two improved learning algorithms called SPDO (SpikeProp with Dropout) and SPDC (SpikeProp with DropConnect). In view that a higher membrane potential of a biological neuron implies a higher probability of neural activation, three adaptive drop algorithms, SpikeProp with Adaptive Dropout (SPADO), SpikeProp with Adaptive DropConnect (SPADC), and SpikeProp with Group Adaptive Drop (SPGAD), are proposed by adaptively adjusting the keep probability for training SNNs. A convergence theorem for SPDC is proven under the assumptions of the bounded norm of connection weights and a finite number of equilibria. In addition, the five proposed algorithms are carried out in a collaborative neurodynamic optimization framework to improve the learning performance of SNNs. The experimental results on the four benchmark data sets demonstrate that the three adaptive algorithms converge faster than SpikeProp, SPDO, and SPDC, and the generalization errors of the five proposed algorithms are significantly smaller than that of SpikeProp. Furthermore, the experimental results also show that the five algorithms based on collaborative neurodynamic optimization can be improved in terms of several measures. [ABSTRACT FROM AUTHOR]

Element matching approaches based on Optimal Combination Algorithms (OCAs) are methods that try to calibrate element mismatches in data converters by rearranging the mapping relations between circuit elements and control logics. This work for the first time proposes a universal method and its corresponding Figure-of-Merit to evaluate the performance of various OCA approaches. It adopts the quantile of integral nonlinearity as the criterion and is consequently compatible with most of matching OCAs. Dynamic performance and the corresponding effective-number-of-bits (ENOB) limited by mismatches are also investigated. A critical feature of the proposed method is its robustness to the disturbance of element standard deviation as long as the standard deviation is in a reasonable region. Merits of our method have been explained and proved mathematically, and also been verified numerically by data extracted from previously reported works and their reproduction by authors, which together make this proposal more convincing and rigorous. Due to those merits, our proposed method could also help designers to specify a feasible standard deviation for circuit elements when they are designing data converters with OCAs. [ABSTRACT FROM AUTHOR]

This article proposes a robust method for the synthesis and design of optimal linear group delay filters with prescribed bandwidth and dispersion properties. The proposed filters are based on a cascade of second-order all-pass filter cells, for which all the parameters are optimized in order to reach the desired wideband linear group delay characteristic with positive slope. Compared to the previously published synthesis method, the proposed approach offers appropriate initialization enabling convergence and aiming for the optimal solution. The method’s convergence and robustness are proved through a 21-cells synthesis, which is the highest number of cascaded cells reported to date, due to the usual convergence problem. Then the electronic feasibility is illustrated through the measurement results of a lumped-elements linear group-delay filter prototype dimensioned with this method. The measured group delay presents a positive slope of 8 ns GHz−1 in the frequency range [100–725] MHz. It is, to the best of our knowledge, the highest positive slope reported to date with an electronic implementation (i.e except SAW and waveguide devices). [ABSTRACT FROM AUTHOR]

Steady-state optimization and real-time controls are conventionally achieved by separate controllers acting at different timescales in dc microgrids. The separation between them not only lowers the overall energy efficiency but also can cause the line currents and the output currents of distributed generators (DGs) to exceed their limits. Targeting these issues, an optimal controller for dc microgrids is proposed in this article to simultaneously achieve the real-time minimization of operating losses (converter losses and line losses) and the regulation of line currents and DG output currents. First, the optimization problem is established, then converted to an optimal control problem by a designed penalty function. Thereafter, the necessary and sufficient optimality conditions of the problem are derived, based on which a distributed controller is proposed to dynamically track the optimal operating point. Driven by the proposed optimal controller in real-time, line current and DG output current regulations are ensured with greatly enhanced performance. Finally, the closed-loop system stability is analyzed rigorously through Lyapunov stability synthesis. Simulations based on a switch-level microgrid model validate the performance of the proposed control solution. [ABSTRACT FROM AUTHOR]

With the development of the Internet of things (IoT), sensors are being mounted on various objects. This trend has prompted demand for low-power, high-performance information processing on the edge side. Here, an Ising model architecture that can efficiently solve optimization problems would be an efficient processing solution for edges. In this study, we implemented a 512- spin fully connected Ising model on an LSI chip fabricated in a 28-nm CMOS process. The fully connected Ising model was implemented in the chip by using pseudo-annealing (PA), which is easier to implement than simulated annealing (SA). In addition, we devised a multi-spin-thread structure, concurrent update structure, and a folded interaction placement for accuracy, speed, and compactness. Because eight spin threads are implemented, the calculation throughput could be increased by a factor of eight in comparison with a single spin-thread implementation. Moreover, as a measure of solution accuracy, the average route length of a 22-city traveling salesman problem was reduced by 19% and the standard deviation (SD) was reduced by 46%. Likewise, the average cut value of a 512- node max-cut problem was increased by 1.6% and SD was decreased by 60%. The concurrent update almost doubled the calculation speed in comparison with the case of no concurrent update. In addition, the circuit area was reduced by about 38% as a result of the folded interaction placement. The time required to obtain a solution was 128 ms. The chip at annealing processing (main processing) had a power consumption of 12 mW at 1 MHz. [ABSTRACT FROM AUTHOR]

Most modern learning problems are highly overparameterized, i.e., have many more model parameters than the number of training data points. As a result, the training loss may have infinitely many global minima (parameter vectors that perfectly “interpolate” the training data). It is therefore imperative to understand which interpolating solutions we converge to, how they depend on the initialization and learning algorithm, and whether they yield different test errors. In this article, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which stochastic gradient descent (SGD) is a special case. Recently, it has been shown that for overparameterized linear models, SMD converges to the closest global minimum to the initialization point, where closeness is in terms of the Bregman divergence corresponding to the potential function of the mirror descent. With appropriate initialization, this yields convergence to the minimum-potential interpolating solution, a phenomenon referred to as implicit regularization. On the theory side, we show that for sufficiently-overparameterized nonlinear models, SMD with a (small enough) fixed step size converges to a global minimum that is “very close” (in Bregman divergence) to the minimum-potential interpolating solution, thus attaining approximate implicit regularization. On the empirical side, our experiments on the MNIST and CIFAR-10 datasets consistently confirm that the above phenomenon occurs in practical scenarios. They further indicate a clear difference in the generalization performances of different SMD algorithms: experiments on the CIFAR-10 dataset with different regularizers, $\ell _{1}$ to encourage sparsity, $\ell _{2}$ (SGD) to encourage small Euclidean norm, and $\ell _{\infty }$ to discourage large components, surprisingly show that the $\ell _{\infty }$ norm consistently yields better generalization performance than SGD, which in turn generalizes better than the $\ell _{1}$ norm. [ABSTRACT FROM AUTHOR]

It is challenging to bridge the performance gap between binary convolutional neural network (BCNN) and floating-point CNN (FCNN). This performance gap is mainly caused by the inferior modeling capability and training strategy of BCNN, which leads to substantial residuals in intermediate feature maps between BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This intuition leads to a more effective training strategy for BCNN, i.e., optimizing each binary convolutional block with blockwise distillation loss derived from FCNN. The goal of minimizing the residuals in intermediate feature maps also motivates us to update the binary convolutional block architecture to facilitate the optimization of blockwise distillation loss. Specifically, a lightweight shortcut branch is inserted into each binary convolutional block to complement residuals at each block. Benefited from its squeeze-and-interaction (SI) structure, this shortcut branch introduces a fraction of parameters, e.g., less than 10% overheads, but effectively boosts the modeling capability of binary convolution blocks in BCNN. Extensive experiments on ImageNet demonstrate the superior performance of our method in both classification efficiency and accuracy, e.g., BCNN trained with our methods achieves the accuracy of 60.45% on ImageNet, better than many state-of-the-art ones. [ABSTRACT FROM AUTHOR]

In online learning, the dynamic regret metric chooses the reference oracle that may change over time, while the typical (static) regret metric assumes the reference solution to be constant over the whole time horizon. The dynamic regret metric is particularly interesting for applications, such as online recommendation (since the customers’ preference always evolves over time). While the online gradient (OG) method has been shown to be optimal for the static regret metric, the optimal algorithm for the dynamic regret remains unknown. In this article, we show that proximal OG (a general version of OG) is optimum to the dynamic regret by showing that the proved lower bound matches the upper bound. It is highlighted that we provide a new and general lower bound of dynamic regret. It provides new understanding about the difficulty to follow the dynamics in the online setting. [ABSTRACT FROM AUTHOR]

This article investigates the nonnegative consensus tracking problem for networked systems with a distributed static output-feedback (SOF) control protocol. The distributed SOF controller design for networked systems presents a more challenging issue compared with the distributed state-feedback controller design. The agents are described by multi-input multi-output (MIMO) positive dynamic systems which may contain uncertain parameters, and the interconnection among the followers is modeled using an undirected connected communication graph. By employing positive systems theory, a series of necessary and sufficient conditions governing the consensus of the nominal, as well as uncertain, networked positive systems, is developed. Semidefinite programming consensus design approaches are proposed for the convergence rate optimization of MIMO agents. In addition, by exploiting the positivity characteristic of the systems, a linear-programming-based design approach is also proposed for the convergence rate optimization of single-input multi-output (SIMO) agents. The proposed approaches and the corresponding theoretical results are validated by case studies. [ABSTRACT FROM AUTHOR]

We propose a novel network pruning approach by information preserving of pretrained network weights (filters). Network pruning with the information preserving is formulated as a matrix sketch problem, which is efficiently solved by the off-the-shelf frequent direction method. Our approach, referred to as FilterSketch, encodes the second-order information of pretrained weights, which enables the representation capacity of pruned networks to be recovered with a simple fine-tuning procedure. FilterSketch requires neither training from scratch nor data-driven iterative optimization, leading to a several-orders-of-magnitude reduction of time cost in the optimization of pruning. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of floating-point operations (FLOPs) and prunes 59.9% of network parameters with negligible accuracy cost for ResNet-110. On ILSVRC-2012, it reduces 45.5% of FLOPs and removes 43.0% of parameters with only 0.69% accuracy drop for ResNet-50. Our code and pruned models can be found at https://github.com/lmbxmu/FilterSketch. [ABSTRACT FROM AUTHOR]

A single clustering refers to the partitioning of data such that the similar data are assigned into the same group, whereas the dissimilar data are separated into different groups. Recently, multiview clustering has received significant attention in recent years. However, most existing works tackle the single-clustering scenario, which only use single clustering to partition the data. In practice, nevertheless, the real-world data are complex and can be clustered in multiple ways depending on different interpretations of the data. Unlike these methods, in this article, we apply dual clustering to multiview subspace clustering. We propose a multiview dual-clustering method to simultaneously explore consensus representation and dual-clustering structure in a unified framework. First, multiview features are integrated into a latent embedding representation through a multiview learning process. Second, the dual-clustering segmentation is incorporated into the subspace clustering framework. Finally, the learned dual representations are assigned to the corresponding clusterings. The proposed approach is efficiently solved using an alternating optimization scheme. Extensive experiments demonstrate the superiority of our method on real-world multiview dual- and single-clustering datasets. [ABSTRACT FROM AUTHOR]

Support vector machine (SVM) is a particularly powerful and flexible supervised learning model that analyzes data for both classification and regression, whose usual algorithm complexity scales polynomially with the dimension of data space and the number of data points. To tackle the big data challenge, a quantum SVM algorithm was proposed, which is claimed to achieve exponential speedup for least squares SVM (LS-SVM). Here, inspired by the quantum SVM algorithm, we present a quantum-inspired classical algorithm for LS-SVM. In our approach, an improved fast sampling technique, namely indirect sampling, is proposed for sampling the kernel matrix and classifying. We first consider the LS-SVM with a linear kernel, and then discuss the generalization of our method to nonlinear kernels. Theoretical analysis shows our algorithm can make classification with arbitrary success probability in logarithmic runtime of both the dimension of data space and the number of data points for low rank, low condition number, and high dimensional data matrix, matching the runtime of the quantum SVM. [ABSTRACT FROM AUTHOR]

Image smoothing is a prerequisite for many computer vision and graphics applications. In this article, we raise an intriguing question whether a dataset that semantically describes meaningful structures and unimportant details can facilitate a deep learning model to smooth complex natural images. To answer it, we generate ground-truth labels from easy samples by candidate generation and a screening test and synthesize hard samples in structure-preserving smoothing by blending intricate and multifarious details with the labels. To take full advantage of this dataset, we present a joint edge detection and structure-preserving image smoothing neural network (JESS-Net). Moreover, we propose the distinctive total variation loss as prior knowledge to narrow the gap between synthetic and real data. Experiments on different datasets and real images show clear improvements of our method over the state of the arts in terms of both the image cleanness and structure-preserving ability. Code and dataset are available at https://github.com/YidFeng/Easy2Hard. [ABSTRACT FROM AUTHOR]

This article concerns with terminal recurrent neural network (RNN) models for time-variant computing, featuring finite-valued activation functions (AFs), and finite-time convergence of error variables. Terminal RNNs stand for specific models that admit terminal attractors, and the dynamics of each neuron retains finite-time convergence. The might-existing imperfection in solving time-variant problems, through theoretically examining the asymptotically convergent RNNs, is pointed out for which the finite-time-convergent models are most desirable. The existing AFs are summarized, and it is found that there is a lack of the AFs that take only finite values. A finitely valued terminal RNN, among others, is taken into account, which involves only basic algebraic operations and taking roots. The proposed terminal RNN model is used to solve the time-variant problems undertaken, including the time-variant quadratic programming and motion planning of redundant manipulators. The numerical results are presented to demonstrate effectiveness of the proposed neural network, by which the convergence rate is comparable with that of the existing power-rate RNN. [ABSTRACT FROM AUTHOR]

Twin support vector machine (TWSVM), which constructs two nonparallel classifying hyperplanes, is widely applied to various fields. However, TWSVM solves two quadratic programming problems (QPPs) separately such that the final classifiers lack consistency and enough prediction accuracy. Moreover, by reason of only considering the 1-norm penalty for slack variables, TWSVM is not well defined in the geometrical view. In this article, we propose a novel elastic net nonparallel hyperplane support vector machine (ENNHSVM), which adopts elastic net penalty for slack variables and constructs two nonparallel separating hyperplanes simultaneously. We further discuss the properties of ENNHSVM theoretically and derive the violation tolerance upper bound to better demonstrate the relative violations of training samples in the same class. In particular, we design a safe screening rule for ENNHSVM to speed up the calculations. We finally compare the performance of ENNHSVM on both synthetic datasets and benchmark datasets with the Lagrangian SVM, the twin parametric-margin SVM, the elastic net SVM, the TWSVM, and the nonparallel hyperplane SVM. [ABSTRACT FROM AUTHOR]

In this article, the $\mathcal {H}_{\infty }$ optimization approach is used to study the consensusability margin optimization problems for distributed second-order sampled-data multiagent systems with communication uncertainties. The considered uncertainties are frequency-dependent and bounded in $\mathcal {H}_{\infty }$ norms. Specifically, for both the state-based protocols with relative damping and absolute damping, this article attempts to answer two questions: 1) how to characterize the control parameters for achieving robust consensus; and 2) what is the maximal consensusability margin and how to find, if one exists, the parameter to achieve this optimal performance. It is shown that the consensusability margin optimization problems are constraint optimization problems, which are to be specified by specific problem parameters and can be discussed by designing the network complementary sensitivity function matrices of the closed-loop multiagent systems. Moreover, it is shown that the infimums of the $\mathcal {H}_{\infty }$ norms of the network complementary sensitivity function matrices under both protocols are independent of network topologies. [ABSTRACT FROM AUTHOR]

This article considers the distributed resource allocation problem over a network of n agents with individual optimization functions and coupled resource constraints. The optimization variables are bounded for each agent. We propose an asynchronous distributed gradient descent based method, which is able to tackle time-varying communication networks and communication problems. The main ideas are twofold. First, the degree of freedom of the network is reduced to deal with the global constraint. Second, we utilize delay compensation to counteract the delay of asynchronous communication. We prove that the convergence rate is $O(\frac{1}{k})$ and analyze both the communication complexity and the time complexity. This method is applied into the parallel pump control problem in HVAC system and shows good numerical results. [ABSTRACT FROM AUTHOR]

We propose decentralized primal-dual methods for cooperative multiagent consensus optimization problems over both static and time-varying communication networks, where only local communications are allowed. The objective is to minimize the sum of agent-specific convex functions over conic constraint sets defined by agent-specific nonlinear functions; hence, the optimal consensus decision should lie in the intersection of these private sets. Under the strong convexity assumption, we provide convergence rates for suboptimality, infeasibility, and consensus violation in terms of the number of communications required; examine the effect of underlying network topology on the convergence rates. [ABSTRACT FROM AUTHOR]

This article is concerned with minimizing the average of $n$ cost functions over a network, in which agents may communicate and exchange information with each other. We consider the setting where only noisy gradient information is available. To solve the problem, we study the distributed stochastic gradient descent (DSGD) method and perform a nonasymptotic convergence analysis. For strongly convex and smooth objective functions, in expectation, DSGD asymptotically achieves the optimal network-independent convergence rate compared to centralized stochastic gradient descent. Our main contribution is to characterize the transient time needed for DSGD to approach the asymptotic convergence rate. Moreover, we construct a “hard” optimization problem that proves the sharpness of the obtained result. Numerical experiments demonstrate the tightness of the theoretical results. [ABSTRACT FROM AUTHOR]

Modern convolutional neural network (CNN)-based object detectors focus on feature configuration during training but often ignore feature optimization during inference. In this article, we propose a new feature optimization approach to enhance features and suppress background noise in both the training and inference stages. We introduce a generic inference-aware feature filtering (IFF) module that can be easily combined with existing detectors, resulting in our iffDetector. Unlike conventional open-loop feature calculation approaches without feedback, the proposed IFF module performs the closed-loop feature optimization by leveraging high-level semantics to enhance the convolutional features. By applying the Fourier transform to analyze our detector, we prove that the IFF module acts as a negative feedback that can theoretically guarantee the stability of the feature learning. IFF can be fused with CNN-based object detectors in a plug-and-play manner with little computational cost overhead. Experiments on the PASCAL VOC and MS COCO datasets demonstrate that our iffDetector consistently outperforms state-of-the-art methods with significant margins. [ABSTRACT FROM AUTHOR]

This article develops a communication-efficient algorithm to solve the stochastic optimization problem defined over a distributed network, aiming at reducing the burdensome communication in applications, such as distributed machine learning. Different from the existing works based on quantization and sparsification, we introduce a communication-censoring technique to reduce the transmissions of variables, which leads to our communication-censored distributed stochastic gradient descent (CSGD) algorithm. Specifically, in CSGD, the latest minibatch stochastic gradient at a worker will be transmitted to the server if and only if it is sufficiently informative. When the latest gradient is not available, the stale one will be reused at the server. To implement this communication-censoring strategy, the batch size is increasing in order to alleviate the effect of stochastic gradient noise. Theoretically, CSGD enjoys the same order of convergence rate as that of SGD but effectively reduces communication. Numerical experiments demonstrate the sizable communication saving of CSGD. [ABSTRACT FROM AUTHOR]

The staggering innovations and emergence of numerous deep learning (DL) applications have forced researchers to reconsider hardware architecture to accommodate fast and efficient application-specific computations. Applications, such as object detection, image recognition, speech translation, as well as music synthesis and image generation, can be performed with high accuracy at the expense of substantial computational resources using DL. Furthermore, the desire to adopt Industry 4.0 and smart technologies within the Internet of Things infrastructure has initiated several studies to enable on-chip DL capabilities for resource-constrained devices. Specialized DL processors reduce dependence on cloud servers, improve privacy, lessen latency, and mitigate bandwidth congestion. As we reach the limits of shrinking transistors, researchers are exploring various application-specific hardware architectures to meet the performance and efficiency requirements for DL tasks. Over the past few years, several software optimizations and hardware innovations have been proposed to efficiently perform these computations. In this article, we review several DL accelerators, as well as technologies with emerging devices, to highlight their architectural features in application-specific integrated circuit (IC) and field-programmable gate array (FPGA) platforms. Finally, the design considerations for DL hardware in portable applications have been discussed, along with some deductions about the future trends and potential research directions to innovate DL accelerator architectures further. By compiling this review, we expect to help aspiring researchers widen their knowledge in custom hardware architectures for DL. [ABSTRACT FROM AUTHOR]

The adaptive hinging hyperplane (AHH) model is a popular piecewise linear representation with a generalized tree structure and has been successfully applied in dynamic system identification. In this article, we aim to construct the deep AHH (DAHH) model to extend and generalize the networking of AHH model for high-dimensional problems. The network structure of DAHH is determined through a forward growth, in which the activity ratio is introduced to select effective neurons and no connecting weights are involved between the layers. Then, all neurons in the DAHH network can be flexibly connected to the output in a skip-layer format, and only the corresponding weights are the parameters to optimize. With such a network framework, the backpropagation algorithm can be implemented in DAHH to efficiently tackle large-scale problems and the gradient vanishing problem is not encountered in the training of DAHH. In fact, the optimization problem of DAHH can maintain convexity with convex loss in the output layer, which brings natural advantages in optimization. Different from the existing neural networks, DAHH is easier to interpret, where neurons are connected sparsely and analysis of variance (ANOVA) decomposition can be applied, facilitating to revealing the interactions between variables. A theoretical analysis toward universal approximation ability and explicit domain partitions are also derived. Numerical experiments verify the effectiveness of the proposed DAHH. [ABSTRACT FROM AUTHOR]

Deep learning-based methods have achieved notable progress in removing blocking artifacts caused by lossy JPEG compression on images. However, most deep learning-based methods handle this task by designing black-box network architectures to directly learn the relationships between the compressed images and their clean versions. These network architectures are always lack of sufficient interpretability, which limits their further improvements in deblocking performance. To address this issue, in this article, we propose a model-driven deep unfolding method for JPEG artifacts removal, with interpretable network structures. First, we build a maximum posterior (MAP) model for deblocking using convolutional dictionary learning and design an iterative optimization algorithm using proximal operators. Second, we unfold this iterative algorithm into a learnable deep network structure, where each module corresponds to a specific operation of the iterative algorithm. In this way, our network inherits the benefits of both the powerful model ability of data-driven deep learning method and the interpretability of traditional model-driven method. By training the proposed network in an end-to-end manner, all learnable modules can be automatically explored to well characterize the representations of both JPEG artifacts and image content. Experiments on synthetic and real-world datasets show that our method is able to generate competitive or even better deblocking results, compared with state-of-the-art methods both quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

Multivariate time series data are invasive in different domains, ranging from data center supervision and e-commerce data to financial transactions. This kind of data presents an important challenge for anomaly detection due to the temporal dependency aspect of its observations. In this article, we investigate the problem of unsupervised local anomaly detection in multivariate time series data from temporal modeling and residual analysis perspectives. The residual analysis has been shown to be effective in classical anomaly detection problems. However, it is a nontrivial task in multivariate time series as the temporal dependency between the time series observations complicates the residual modeling process. Methodologically, we propose a unified learning framework to characterize the residuals and their coherence with the temporal aspect of the whole multivariate time series. Experiments on real-world datasets are provided showing the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Recent years have witnessed significant progress of person reidentification (reID) driven by expert-designed deep neural network architectures. Despite the remarkable success, such architectures often suffer from high model complexity and time-consuming pretraining process, as well as the mismatches between the image classification-driven backbones and the reID task. To address these issues, we introduce neural architecture search (NAS) into automatically designing person reID backbones, i.e., reID-NAS, which is achieved via automatically searching attention-based network architectures from scratch. Different from traditional NAS approaches that originated for image classification, we design a reID-based search space as well as a search objective to fit NAS for the reID tasks. In terms of the search space, reID-NAS includes a lightweight attention module to precisely locate arbitrary pedestrian bounding boxes, which is automatically added as attention to the reID architectures. In terms of the search objective, reID-NAS introduces a new retrieval objective to search and train reID architectures from scratch. Finally, we propose a hybrid optimization strategy to improve the search stability in reID-NAS. In our experiments, we validate the effectiveness of different parts in reID-NAS, and show that the architecture searched by reID-NAS achieves a new state of the art, with one order of magnitude fewer parameters on three-person reID datasets. As a concomitant benefit, the reliance on the pretraining process is vastly reduced by reID-NAS, which facilitates one to directly search and train a lightweight reID model from scratch. [ABSTRACT FROM AUTHOR]

Deep learning has become the most powerful machine learning tool in the last decade. However, how to efficiently train deep neural networks remains to be thoroughly solved. The widely used minibatch stochastic gradient descent (SGD) still needs to be accelerated. As a promising tool to better understand the learning dynamic of minibatch SGD, the information bottleneck (IB) theory claims that the optimization process consists of an initial fitting phase and the following compression phase. Based on this principle, we further study typicality sampling, an efficient data selection method, and propose a new explanation of how it helps accelerate the training process of the deep networks. We show that the fitting phase depicted in the IB theory will be boosted with a high signal-to-noise ratio of gradient approximation if the typicality sampling is appropriately adopted. Furthermore, this finding also implies that the prior information of the training set is critical to the optimization process, and the better use of the most important data can help the information flow through the bottleneck faster. Both theoretical analysis and experimental results on synthetic and real-world datasets demonstrate our conclusions. [ABSTRACT FROM AUTHOR]

Although deep neural networks have been proved effective in many applications, they are data hungry, and training deep models often requires laboriously labeled data. However, when labeled data contain erroneous labels, they often lead to model performance degradation. A common solution is to assign each sample with a dynamic weight during optimization, and the weight is adjusted in accordance with the loss. However, those weights are usually unreliable since they are measured by the losses of corrupted labels. Thus, this scheme might impede the discriminative ability of neural networks trained on noisy data. To address this issue, we propose a novel reweighting method, dubbed self-reweighting from class centroids (SRCC), by assigning sample weights based on the similarities between the samples and our online learned class centroids. Since we exploit statistical class centers in the image feature space to reweight data samples in learning, our method is robust to noise caused by corrupted labels. In addition, even after reweighting the noisy data, the decision boundaries might still suffer distortions. Thus, we leverage mixed inputs that are generated by linearly interpolating two random images and their labels to further regularize the boundaries. We employ the learned class centroids to evaluate the confidence of our generated mixed data via measuring feature similarities. During the network optimization, the class centroids are updated as more discriminative feature representations of original images are learned. In doing so, SRCC will generate more robust weighting coefficients for noisy and mixed data and facilitates our feature representation learning in return. Extensive experiments on both the synthetic and real image recognition tasks demonstrate that our method SRCC outperforms the state of the art on learning with noisy data. [ABSTRACT FROM AUTHOR]

The neural-network (NN)-based control method is a new emerging promising technique for controller design in a power electronic circuit (PEC). However, the optimization of NN-based controllers (NNCs) has significant challenges in two aspects. The first challenge is that the search space of the NNC optimization problem is such complex that the global optimization ability of the existing algorithms still needs to be improved. The second challenge is that the training process of the NNC parameters is very computationally expensive and requires a long execution time. Thus, in this article, we develop a powerful evolutionary computation-based algorithm to find a high-quality solution and reduce computational time. First, the differential evolution (DE) algorithm is adopted because it is a powerful global optimizer in solving a complex optimization problem. This can help to overcome the premature convergence in local optima to train the NNC parameters well. Second, to reduce the computational time, the DE is extended to distribute DE (DDE) by dispatching all the individuals to different distributed computing resources for parallel computing. Moreover, a resource-aware strategy (RAS) is designed to further efficiently utilize the resources by adaptively dispatching individuals to resources according to the real-time performance of the resources, which can simultaneously concern the computing ability and load state of each resource. Experimental results show that, compared with some other typical evolutionary algorithms, the proposed algorithm can get significantly better solutions within a shorter computational time. [ABSTRACT FROM AUTHOR]

Domain adaptation is proposed to deal with the challenging problem where the probability distribution of the training source is different from the testing target. Recently, adversarial learning has become the dominating technique for domain adaptation. Usually, adversarial domain adaptation methods simultaneously train a feature learner and a domain discriminator to learn domain-invariant features. Accordingly, how to effectively train the domain-adversarial model to learn domain-invariant features becomes a challenge in the community. To this end, we propose in this article a novel domain adaptation scheme named adversarial entropy optimization (AEO) to address the challenge. Specifically, we minimize the entropy when samples are from the independent distributions of source domain or target domain to improve the discriminability of the model. At the same time, we maximize the entropy when features are from the combined distribution of source domain and target domain so that the domain discriminator can be confused and the transferability of representations can be promoted. This minimax regime is well matched with the core idea of adversarial learning, empowering our model with transferability as well as discriminability for domain adaptation tasks. Also, AEO is flexible and compatible with different deep networks and domain adaptation frameworks. Experiments on five data sets show that our method can achieve state-of-the-art performance across diverse domain adaptation tasks. [ABSTRACT FROM AUTHOR]

We present a probabilistic formulation of risk aware optimal control problems for stochastic differential equations. Risk awareness is in our framework captured by objective functions in which the risk neutral expectation is replaced by a risk function, a nonlinear functional of random variables that accounts for the controller's risk preferences. We state and prove a risk aware minimum principle that gives necessary and sufficient conditions for optimality of generalized control processes taking values on probability measures defined on a given action space. We show that going from the risk neutral to the risk aware case, the minimum principle is modified by the introduction of one additional real-valued stochastic process that acts as a risk adjustment factor. This adjustment process is explicitly given as the expectation, conditional on the filtration at the given time, of an appropriately defined functional derivative of the risk function evaluated at the random total cost. The control model we employ differs from standard relaxed controls, and we elaborate on the differences, and benefits and drawbacks, of the control types; we further give conditions under which the generalized control can be realized using a strict control process. We present an application of the results for a portfolio allocation problem and show that the risk awareness of the objective function gives rise to a risk premium term that is characterized by the risk adjustment process described above. This suggests uses of our results in e.g. pricing of risk modeled by generic risk functions in financial applications. [ABSTRACT FROM AUTHOR]

Learning with label proportions (LLP) deals with the problem that the training data are provided as bags, where the label proportions of training bags rather than the labels of individual training instances are accessible. Existing LLP studies assume that the label proportions of all training bags are accessible. However, in many applications, it is time-consuming to mark all training bags with label proportions, which leads to the problem of learning with both marked and unmarked bags, namely, semisupervised LLP (SLLP). In this work, we propose semisupervised proportional support vector machine (SS- $\propto $ SVM), which extends the proportional SVM ($\propto $ SVM) model to its semisupervised version. To the best of our knowledge, SS- $\propto $ SVM is the first attempt to cope with the SLLP problem. Two realizations, alter-SS- $\propto $ SVM and conv-SS- $\propto $ SVM, which are based on alternating optimization and convex relaxation, respectively, are developed to solve the proposed SS- $\propto $ SVM model. Moreover, we design a cutting plane (CP) method to optimize conv-SS- $\propto $ SVM with a guaranteed convergence rate and present a fast accelerated proximal gradient method to solve the multiple kernel learning subproblem in conv-SS- $\propto $ SVM efficiently. Empirical experiments not only justify the superiority of SS- $\propto $ SVM over its supervised counterpart in classification accuracy but also demonstrate the high competitive computational efficiency of the CP optimization of conv-SS- $\propto $ SVM. [ABSTRACT FROM AUTHOR]

In this article, we investigate the routing problem of packet networks through multiagent reinforcement learning (RL), which is a very challenging topic in distributed and autonomous networked systems. In specific, the routing problem is modeled as a networked multiagent partially observable Markov decision process (MDP). Since the MDP of a network node is not only affected by its neighboring nodes’ policies but also the network traffic demand, it becomes a multitask learning problem. Inspired by recent success of RL and metalearning, we propose two novel model-free multiagent RL algorithms, named multiagent proximal policy optimization (MAPPO) and multiagent metaproximal policy optimization (meta-MAPPO), to optimize the network performances under fixed and time-varying traffic demand, respectively. A practicable distributed implementation framework is designed based on the separability of exploration and exploitation in training MAPPO. Compared with the existing routing optimization policies, our simulation results demonstrate the excellent performances of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Weakly supervised object detection (WSOD) is a challenging task that requires simultaneously learning object detectors and estimating object locations under the supervision of image category labels. Many WSOD methods that adopt multiple instance learning (MIL) have nonconvex objective functions and, therefore, are prone to get stuck in local minima (falsely localize object parts) while missing full object extent during training. In this article, we introduce classical continuation optimization into MIL, thereby creating continuation MIL (C-MIL) with the aim to alleviate the nonconvexity problem in a systematic way. To fulfill this purpose, we partition instances into class-related and spatially related subsets and approximate MIL’s objective function with a series of smoothed objective functions defined within the subsets. We further propose a parametric strategy to implement continuation smooth functions, which enables C-MIL to be applied to instance selection tasks in a uniform manner. Optimizing smoothed loss functions prevents the training procedure from falling prematurely into local minima and facilities learning full object extent. Extensive experiments demonstrate the superiority of CMIL over conventional MIL methods. As a general instance selection method, C-MIL is also applied to supervised object detection to optimize anchors/features, improving the detection performance with a significant margin. [ABSTRACT FROM AUTHOR]

In this article, distributed algorithms are proposed for training a group of neural networks with private data sets. Stochastic gradients are utilized in order to eliminate the requirement for true gradients. To obtain a universal model of the distributed neural networks trained using local data sets only, consensus tools are introduced to derive the model toward the optimum. Most of the existing works employ diminishing learning rates, which are often slow and impracticable for online learning, while constant learning rates are studied in some recent works, but the principle for choosing the rates is not well established. In this article, constant learning rates are adopted to empower the proposed algorithms with tracking ability. Under mild conditions, the convergence of the proposed algorithms is established by exploring the error dynamics of the connected agents, which provides an upper bound for selecting the constant learning rates. Performances of the proposed algorithms are analyzed with and without gradient noises, in the sense of mean square error (MSE). It is proved that the MSE converges with bounded errors determined by the gradient noises, and the MSE converges to zero if the gradient noises are absent. Simulation results are provided to validate the effectiveness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Network controllability robustness (CR) reflects how well a networked system can maintain its controllability against destructive attacks. Its measure is quantified by a sequence of values that record the remaining controllability of the network after a sequence of node-removal or edge-removal attacks. Traditionally, the CR is determined by attack simulations, which is computationally time-consuming or even infeasible. In this article, an improved method for predicting the network CR is developed based on machine learning using a group of convolutional neural networks (CNNs). In this scheme, a number of training data generated by simulations are used to train the group of CNNs for classification and prediction, respectively. Extensive experimental studies are carried out, which demonstrate that 1) the proposed method predicts more precisely than the classical single-CNN predictor; 2) the proposed CNN-based predictor provides a better predictive measure than the traditional spectral measures and network heterogeneity. [ABSTRACT FROM AUTHOR]

Hash bit selection (HBS) aims to find the most discriminative and informative hash bits from a hash pool generated by using different hashing algorithms. It is usually formulated as a binary quadratic programming problem with an information-theoretic objective function and a string-length constraint. In this article, it is equivalently reformulated in the form of a quadratic unconstrained binary optimization problem by augmenting the objective function with a penalty function. The reformulated problem is solved via collaborative neurodynamic optimization (CNO) with a population of classic discrete Hopfield networks. The two most important hyperparameters of the CNO approach are determined based on Monte Carlo test results. Experimental results on three benchmark data sets are elaborated to substantiate the superiority of the collaborative neurodynamic approach to several existing methods for HBS. [ABSTRACT FROM AUTHOR]

We consider inference (filtering) problems over probabilistic graphical models with aggregate data generated by a large population of individuals. We propose a new efficient belief propagation type algorithm over tree graphs with polynomial computational complexity as well as a global convergence guarantee. This is in contrast to previous methods that either exhibit prohibitive complexity as the population grows or do not guarantee convergence. Our method is based on optimal transport, or more specifically, multimarginal optimal transport theory. In particular, we consider an inference problem with aggregate observations, that can be seen as a structured multimarginal optimal transport problem where the cost function decomposes according to the underlying graph. Consequently, the celebrated Sinkhorn/iterative scaling algorithm for multi-marginal optimal transport can be leveraged together with the standard belief propagation algorithm to establish an efficient inference scheme which we call Sinkhorn belief propagation (SBP). We further specialize the SBP algorithm to cases associated with hidden Markov models due to their significance in control and estimation. We demonstrate the performance of our algorithm on applications such as inferring population flow from aggregate observations. We also show that in the special case where the aggregate observations are in Dirac form, our algorithm naturally reduces to the standard belief propagation algorithm. [ABSTRACT FROM AUTHOR]

Optimal control problems are prevalent in model-based control, state and parameter estimation, and experimental design for complex dynamical systems. An approach for obtaining solutions to these problems is based on the notion of parsimonious input parameterization and comprises two tasks: the enumeration of arc sequences followed by the computation of optimal values of a small number of decision variables for each sequence. This article proposes an efficient global solution method for single-input optimal control problems for nonlinear dynamical systems with a potentially large number of states or complex dynamics via sum-of-squares polynomials and parallel computing. The method approximates the problem for a given arc sequence as a polynomial optimization problem that can be efficiently solved to global optimality via semidefinite programming. It is established that the difference between the cost obtained by the proposed method and the globally optimal cost of the original problem is bounded and depends on the polynomial approximation error. The method is illustrated by simulation examples of a reaction system and a rocket. [ABSTRACT FROM AUTHOR]

This article deals with the input–output finite-time stability (IO-FTS) of linear discrete-time systems. In current control science, discrete-time systems play a very important role in many engineering contexts. Moreover, many discrete-time control problems are defined over a finite-interval of time; therefore, the development and application of finite-time control methodologies is of particular relevance. The first contribution of this article is a pair of necessary and sufficient conditions for IO-FTS. The former involves the solution of a set of generalized difference Lyapunov equations; the latter allows one to establish the feasibility of an optimization problem by solving a set of difference linear matrix inequalities (DLMIs). The second contribution of this article is a theorem for IO finite-time stabilization via state feedback, followed by a more general one for stabilization via output feedback. Both conditions are necessary and sufficient, and lead to optimization problems cast in the form of DLMIs. The applicability of the devised results is illustrated through a numerical example. [ABSTRACT FROM AUTHOR]

Due to the corruptions or noises that existed in real-world data sets, the affinity graphs constructed by the classical spectral clustering-based subspace clustering algorithms may not be able to reveal the intrinsic subspace structures of data sets faithfully. In this article, we reconsidered the data reconstruction problem in spectral clustering-based algorithms and proposed the idea of “relation reconstruction.” We pointed out that a data sample could be represented by the neighborhood relation computed between its neighbors and itself. The neighborhood relation could indicate the true membership of its corresponding original data sample to the subspaces of a data set. We also claimed that a data sample’s neighborhood relation could be reconstructed by the neighborhood relations of other data samples; then, we suggested a much different way to define affinity graphs consequently. Based on these propositions, a sparse relation representation (SRR) method was proposed for solving subspace clustering problems. Moreover, by introducing the local structure information of original data sets into SRR, an extension of SRR, namely structured sparse relation representation (SSRR) was presented. We gave an optimization algorithm for solving SRR and SSRR problems and analyzed its computation burden and convergence. Finally, plentiful experiments conducted on different types of databases showed the superiorities of SRR and SSRR. [ABSTRACT FROM AUTHOR]

Low-rank minimization aims to recover a matrix of minimum rank subject to linear system constraint. It can be found in various data analysis and machine learning areas, such as recommender systems, video denoising, and signal processing. Nuclear norm minimization is a dominating approach to handle it. However, such a method ignores the difference among singular values of target matrix. To address this issue, nonconvex low-rank regularizers have been widely used. Unfortunately, existing methods suffer from different drawbacks, such as inefficiency and inaccuracy. To alleviate such problems, this article proposes a flexible model with a novel nonconvex regularizer. Such a model not only promotes low rankness but also can be solved much faster and more accurate. With it, the original low-rank problem can be equivalently transformed into the resulting optimization problem under the rank restricted isometry property (rank-RIP) condition. Subsequently, Nesterov’s rule and inexact proximal strategies are adopted to achieve a novel algorithm highly efficient in solving this problem at a convergence rate of $O(1/K)$ , with $K$ being the iterate count. Besides, the asymptotic convergence rate is also analyzed rigorously by adopting the Kurdyka- ojasiewicz (KL) inequality. Furthermore, we apply the proposed optimization model to typical low-rank problems, including matrix completion, robust principal component analysis (RPCA), and tensor completion. Exhaustively empirical studies regarding data analysis tasks, i.e., synthetic data analysis, image recovery, personalized recommendation, and background subtraction, indicate that the proposed model outperforms state-of-the-art models in both accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Neural networks have been demonstrated to be trainable even with hundreds of layers, which exhibit remarkable improvement on expressive power and provide significant performance gains in a variety of tasks. However, the prohibitive computational cost has become a severe challenge for deploying them on resource-constrained platforms. Meanwhile, widely adopted deep neural network architectures, for example, ResNets or DenseNets, are manually crafted on benchmark datasets, which hamper their generalization ability to other domains. To cope with these issues, we propose an evolutionary algorithm-based method for shallowing deep neural networks (DNNs) at block levels, which is termed as ESNB. Different from existing studies, ESNB utilizes the ensemble view of block-wise DNNs and employs the multiobjective optimization paradigm to reduce the number of blocks while avoiding performance degradation. It automatically discovers shallower network architectures by pruning less informative blocks, and employs knowledge distillation to recover the performance. Moreover, a novel prior knowledge incorporation strategy is proposed to improve the exploration ability of the evolutionary search process, and a correctness-aware knowledge distillation strategy is designed for better knowledge transferring. Experimental results show that the proposed method can effectively accelerate the inference of DNNs while achieving superior performance when compared with the state-of-the-art competing methods. [ABSTRACT FROM AUTHOR]