Search

Gradient-based methods for the distributed training of residual networks (ResNets) typically require a forward pass of the input data, followed by back-propagating the error gradient to update model parameters, which becomes time-consuming as the network goes deeper. To break the algorithmic locking and exploit synchronous module parallelism in both the forward and backward modes, auxiliary-variable methods have attracted much interest lately but suffer from significant communication overhead and lack of data augmentation. In this work, a novel joint learning framework for training realistic ResNets across multiple compute devices is established by trading off the storage and recomputation of external auxiliary variables. More specifically, the input data of each independent processor is generated from its low-capacity auxiliary network (AuxNet), which permits the use of data augmentation and realizes forward unlocking. The backward passes are then executed in parallel, each with a local loss function that originates from the penalty or augmented Lagrangian (AL) methods. Finally, the proposed AuxNet is employed to reproduce the updated auxiliary variables through an end-to-end training process. We demonstrate the effectiveness of our methods on ResNets and WideResNets across CIFAR-10, CIFAR-100, and ImageNet datasets, achieving speedup over the traditional layer-serial training method while maintaining comparable testing accuracy.

Recently, some works found an interesting phenomenon that adversarially robust classifiers can generate good images comparable to generative models. We investigate this phenomenon from an energy perspective and provide a novel explanation. We reformulate adversarial example generation, adversarial training, and image generation in terms of an energy function. We find that adversarial training contributes to obtaining an energy function that is flat and has low energy around the real data, which is the key for generative capability. Based on our new understanding, we further propose a better adversarial training method, Joint Energy Adversarial Training (JEAT), which can generate high-quality images and achieve new state-of-the-art robustness under a wide range of attacks. The Inception Score of the images (CIFAR-10) generated by JEAT is 8.80, much better than original robust classifiers (7.50)., Comment: Accepted by ICCV 2021, Oral

Recent breakthroughs of Neural Architecture Search (NAS) extend the field's research scope towards a broader range of vision tasks and more diversified search spaces. While existing NAS methods mostly design architectures on a single task, algorithms that look beyond single-task search are surging to pursue a more efficient and universal solution across various tasks. Many of them leverage transfer learning and seek to preserve, reuse, and refine network design knowledge to achieve higher efficiency in future tasks. However, the enormous computational cost and experiment complexity of cross-task NAS are imposing barriers for valuable research in this direction. Existing NAS benchmarks all focus on one type of vision task, i.e., classification. In this work, we propose TransNAS-Bench-101, a benchmark dataset containing network performance across seven tasks, covering classification, regression, pixel-level prediction, and self-supervised tasks. This diversity provides opportunities to transfer NAS methods among tasks and allows for more complex transfer schemes to evolve. We explore two fundamentally different types of search space: cell-level search space and macro-level search space. With 7,352 backbones evaluated on seven tasks, 51,464 trained models with detailed training information are provided. With TransNAS-Bench-101, we hope to encourage the advent of exceptional NAS algorithms that raise cross-task search efficiency and generalizability to the next level. Our dataset file will be available at Mindspore, VEGA., Comment: Published at CVPR 2021. 8 pages main paper, 13 pages in total

Gradient-based algorithms for training ResNets typically require a forward pass of the input data, followed by back-propagating the objective gradient to update parameters, which are time-consuming for deep ResNets. To break the dependencies between modules in both the forward and backward modes, auxiliary-variable methods such as the penalty and augmented Lagrangian (AL) approaches have attracted much interest lately due to their ability to exploit layer-wise parallelism. However, we observe that large communication overhead and lacking data augmentation are two key challenges of these methods, which may lead to low speedup ratio and accuracy drop across multiple compute devices. Inspired by the optimal control formulation of ResNets, we propose a novel serial-parallel hybrid training strategy to enable the use of data augmentation, together with downsampling filters to reduce the communication cost. The proposed strategy first trains the network parameters by solving a succession of independent sub-problems in parallel and then corrects the network parameters through a full serial forward-backward propagation of data. Such a strategy can be applied to most of the existing layer-parallel training methods using auxiliary variables. As an example, we validate the proposed strategy using penalty and AL methods on ResNet and WideResNet across MNIST, CIFAR-10 and CIFAR-100 datasets, achieving significant speedup over the traditional layer-serial training methods while maintaining comparable accuracy.

Neural Architecture Search (NAS) achieved many breakthroughs in recent years. In spite of its remarkable progress, many algorithms are restricted to particular search spaces. They also lack efficient mechanisms to reuse knowledge when confronting multiple tasks. These challenges preclude their applicability, and motivate our proposal of CATCH, a novel Context-bAsed meTa reinforcement learning (RL) algorithm for transferrable arChitecture searcH. The combination of meta-learning and RL allows CATCH to efficiently adapt to new tasks while being agnostic to search spaces. CATCH utilizes a probabilistic encoder to encode task properties into latent context variables, which then guide CATCH's controller to quickly "catch" top-performing networks. The contexts also assist a network evaluator in filtering inferior candidates and speed up learning. Extensive experiments demonstrate CATCH's universality and search efficiency over many other widely-recognized algorithms. It is also capable of handling cross-domain architecture search as competitive networks on ImageNet, COCO, and Cityscapes are identified. This is the first work to our knowledge that proposes an efficient transferrable NAS solution while maintaining robustness across various settings., Comment: Published at ECCV2020

Real-time scheduling and locking protocols are fundamental facilities to construct time-critical systems. For parallel real-time tasks, predictable locking protocols are required when concurrent sub-jobs mutually exclusive access to shared resources. This paper for the first time studies the distributed synchronization framework of parallel real-time tasks, where both tasks and global resources are partitioned to designated processors, and requests to each global resource are conducted on the processor on which the resource is partitioned. We extend the Distributed Priority Ceiling Protocol (DPCP) for parallel tasks under federated scheduling, with which we proved that a request can be blocked by at most one lower-priority request. We develop task and resource partitioning heuristics and propose analysis techniques to safely bound the task response times. Numerical evaluation (with heavy tasks on 8-, 16-, and 32-core processors) indicates that the proposed methods improve the schedulability significantly compared to the state-of-the-art locking protocols under federated scheduling.

In recent years, a variety of normalization methods have been proposed to help train neural networks, such as batch normalization (BN), layer normalization (LN), weight normalization (WN), group normalization (GN), etc. However, mathematical tools to analyze all these normalization methods are lacking. In this paper, we first propose a lemma to define some necessary tools. Then, we use these tools to make a deep analysis on popular normalization methods and obtain the following conclusions: 1) Most of the normalization methods can be interpreted in a unified framework, namely normalizing pre-activations or weights onto a sphere; 2) Since most of the existing normalization methods are scaling invariant, we can conduct optimization on a sphere with scaling symmetry removed, which can help stabilize the training of network; 3) We prove that training with these normalization methods can make the norm of weights increase, which could cause adversarial vulnerability as it amplifies the attack. Finally, a series of experiments are conducted to verify these claims., Comment: Accepted by AAAI 2020

Multi-objective Neural Architecture Search (NAS) aims to discover novel architectures in the presence of multiple conflicting objectives. Despite recent progress, the problem of approximating the full Pareto front accurately and efficiently remains challenging. In this work, we explore the novel reinforcement learning (RL) based paradigm of non-stationary policy gradient (NPG). NPG utilizes a non-stationary reward function, and encourages a continuous adaptation of the policy to capture the entire Pareto front efficiently. We introduce two novel reward functions with elements from the dominant paradigms of scalarization and evolution. To handle non-stationarity, we propose a new exploration scheme using cosine temperature decay with warm restarts. For fast and accurate architecture evaluation, we introduce a novel pre-trained shared model that we continuously fine-tune throughout training. Our extensive experimental study with various datasets shows that our framework can approximate the full Pareto front well at fast speeds. Moreover, our discovered cells can achieve supreme predictive performance compared to other multi-objective NAS methods, and other single-objective NAS methods at similar network sizes. Our work demonstrates the potential of NPG as a simple, efficient, and effective paradigm for multi-objective NAS.

The Neural Architecture Search (NAS) problem is typically formulated as a graph search problem where the goal is to learn the optimal operations over edges in order to maximise a graph-level global objective. Due to the large architecture parameter space, efficiency is a key bottleneck preventing NAS from its practical use. In this paper, we address the issue by framing NAS as a multi-agent problem where agents control a subset of the network and coordinate to reach optimal architectures. We provide two distinct lightweight implementations, with reduced memory requirements (1/8th of state-of-the-art), and performances above those of much more computationally expensive methods. Theoretically, we demonstrate vanishing regrets of the form O(sqrt(T)), with T being the total number of rounds. Finally, aware that random search is an, often ignored, effective baseline we perform additional experiments on 3 alternative datasets and 2 network configurations, and achieve favourable results in comparison.

In this paper we extend the Falicov-Kimball model (FKM) to the case where the "quasi-particles" entering the FKM are not "ordinary" fermions. As an example we first discuss how the FKM can be generalized to the case with spin-dependent hopping. Afterwards we discuss several cases where the "quasi-particles" entering the FKM are Majorana fermions (extended Majorana-Falicov-Kimball Model (MFKM). Two examples of extended MFKM are discussed in detail: (i) a $p$-wave BCS superconductor on a bipartite lattice and (ii) a BCS-Anderson model. We also discuss the most general forms of extended MFKM, including a brief discussion on the case where the Majorana fermions represent spins, but not real fermion particles., Comment: 9 pages, 6 figures

In this paper, we study doped HgTe/CdTe quantum well with Hubbard-type interaction under perpendicular magnetic field using a lattice Bernevig-Hughes-Zhang (BHZ) model with a bulk inversion asymmetry (BIA) term. We show that the BIA term is strongly enhanced by interaction around the region when the band inversion of the topological insulator is destroyed by a magnetic field. The enhanced BIA term creates edge-like electronic states which can explain the experimentally discovered edge conductance in doped HgTe/CdTe quantum well at similar magnetic field regime., Comment: 8 pages and 7 figures

We introduce in this paper an exact solvable BCS-Hubbard model in arbitrary dimensions. The model describes a p-wave BCS superconductor with equal spin pairing moving on a bipartite (cubic, square etc.) lattice with on site Hubbard interaction $U$. We show that the model becomes exactly solvable for arbitrary $U$ when the BCS pairing amplitude $\Delta$ equals the hopping amplitude $t$. The nature of the solution is described in detail in this paper. The construction of the exact solution is parallel to the exactly solvable Kitaev honeycomb model for $S=1/2$ quantum spins and can be viewed as a generalization of Kitaev's construction to $S=1/2$ interacting lattice fermions. The BCS-Hubbard model discussed in this paper is just an example of a large class of exactly solvable lattice fermion models that can be constructed similarly., Comment: 5 pages, 3 figures

The effects of Hubbard-type on-site interactions on the BHZ model is studied in this paper for model parameters appropriate for the HgTe/CdTe quantum well. Within a simple mean field theory we search for plausible magnetic instabilities in the model and find that the ground state becomes {\em ferromagnetic} when the interaction strength between electrons in hole orbital is strong enough. The result can be understood by an approximate mapping of the Hubbard-BHZ model to the one band Hubbard model. The same mapping suggests that the magnetic and/or other ordered phases are more likely to occur in large gap topological insulators whose occupations are close to 1/2 for both electron and hole orbital., Comment: 5 pages, 5 figures

There is a new class of two-dimensional magnetic materials polymeric iron (III) acetate fabricated recently in which Fe ions form a star lattice. We study the thermodynamics of Ising spins on the star lattice with exact analytic method and Monte Carlo simulations. Mapping the star lattice to the honeycomb lattice, we obtain the partition function for the system with asymmetric interactions. The free energy, internal energy, specific heat, entropy and susceptibility are presented, which can be used to determine the sign of the interactions in the real materials. Moreover, we find the rich phase diagrams of the system as a function of interactions, temperature and external magnetic field. For frustrated interactions without external field, the ground state is disordered (spin liquid) with residual entropy 1.522 per unit cell. When a weak field is applied, the system enters a ferrimagnetic phase with residual entropy ln4 per unit cell., Comment: 7 pages, 14 figures

We analyze and compare the effect of spatial and spin anisotropy on spin conductivity in a two dimensional S=1/2 Heisenberg quantum magnet on a square lattice. We explore the model in both the Neel antiferromagnetic (AF) phase and the collinear antiferromagnetic (CAF) phase. We find that in contrast to the effects of spin anisotropy in the Heisenberg model, spatial anisotropy in the AF phase does not suppress the zero temperature regular part of the spin conductivity in the zero frequency limit - rather it enhances it. We also explore the finite temperature effects on the Drude weight in the AF phase for various spatial and spin anisotropy parameters. We find that the Drude weight goes to zero as the temperature approaches zero. At finite temperatures (within the collision less approximation) enhancing spatial anisotropy increases the Drude weight value and increasing spin anisotropy decreases the Drude weight value. In the CAF phase (within the non-interacting approximation) the zero frequency spin conductivity has a finite value for non-zero values of the spatial anisotropy parameter. In the CAF phase increasing the spatial anisotropy parameter suppresses the regular part of the spin conductivity response at zero frequency. Furthermore, we find that the CAF phase displays a spike in the spin conductivity not seen in the AF phase. Inclusion of the smallest amount of spin anisotropy causes a gap to develop in the spin conductivity response of both the AF and CAF phase. Based on these studies we conclude that materials with spatial anisotropy are better spin conductors than those with spin anisotropy both at zero and finite temperatures. We utilize exchange parameter ratios for real material systems as inputs to the computation of spin conductivity., Comment: 10 pages, 8 figures