Search

Generative adversarial network (GAN) has greatly improved the quality of unsupervised image generation. Previous GAN-based methods often require a large amount of high-quality training data while producing a small number (e.g., tens) of classes. This work aims to scale up GANs to thousands of classes meanwhile reducing the use of high-quality data in training. We propose an image generation method based on conditional transferring features, which can capture pixel-level semantic changes when transforming low-quality images into high-quality ones. Moreover, self-supervision learning is integrated into our GAN architecture to provide more label-free semantic supervisory information observed from the training data. As such, training our GAN architecture requires much fewer high-quality images with a small number of additional low-quality images. The experiments on CIFAR-10 and STL-10 show that even removing 30% high-quality images from the training set, our method can still outperform previous ones. The scalability on object classes has been experimentally validated: our method with 30% fewer high-quality images obtains the best quality in generating 1,000 ImageNet classes, as well as generating all 3,755 classes of CASIA-HWDB1.0 Chinese handwriting characters.

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradients, and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant, causing the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning., Comment: To be appeared in ICLR 2020 (poster presentation)

Decomposing an exposure effect on an outcome into separate natural indirect effects through multiple mediators requires strict assumptions, such as correctly postulating the causal structure of the mediators, and no unmeasured confounding among the mediators. In contrast, interventional indirect effects for multiple mediators can be identified even when - as often - the mediators either have an unknown causal structure, or share unmeasured common causes, or both. Existing estimation methods for interventional indirect effects require calculating each distinct indirect effect in turn. This can quickly become unwieldy or unfeasible, especially when investigating indirect effect measures that may be modified by observed baseline characteristics. In this article, we introduce simplified estimation procedures for such heterogeneous interventional indirect effects using interventional effect models. Interventional effect models are a class of marginal structural models that encode the interventional indirect effects as causal model parameters, thus readily permitting effect modification by baseline covariates using (statistical) interaction terms. The mediators and outcome can be continuous or noncontinuous. We propose two estimation procedures: one using inverse weighting by the counterfactual mediator density or mass functions, and another using Monte Carlo integration. The former has the advantage of not requiring an outcome model, but is susceptible to finite sample biases due to highly variable weights. The latter has the advantage of consistent estimation under a correctly specified (parametric) outcome model, but is susceptible to biases due to extrapolation.

With the rapid scaling up of deep neural networks (DNNs), extensive research studies on network model compression such as weight pruning have been performed for improving deployment efficiency. This work aims to advance the compression beyond the weights to neuron activations. We propose the joint regularization technique which simultaneously regulates the distribution of weights and activations. By distinguishing and leveraging the significance difference among neuron responses and connections during learning, the jointly pruned network, namely \textit{JPnet}, optimizes the sparsity of activations and weights for improving execution efficiency. The derived deep sparsification of JPnet reveals more optimization space for the existing DNN accelerators dedicated for sparse matrix operations. We thoroughly evaluate the effectiveness of joint regularization through various network models with different activation functions and on different datasets. With $0.4\%$ degradation constraint on inference accuracy, a JPnet can save $72.3\% \sim 98.8\%$ of computation cost compared to the original dense models, with up to $5.2\times$ and $12.3\times$ reductions in activation and weight numbers, respectively.

Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at https://github.com/wenwei202/autogrow., Comment: KDD 2020

Let $[X,\lambda]$ be a principally polarized abelian variety over a finite field with commutative endomorphism ring; further suppose that either $X$ is ordinary or the field is prime. Motivated by an equidistribution heuristic, we introduce a factor $\nu_v([X,\lambda])$ for each place $v$ of $\mathbb Q$, and show that the product of these factors essentially computes the size of the isogeny class of $[X,\lambda]$. The derivation of this mass formula depends on a formula of Kottwitz and on analysis of measures on the group of symplectic similitudes, and in particular does not rely on a calculation of class numbers., Comment: Added author; significantly simplified global calculation in section 5; made other, smaller improvements

Following the ideas of Ginzburg, for a subgroup $K$ of a connected reductive $\mathbb{R}$-group $G$ we introduce the notion of $K$-admissible $D$-modules on a homogeneous $G$-variety $Z$. We show that $K$-admissible $D$-modules are regular holonomic when $K$ and $Z$ are absolutely spherical. This framework includes: (i) the relative characters attached to two spherical subgroups $H_1$ and $H_2$, provided that the twisting character $\chi_i$ factors through the maximal reductive quotient of $H_i$, for $i = 1, 2$; (ii) localization on $Z$ of Harish-Chandra modules; (iii) the generalized matrix coefficients when $K(\mathbb{R})$ is maximal compact. This complements the holonomicity proven by Aizenbud--Gourevitch--Minchenko. The use of regularity is illustrated by a crude estimate on the growth of $K$-admissible distributions which based on tools from subanalytic geometry., Comment: 33 pages. Various improvements

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured group-lasso regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end training time of ResNet50 for ImageNet by reducing computation cost by 40% in FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the inter-accelerator communication by 55%.

Motivated by recent experimental observations of coherent many-body revivals in a constrained Rydberg atom chain, we construct a weak quasi-local deformation of the Rydberg blockade Hamiltonian, which makes the revivals virtually perfect. Our analysis suggests the existence of an underlying non-integrable Hamiltonian which supports an emergent SU(2)-spin dynamics within a small subspace of the many-body Hilbert space. We show that such perfect dynamics necessitates the existence of atypical, nonergodic energy eigenstates - quantum many-body scars. Furthermore, using these insights, we construct a toy model that hosts exact quantum many-body scars, providing an intuitive explanation of their origin. Our results offer specific routes to enhancing coherent many-body revivals, and provide a step towards establishing the stability of quantum many-body scars in the thermodynamic limit., Comment: 6+11 pages, 3+6 figures

The performance of Deep Neural Networks (DNNs) keeps elevating in recent years with increasing network depth and width. To enable DNNs on edge devices like mobile phones, researchers proposed several network compression methods including pruning, quantization and factorization. Among the factorization-based approaches, low-rank approximation has been widely adopted because of its solid theoretical rationale and efficient implementations. Several previous works attempted to directly approximate a pre-trained model by low-rank decomposition; however, small approximation errors in parameters can ripple a large prediction loss. As a result, performance usually drops significantly and a sophisticated fine-tuning is required to recover accuracy. We argue that it is not optimal to separate low-rank approximation from training. Unlike previous works, this paper integrates low rank approximation and regularization into the training. We propose Trained Rank Pruning (TRP), which iterates low rank approximation and training. TRP maintains the capacity of original network while imposes low-rank constraints during training. A stochastic sub-gradient descent optimized nuclear regularization is utilized to further encourage low rank in TRP. The TRP trained network has low-rank structure in nature, and can be approximated with negligible performance loss, eliminating fine-tuning after low rank approximation. The methods are comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous compression methods using low rank approximation. Code is available: https://github.com/yuhuixu1993/Trained-Rank-Pruning, Comment: Accepted by NIPS2019 EMC2 workshop, the same version as the withdrawn arXiv:1910.04576