Search

In this work, we investigate a particular implicit bias in the gradient descent training process, which we term "Feature Averaging", and argue that it is one of the principal factors contributing to non-robustness of deep neural networks. Despite the existence of multiple discriminative features capable of classifying data, neural networks trained by gradient descent exhibit a tendency to learn the average (or certain combination) of these features, rather than distinguishing and leveraging each feature individually. In particular, we provide a detailed theoretical analysis of the training dynamics of gradient descent in a two-layer ReLU network for a binary classification task, where the data distribution consists of multiple clusters with orthogonal cluster center vectors. We rigorously prove that gradient descent converges to the regime of feature averaging, wherein the weights associated with each hidden-layer neuron represent an average of the cluster centers (each center corresponding to a distinct feature). It leads the network classifier to be non-robust due to an attack that aligns with the negative direction of the averaged features. Furthermore, we prove that, with the provision of more granular supervised information, a two-layer multi-class neural network is capable of learning individual features, from which one can derive a binary classifier with the optimal robustness under our setting. Besides, we also conduct extensive experiments using synthetic datasets, MNIST and CIFAR-10 to substantiate the phenomenon of feature averaging and its role in adversarial robustness of neural networks. We hope the theoretical and empirical insights can provide a deeper understanding of the impact of the gradient descent training on feature learning process, which in turn influences the robustness of the network, and how more detailed supervision may enhance model robustness., Comment: 78 pages, 10 figures

Adversarial training is a widely-applied approach to training deep neural networks to be robust against adversarial perturbation. However, although adversarial training has achieved empirical success in practice, it still remains unclear why adversarial examples exist and how adversarial training methods improve model robustness. In this paper, we provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory. Specifically, we focus on a multiple classification setting, where the structured data can be composed of two types of features: the robust features, which are resistant to perturbation but sparse, and the non-robust features, which are susceptible to perturbation but dense. We train a two-layer smoothed ReLU convolutional neural network to learn our structured data. First, we prove that by using standard training (gradient descent over the empirical risk), the network learner primarily learns the non-robust feature rather than the robust feature, which thereby leads to the adversarial examples that are generated by perturbations aligned with negative non-robust feature directions. Then, we consider the gradient-based adversarial training algorithm, which runs gradient ascent to find adversarial examples and runs gradient descent over the empirical risk at adversarial examples to update models. We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning to improve the network robustness. Finally, we also empirically validate our theoretical findings with experiments on real-image datasets, including MNIST, CIFAR10 and SVHN., Comment: 34 pages, Mathematics of Modern Machine Learning Workshop at NeurIPS 2024

Similar to surprising performance in the standard deep learning, deep nets trained by adversarial training also generalize well for $\textit{unseen clean data (natural data)}$. However, despite adversarial training can achieve low robust training error, there exists a significant $\textit{robust generalization gap}$. We call this phenomenon the $\textit{Clean Generalization and Robust Overfitting (CGRO)}$. In this work, we study the CGRO phenomenon in adversarial training from two views: $\textit{representation complexity}$ and $\textit{training dynamics}$. Specifically, we consider a binary classification setting with $N$ separated training data points. $\textit{First}$, we prove that, based on the assumption that we assume there is $\operatorname{poly}(D)$-size clean classifier (where $D$ is the data dimension), ReLU net with only $O(N D)$ extra parameters is able to leverages robust memorization to achieve the CGRO, while robust classifier still requires exponential representation complexity in worst case. $\textit{Next}$, we focus on a structured-data case to analyze training dynamics, where we train a two-layer convolutional network with $O(N D)$ width against adversarial perturbation. We then show that a three-stage phase transition occurs during learning process and the network provably converges to robust memorization regime, which thereby results in the CGRO. $\textit{Besides}$, we also empirically verify our theoretical analysis by experiments in real-image recognition datasets., Comment: 28 pages, Bridging the Gap Between Practice and Theory in Deep Learning Workshop at ICLR 2024

Moire materials, created by lattice-mismatch or/and twist-angle, have spurred great interest in excavating novel quantum phases of matter. Latterly, emergent interfacial ferroelectricity has been surprisingly found in spatial inversion symmetry broken systems, such as rhombohedral-stacked bilayer transition metal dichalcogenides (TMDs). However, the evolution of moire superlattices corresponding to polarization switching and hysteresis is still unclear, which is crucial for giving insight into the interplay between lattice symmetry and band topology, as well as developing optoelectronic memory devices. Here we report on the observation of phonon splitting at strain soliton networks in reconstructed moire superlattices, arising from the twisting and relaxing induced strong three-fold rotational symmetry (C3) breaking. The interval of phonon splitting is tunable by a perpendicular displacement field and exhibits ferroelectric-related hysteresis loops. These phonon evolution features are attributed to the contribution of moire solitons disentangling and lattice viscosity during the motion of domain walls. Moreover, we demonstrate a proof-of-principle moire ferroelectric tunneling junction, whose barrier is modified by net polarization with a tunneling electroresistance of ~10^4. Our work not only reveals the lattice dynamics of moire solitons but also presents a potential pathway for future ferroelectric-based optoelectronic memory devices., Comment: 24 pages, 4 figures

It is well-known that modern neural networks are vulnerable to adversarial examples. To mitigate this problem, a series of robust learning algorithms have been proposed. However, although the robust training error can be near zero via some methods, all existing algorithms lead to a high robust generalization error. In this paper, we provide a theoretical understanding of this puzzling phenomenon from the perspective of expressive power for deep neural networks. Specifically, for binary classification problems with well-separated data, we show that, for ReLU networks, while mild over-parameterization is sufficient for high robust training accuracy, there exists a constant robust generalization gap unless the size of the neural network is exponential in the data dimension $d$. This result holds even if the data is linear separable (which means achieving standard generalization is easy), and more generally for any parameterized function classes as long as their VC dimension is at most polynomial in the number of parameters. Moreover, we establish an improved upper bound of $\exp({\mathcal{O}}(k))$ for the network size to achieve low robust generalization error when the data lies on a manifold with intrinsic dimension $k$ ($k \ll d$). Nonetheless, we also have a lower bound that grows exponentially with respect to $k$ -- the curse of dimensionality is inevitable. By demonstrating an exponential separation between the network size for achieving low robust training and generalization error, our results reveal that the hardness of robust generalization may stem from the expressive power of practical models., Comment: 25 pages; to appear in NeurIPS 2022

Pruning well-trained neural networks is effective to achieve a promising accuracy-efficiency trade-off in computer vision regimes. However, most of existing pruning algorithms only focus on the classification task defined on the source domain. Different from the strong transferability of the original model, a pruned network is hard to transfer to complicated downstream tasks such as object detection arXiv:arch-ive/2012.04643. In this paper, we show that the image-level pretrain task is not capable of pruning models for diverse downstream tasks. To mitigate this problem, we introduce image reconstruction, a pixel-level task, into the traditional pruning framework. Concretely, an autoencoder is trained based on the original model, and then the pruning process is optimized with both autoencoder and classification losses. The empirical study on benchmark downstream tasks shows that the proposed method can outperform state-of-the-art results explicitly., Comment: Under review

Excitons in van der Waals (vdW) stacking interfaces can be trapped in ordered moire potential arrays giving rise to attractive phenomenons of quantum optics and bosonic many-body effects. Compare to the prevalent transition metal dichalcogenides (TMDs) systems, due to the wide bandgap and low dielectric constant, excitons in twist-stacked hexagonal boron nitride (hBN) are anticipated trapped in deeper moire potential, which enhances the strength of interactions. However, constrained by the common low detectivity of weak light-emitting in the deep-ultraviolet (DUV) bands, the moire excitons in twisthBN remain elusive. Here, we report that a remarkable DUV emitting band (peak located at ~260 nm) only emerges at the twisted stacking area of hBN, which is performed by a high collection efficiency and spatially-resolved cathodoluminescence (CL) at room temperature. Significant peak redshifting contrast to defect-bound excitons of bulk hBN indicates the giant trapping effects of moir\'e potential for excitons. The observation of deeply trapped excitons motivates further studies of bosonic strongly correlation physics based on the twist-hBN system., Comment: 20 pages, 9 pictures

Two-dimensional van der Waals crystals arise limitless scope for designing novel combinations of physical properties via controlling the stacking order or twist angle of individual layers. Lattice orientation between stacked monolayers is significant not only for the engineering symmetry breaking but also for the study of many-body quantum phases and band topology. So far the state-of-art exfoliation approaches focusing on achievements of quality, size, yield, and scalability while lacking sufficient information on lattice orientation. Consequently, interlayer alignment is usually determined by later experiments, such as second harmonic generation spectroscopy, which increased the number of trials and errors for a designed artificial ordering and hampered the efficiency of systematic study. Herein, we report a lattice orientation distinguishable exfoliation method via gold favor epitaxy along the specific atomic step edges, meanwhile, fulfill requirements of high-quality, large-size, and high-yield of monolayers. Hexagonal- and rhombohedral-stacking configurations of bilayer transition metal dichalcogenides are built directly at once result of foreseeing the lattice orientation. Optical spectroscopy, electron diffraction, and angle-resolved photoemission spectroscopy are used to study crystals quality, symmetric breaking, and band tuning, which supports the exfoliating mechanism we proposed. This strategy shows the ability for facilitates the development of ordering stacking especially for multilayers assembling in the future., Comment: 29 pages, 7 figures

The neural network with $1$-Lipschitz property based on $\ell_\infty$-dist neuron has a theoretical guarantee in certified $\ell_\infty$ robustness. However, due to the inherent difficulties in the training of the network, the certified accuracy of previous work is limited. In this paper, we propose two approaches to deal with these difficuties. Aiming at the characteristics of the training process based on $\ell_\infty$-norm neural network, we introduce the EMA method to improve the training process. Considering the randomness of the training algorithm, we propose an ensemble method based on trained base models that have the $1$-Lipschitz property and gain significant improvement in the small parameter network. Moreover, we give the theoretical analysis of the ensemble method based on the $1$-Lipschitz property on the certified robustness, which ensures the effectiveness and stability of the algorithm. Our code is available at https://github.com/Theia-4869/EMA-and-Ensemble-Lip-Networks.

Rigorous model-based analysis can help inform state-level energy and climate policy. In this study, we utilize an open-source energy system optimization model and publicly available datasets to examine future electricity generation, CO2 emissions, and CO2 abatement costs for the North Carolina electric power sector through 2050. Model scenarios include uncertainty in future fuel prices, a hypothetical CO2 cap, and an extended renewable portfolio standard. Across the modeled scenarios, solar photovoltaics represent the most cost-effective low-carbon technology, while trade-offs among carbon constrained scenarios largely involve natural gas and renewables. We also develop a new method to calculate break-even costs, which indicate the capital costs at which different technologies become cost-effective within the model. Significant variation in break-even costs are observed across different technologies and scenarios. We illustrate how break-even costs can be used to inform the development of an extended renewable portfolio standard in North Carolina. Utilizing the break-even costs to calibrate a tax credit for onshore wind, we find that the resultant wind deployment displaces other renewables, and thus has a negligible effect on CO2 emissions. Such insights can provide crucial guidance to policymakers weighing different policy options. This study provides an analytical framework to conduct similar analyses in other states using an open source model and freely available datasets., Comment: In press; 24 pages, 4 figures. Environmental Science & Technology (2019)

Energy system optimization models (ESOMs) should be used in an interactive way to uncover knife-edge solutions, explore alternative system configurations, and suggest different ways to achieve policy objectives under conditions of deep uncertainty. In this paper, we do so by employing an existing optimization technique called modeling to generate alternatives (MGA), which involves a change in the model structure in order to systematically explore the near-optimal decision space. The MGA capability is incorporated into Tools for Energy Model Optimization and Analysis (Temoa), an open source framework that also includes a technology rich, bottom up ESOM. In this analysis, Temoa is used to explore alternative energy futures in a simplified single region energy system that represents the U.S. electric sector and a portion of the light duty transport sector. Given the dataset limitations, we place greater emphasis on the methodological approach rather than specific results., Comment: 22 pages, 8 figures