Search

Large-scale computing systems are increasingly using accelerators such as GPUs to enable peta- and exa-scale levels of compute to meet the needs of Machine Learning (ML) and scientific computing applications. Given the widespread and growing use of ML, including in some scientific applications, optimizing these clusters for ML workloads is particularly important. However, recent work has demonstrated that accelerators in these clusters can suffer from performance variability and this variability can lead to resource under-utilization and load imbalance. In this work we focus on how clusters schedulers, which are used to share accelerator-rich clusters across many concurrent ML jobs, can embrace performance variability to mitigate its effects. Our key insight to address this challenge is to characterize which applications are more likely to suffer from performance variability and take that into account while placing jobs on the cluster. We design a novel cluster scheduler, PAL, which uses performance variability measurements and application-specific profiles to improve job performance and resource utilization. PAL also balances performance variability with locality to ensure jobs are spread across as few nodes as possible. Overall, PAL significantly improves GPU-rich cluster scheduling: across traces for six ML workload applications spanning image, language, and vision models with a variety of variability profiles, PAL improves geomean job completion time by 42%, cluster utilization by 28%, and makespan by 47% over existing state-of-the-art schedulers.

Scientists are increasingly exploring and utilizing the massive parallelism of general-purpose accelerators such as GPUs for scientific breakthroughs. As a result, datacenters, hyperscalers, national computing centers, and supercomputers have procured hardware to support this evolving application paradigm. These systems contain hundreds to tens of thousands of accelerators, enabling peta- and exa-scale levels of compute for scientific workloads. Recent work demonstrated that power management (PM) can impact application performance in CPU-based HPC systems, even when machines have the same architecture and SKU (stock keeping unit). This variation occurs due to manufacturing variability and the chip's PM. However, while modern HPC systems widely employ accelerators such as GPUs, it is unclear how much this variability affects applications. Accordingly, we seek to characterize the extent of variation due to GPU PM in modern HPC and supercomputing systems. We study a variety of applications that stress different GPU components on five large-scale computing centers with modern GPUs: Oak Ridge's Summit, Sandia's Vortex, TACC's Frontera and Longhorn, and Livermore's Corona. These clusters use a variety of cooling methods and GPU vendors. In total, we collect over 18,800 hours of data across more than 90% of the GPUs in these clusters. Regardless of the application, cluster, GPU vendor, and cooling method, our results show significant variation: 8% (max 22%) average performance variation even though the GPU architecture and vendor SKU are identical within each cluster, with outliers up to 1.5X slower than the median GPU. These results highlight the difficulty in efficiently using existing GPU clusters for modern HPC and scientific workloads, and the need to embrace variability in future accelerator-based systems., Comment: 14 pages, 18 figures, to appear at The 34th International Conference for High Performance Computing, Networking, Storage, and Analysis (SC '22)

Gamma-ray bursts (GRBs) are fascinating events due to their panchromatic nature. We study optical plateaus in GRB afterglows via an extended search into archival data. We comprehensively analyze all published GRBs with known redshifts and optical plateaus observed by many ground-based telescopes (e.g., Subaru Telescope, RATIR) around the world and several space-based observatories such as the Neil Gehrels Swift Observatory. We fit 500 optical light curves (LCs), showing the existence of the plateau in 179 cases. This sample is 75% larger than the previous one (arXiv:2105.10717), and it is the largest compilation so far of optical plateaus. We discover the 3D fundamental plane relation at optical wavelengths using this sample. This correlation is between the rest-frame time at the end of the plateau emission, $T^{*}_{\rm opt}$, its optical luminosity, $L_{\rm opt}$, and the peak in the optical prompt emission, $L_{\rm peak, opt}$, thus resembling the three-dimensional (3D) X-ray fundamental plane relation (arXiv:1604.06840). We correct our sample for redshift evolution and selection effects, discovering that this correlation is indeed intrinsic to GRB physics. We investigate the rest-frame end time distributions in X-rays and optical ($T^{*}_{\rm opt}$, $T^{*}_{\rm X}$), and conclude that the plateau is achromatic only when selection biases are not considered. We also investigate if the 3D optical correlation may be a new discriminant between optical GRB classes and find that there is no significant separation between the classes compared to the Gold sample plane after correcting for evolution., Comment: 27 pages, 9 figures, 4 tables. Accepted for publication in ApJS. Reworded one sentence for clarity

In the enterprise email search setting, the same search engine often powers multiple enterprises from various industries: technology, education, manufacturing, etc. However, using the same global ranking model across different enterprises may result in suboptimal search quality, due to the corpora differences and distinct information needs. On the other hand, training an individual ranking model for each enterprise may be infeasible, especially for smaller institutions with limited data. To address this data challenge, in this paper we propose a domain adaptation approach that fine-tunes the global model to each individual enterprise. In particular, we propose a novel application of the Maximum Mean Discrepancy (MMD) approach to information retrieval, which attempts to bridge the gap between the global data distribution and the data distribution for a given individual enterprise. We conduct a comprehensive set of experiments on a large-scale email search engine, and demonstrate that the MMD approach consistently improves the search quality for multiple individual domains, both in comparison to the global ranking model, as well as several competitive domain adaptation baselines including adversarial learning methods., Comment: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

We show that the basic classification framework alone can be used to tackle some of the most challenging tasks in image synthesis. In contrast to other state-of-the-art approaches, the toolkit we develop is rather minimal: it uses a single, off-the-shelf classifier for all these tasks. The crux of our approach is that we train this classifier to be adversarially robust. It turns out that adversarial robustness is precisely what we need to directly manipulate salient features of the input. Overall, our findings demonstrate the utility of robustness in the broader machine learning context. Code and models for our experiments can be found at https://git.io/robust-apps.

An important goal in deep learning is to learn versatile, high-level feature representations of input data. However, standard networks' representations seem to possess shortcomings that, as we illustrate, prevent them from fully realizing this goal. In this work, we show that robust optimization can be re-cast as a tool for enforcing priors on the features learned by deep neural networks. It turns out that representations learned by robust models address the aforementioned shortcomings and make significant progress towards learning a high-level encoding of inputs. In particular, these representations are approximately invertible, while allowing for direct visualization and manipulation of salient input features. More broadly, our results indicate adversarial robustness as a promising avenue for improving learned representations. Our code and models for reproducing these results is available at https://git.io/robust-reps .

Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and pervasiveness remain unclear. We demonstrate that adversarial examples can be directly attributed to the presence of non-robust features: features derived from patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans. After capturing these features within a theoretical framework, we establish their widespread existence in standard datasets. Finally, we present a simple setting where we can rigorously tie the phenomena we observe in practice to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

A recent line of work has uncovered a new form of data poisoning: so-called \emph{backdoor} attacks. These attacks are particularly dangerous because they do not affect a network's behavior on typical, benign data. Rather, the network only deviates from its expected output when triggered by a perturbation planted by an adversary. In this paper, we identify a new property of all known backdoor attacks, which we call \emph{spectral signatures}. This property allows us to utilize tools from robust statistics to thwart the attacks. We demonstrate the efficacy of these signatures in detecting and removing poisoned examples on real image sets and state of the art neural network architectures. We believe that understanding spectral signatures is a crucial first step towards designing ML systems secure against such backdoor attacks, Comment: 16 pages, accepted to NIPS 2018

The study of adversarial robustness has so far largely focused on perturbations bound in p-norms. However, state-of-the-art models turn out to be also vulnerable to other, more natural classes of perturbations such as translations and rotations. In this work, we thoroughly investigate the vulnerability of neural network--based classifiers to rotations and translations. While data augmentation offers relatively small robustness, we use ideas from robust optimization and test-time input aggregation to significantly improve robustness. Finally we find that, in contrast to the p-norm case, first-order methods cannot reliably find worst-case perturbations. This highlights spatial robustness as a fundamentally different setting requiring additional study. Code available at https://github.com/MadryLab/adversarial_spatial and https://github.com/MadryLab/spatial-pytorch., Comment: ICML 2019. Presented in NIPS 2017 Workshop on Machine Learning and Computer Security as "A Rotation and a Translation Suffice: Fooling CNNs with Simple Transformations."

A simple graph-product type construction shows that for all natural numbers $r \ge q$, there exists an edge-coloring of the complete graph on $2^r$ vertices using $r$ colors where the graph consisting of the union of arbitrary $q$ color classes has chromatic number $2^q$. We show that for each fixed natural number $q$, if there exists an edge-coloring of the complete graph on $n$ vertices using $r$ colors where the graph consisting of the union of arbitrary $q$ color classes has chromatic number at most $2^q -1 $, then $n$ must be sub-exponential in $r$. This answers a question of Conlon, Fox, Lee, and Sudakov.

Curtis-Ingerman-Morrow characterize response matrices for circular planar electrical networks as symmetric square matrices with row sums zero and non-negative circular minors. In this paper, we study this positivity phenomenon more closely, from both algebraic and combinatorial perspectives. Extending work of Postnikov, we introduce electrical positroids, which are the sets of circular minors which can simultaneously be positive in a response matrix. We give a self-contained axiomatic description of these electrical positroids. In the second part of the paper, we discuss a naturally arising example of a Laurent phenomenon algebra, as studied by Lam-Pylyavskyy. We investigate the clusters in this algebra, building off of initial work by Kenyon-Wilson, using an analogue of weak separation, as was originally introduced by Leclerc-Zelevinsky., Comment: 30 pages, 4 figures

Following de Verdi\`{e}re-Gitler-Vertigan and Curtis-Ingerman-Morrow, we prove a host of new results on circular planar electrical networks. We introduce a poset EP_{n} of electrical networks with n boundary vertices, giving two equivalent characterizations, one combinatorial and the other topological. We then investigate various properties of the EP_{n}, proving that it is graded by number of edges of critical representatives. Finally, we answer various enumerative questions related to EP_{n}, adapting methods of Callan and Stein-Everett., Comment: 29 pages, 12 figures