Search

Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference., Comment: 17 pages

Concept erasure techniques have recently gained significant attention for their potential to remove unwanted concepts from text-to-image models. While these methods often demonstrate success in controlled scenarios, their robustness in real-world applications and readiness for deployment remain uncertain. In this work, we identify a critical gap in evaluating sanitized models, particularly in terms of their performance across various concept dimensions. We systematically investigate the failure modes of current concept erasure techniques, with a focus on visually similar, binomial, and semantically related concepts. We propose that these interconnected relationships give rise to a phenomenon of concept entanglement resulting in ripple effects and degradation in image quality. To facilitate more comprehensive evaluation, we introduce EraseBENCH, a multi-dimensional benchmark designed to assess concept erasure methods with greater depth. Our dataset includes over 100 diverse concepts and more than 1,000 tailored prompts, paired with a comprehensive suite of metrics that together offer a holistic view of erasure efficacy. Our findings reveal that even state-of-the-art techniques struggle with maintaining quality post-erasure, indicating that these approaches are not yet ready for real-world deployment. This highlights the gap in reliability of the concept erasure techniques., Comment: 11 pages main; 9 pages supplemental material

This paper presents an open-loop articulated 6-degree-of-freedom (DoF) robotic system for three-dimensional (3D) scanning of objects by contact-based method. A digitizer probe was used to detect contact with the object. Inverse kinematics (IK) was used to determine the joint angles of the robot corresponding to the probe position and orientation, and straight-line trajectory planning was implemented for motion. The system can take single-point measurements and 3D scans of freeform surfaces. Specifying the scanning area's size, position, and density, the system automatically scans the designated volume. The system produces 3D scans in Standard Triangle Language (STL) format, ensuring compatibility with commonly used 3D software. Tests based on ASME B89.4.22 standards were conducted to quantify accuracy and repeatability. The point cloud from the scans was compared to the original 3D model of the object.

Lubricant viscoelasticity arises due to a finite polymer relaxation time ($\lambda$) and can provide beneficial effects. In applications, such as bearings, gears, biological joints, etc., where the height-to-length ratio is small ($H_0 / \ell_x$) and the shear due to the wall velocity ($U_0$) is high, a simplified two-dimensional computational analysis across the channel length and height reveals a finite increase in the load carrying capacity of the film purely due to polymer elasticity. In channels with a finite length-to-width ratio, $a$, the spanwise effects can be significant, but the resulting mathematical model is computationally intensive. In this work, we propose simpler reduced-order models, namely via a (i) first-order perturbation in the Deborah number ($\lambda U_0 / H_0$), and the (ii) viscoelastic Reynolds approach extended from \textit{Ahmed, H., \& Biancofiore, L. (2021). A new approach for modeling viscoelastic thin film lubrication. Journal of Non-Newtonian Fluid Mechanics, 292, 104524}. We predict the variation in the net vertical force exerted on the channel walls (for a fixed film height) versus increasing viscoelasticity, and the channel aspect ratio. The models predict an increase in the net force, which is zero for the Newtonian case, versus both the Deborah number and the channel aspect ratio. Interestingly, for a fixed $De$, this force varies strongly between the two limiting cases (i) $a << 1$; an infinitely wide, and (ii) $a >> 1$; an infinitely short channel channel, implying a change in the polymers response. Furthermore, we observe a different trend (i) for a spanwise varying channel, in which a peak is observed between the two limits, and (ii) for a spanwise uniform channel, where the largest load value is for $a << 1$. When $a$ is O($1$), the viscoelastic response varies strongly and spanwise effects cannot be ignored.

Residual resistance ratio (RRR) of Cu stabilizer in REBCO coated conductor is an important design parameter for REBCO magnets. In this work, we measured RRR of electroplated Cu stabilizer in commercial REBCO tapes. Over 130 samples were measured for the quality assurance programs of REBCO magnet projects at the National High Magnetic Field Laboratory, USA (NHMFL). The average RRR value was above 50. In order to investigate the factors that influence RRR, several samples were analyzed by using scanning electron microscopy, secondary ion mass spectroscopy, and inductively coupled plasma mass spectroscopy. We found that, in our samples, RRR was strongly correlated with the grain size. We demonstrated that RRR was primarily determined by grain boundary resistivity. Lower RRR was also strongly correlated with higher concentration of chlorine impurity. This is explained by that higher chlorine impurity hindered the grain growth in the room temperature self annealing process resulting smaller grain. Smaller grain resulted in lower RRR. In addition, thermal annealing significantly enhanced RRR. An activation energy of 0.4 eV was obtained from the annealing experiment which corresponds to the activation of Cu grain growth., Comment: 18 pages

Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents by avoiding the need for expensive online learning. Despite strong generalization in some respects, agents are often remarkably brittle to minor visual variations in control-irrelevant factors such as the background or camera viewpoint. In this paper, we present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents for solving continuous control tasks from visual input in the presence of visual distractors. In contrast to prior works, our dataset (a) combines locomotion and navigation tasks of varying difficulties, (b) includes static and dynamic visual variations, (c) considers data generated by policies with different skill levels, (d) systematically returns pairs of state and pixel observation, (e) is an order of magnitude larger, and (f) includes tasks with hidden goals. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods. First, we find that pretrained representations do not help policy learning on DMC-VB, and we highlight a large representation gap between policies learned on pixel observations and on states. Second, we demonstrate when expert data is limited, policy learning can benefit from representations pretrained on (a) suboptimal data, and (b) tasks with stochastic hidden goals. Our dataset and benchmark code to train and evaluate agents are available at: https://github.com/google-deepmind/dmc_vision_benchmark., Comment: NeurIPS 2024 Datasets and Benchmarks Track. Dataset available at: https://github.com/google-deepmind/dmc_vision_benchmark

Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferrable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for $\approx +10dB$ gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER's features. Our demo can be accessed at https://kushalvyas.github.io/strainer.html ., Comment: Project Website: https://kushalvyas.github.io/strainer.html

In signals of opportunity (SOPs)-based positioning utilizing low Earth orbit (LEO) satellites, ephemeris data derived from two-line element files can introduce increasing error over time. To handle the erroneous measurement, an additional base receiver with a known position is often used to compensate for the effect of ephemeris error when positioning the user terminal (UT). However, this approach is insufficient for the long baseline (the distance between the base receiver and UT) as it fails to adequately correct Doppler shift measurement errors caused by ephemeris inaccuracies, resulting in degraded positioning performance. Moreover, the lack of clock synchronization between the base receiver and UT exacerbates erroneous Doppler shift measurements. To address these challenges, we put forth a robust double-difference Doppler shift-based positioning framework, coined 3DPose, to handle the clock synchronization issue between the base receiver and UT, and positioning degradation due to the long baseline. The proposed 3DPose framework leverages double-difference Doppler shift measurements to eliminate the clock synchronization issue and incorporates a novel ephemeris error correction algorithm to enhance UT positioning accuracy in case of the long baseline. The algorithm specifically characterizes and corrects the Doppler shift measurement errors arising from erroneous ephemeris data, focusing on satellite position errors in the tangential direction. To validate the effectiveness of the proposed framework, we conduct comparative analyses across three different scenarios, contrasting its performance with the existing differential Doppler positioning method. The results demonstrate that the proposed 3DPose framework achieves an average reduction of 90% in 3-dimensional positioning errors compared to the existing differential Doppler approach., Comment: 32 pages, 8 figures, 2 tables

The artificial intelligence (AI) world is running out of real data for training increasingly large generative models, resulting in accelerating pressure to train on synthetic data. Unfortunately, training new generative models with synthetic data from current or past generation models creates an autophagous (self-consuming) loop that degrades the quality and/or diversity of the synthetic data in what has been termed model autophagy disorder (MAD) and model collapse. Current thinking around model autophagy recommends that synthetic data is to be avoided for model training lest the system deteriorate into MADness. In this paper, we take a different tack that treats synthetic data differently from real data. Self-IMproving diffusion models with Synthetic data (SIMS) is a new training concept for diffusion models that uses self-synthesized data to provide negative guidance during the generation process to steer a model's generative process away from the non-ideal synthetic data manifold and towards the real data distribution. We demonstrate that SIMS is capable of self-improvement; it establishes new records based on the Fr\'echet inception distance (FID) metric for CIFAR-10 and ImageNet-64 generation and achieves competitive results on FFHQ-64 and ImageNet-512. Moreover, SIMS is, to the best of our knowledge, the first prophylactic generative AI algorithm that can be iteratively trained on self-generated synthetic data without going MAD. As a bonus, SIMS can adjust a diffusion model's synthetic data distribution to match any desired in-domain target distribution to help mitigate biases and ensure fairness.

Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fr\'echet Inception Distance using a large number of generated and real samples. However, generative model performance is not uniform across the learned manifold, e.g., for \textit{foundation models} like Stable Diffusion generation performance can vary significantly based on the conditioning or initial noise vector being denoised. In this paper we study the relationship between the \textit{local geometry of the learned manifold} and downstream generation. Based on the theory of continuous piecewise-linear (CPWL) generators, we use three geometric descriptors - scaling ($\psi$), rank ($\nu$), and complexity ($\delta$) - to characterize a pre-trained generative model manifold locally. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization. Finally we demonstrate that training a \textit{reward model} on the local geometry can allow controlling the likelihood of a generated sample under the learned distribution., Comment: Pre-print. 11 pages main, 8 pages app., 28 figures

In this paper, we overview one promising avenue of progress at the mathematical foundation of deep learning: the connection between deep networks and function approximation by affine splines (continuous piecewise linear functions in multiple dimensions). In particular, we will overview work over the past decade on understanding certain geometrical properties of a deep network's affine spline mapping, in particular how it tessellates its input space. As we will see, the affine spline connection and geometrical viewpoint provide a powerful portal through which to view, analyze, and improve the inner workings of a deep network., Comment: Accepted for publication at 'Notices of the American Mathematical Society'

We develop Scalable Latent Exploration Score (ScaLES) to mitigate over-exploration in Latent Space Optimization (LSO), a popular method for solving black-box discrete optimization problems. LSO utilizes continuous optimization within the latent space of a Variational Autoencoder (VAE) and is known to be susceptible to over-exploration, which manifests in unrealistic solutions that reduce its practicality. ScaLES is an exact and theoretically motivated method leveraging the trained decoder's approximation of the data distribution. ScaLES can be calculated with any existing decoder, e.g. from a VAE, without additional training, architectural changes, or access to the training data. Our evaluation across five LSO benchmark tasks and three VAE architectures demonstrates that ScaLES enhances the quality of the solutions while maintaining high objective values, leading to improvements over existing solutions. We believe that new avenues to LSO will be opened by ScaLES ability to identify out of distribution areas, differentiability, and computational tractability. Open source code for ScaLES is available at https://github.com/OmerRonen/scales.

Grokking, or delayed generalization, is a phenomenon where generalization in a deep neural network (DNN) occurs long after achieving near zero training error. Previous studies have reported the occurrence of grokking in specific controlled settings, such as DNNs initialized with large-norm parameters or transformers trained on algorithmic datasets. We demonstrate that grokking is actually much more widespread and materializes in a wide range of practical settings, such as training of a convolutional neural network (CNN) on CIFAR10 or a Resnet on Imagenette. We introduce the new concept of delayed robustness, whereby a DNN groks adversarial examples and becomes robust, long after interpolation and/or generalization. We develop an analytical explanation for the emergence of both delayed generalization and delayed robustness based on the local complexity of a DNN's input-output mapping. Our local complexity measures the density of so-called linear regions (aka, spline partition regions) that tile the DNN input space and serves as a utile progress measure for training. We provide the first evidence that, for classification problems, the linear regions undergo a phase transition during training whereafter they migrate away from the training samples (making the DNN mapping smoother there) and towards the decision boundary (making the DNN mapping less smooth there). Grokking occurs post phase transition as a robust partition of the input space thanks to the linearization of the DNN mapping around the training points. Website: https://bit.ly/grok-adversarial, Comment: ICML 2024. Website: https://bit.ly/grok-adversarial. Pages 24, Figures 36

This paper introduces Low-shot Object Learning with Mutual Exclusivity Bias (LSME), the first computational framing of mutual exclusivity bias, a phenomenon commonly observed in infants during word learning. We provide a novel dataset, comprehensive baselines, and a state-of-the-art method to enable the ML community to tackle this challenging learning task. The goal of LSME is to analyze an RGB image of a scene containing multiple objects and correctly associate a previously-unknown object instance with a provided category label. This association is then used to perform low-shot learning to test category generalization. We provide a data generation pipeline for the LSME problem and conduct a thorough analysis of the factors that contribute to its difficulty. Additionally, we evaluate the performance of multiple baselines, including state-of-the-art foundation models. Finally, we present a baseline approach that outperforms state-of-the-art models in terms of low-shot accuracy., Comment: Accepted at NeurIPS 2023, Datasets and Benchmarks Track. Project website https://ngailapdi.github.io/projects/lsme/