Search

Analog in-memory computing (AIMC) has emerged as a promising solution to overcome the von Neumann bottleneck, accelerating neural network computations and improving computational efficiency. While AIMC has demonstrated success with architectures such as CNNs, MLPs, and RNNs, deploying transformer-based models using AIMC presents unique challenges. Transformers are expected to handle diverse downstream tasks and adapt to new user data or instructions after deployment, which requires more flexible approaches to suit AIMC constraints. In this paper, we propose a novel method for deploying pre-trained transformer models onto AIMC hardware. Unlike traditional approaches requiring hardware-aware training, our technique allows direct deployment without the need for retraining the original model. Instead, we utilize lightweight, low-rank adapters -- compact modules stored in digital cores -- to adapt the model to hardware constraints. We validate our approach on MobileBERT, demonstrating accuracy on par with, or even exceeding, a traditional hardware-aware training approach. Our method is particularly appealing in multi-task scenarios, as it enables a single analog model to be reused across multiple tasks. Moreover, it supports on-chip adaptation to new hardware constraints and tasks without updating analog weights, providing a flexible and versatile solution for real-world AI applications. Code is available.

Virtual environments provide a rich and controlled setting for collecting detailed data on human behavior, offering unique opportunities for predicting human trajectories in dynamic scenes. However, most existing approaches have overlooked the potential of these environments, focusing instead on static contexts without considering userspecific factors. Employing the CREATTIVE3D dataset, our work models trajectories recorded in virtual reality (VR) scenes for diverse situations including road-crossing tasks with user interactions and simulated visual impairments. We propose Diverse Context VR Human Motion Prediction (DiVR), a cross-modal transformer based on the Perceiver architecture that integrates both static and dynamic scene context using a heterogeneous graph convolution network. We conduct extensive experiments comparing DiVR against existing architectures including MLP, LSTM, and transformers with gaze and point cloud context. Additionally, we also stress test our model's generalizability across different users, tasks, and scenes. Results show that DiVR achieves higher accuracy and adaptability compared to other models and to static graphs. This work highlights the advantages of using VR datasets for context-aware human trajectory modeling, with potential applications in enhancing user experiences in the metaverse. Our source code is publicly available at https://gitlab.inria.fr/ffrancog/creattive3d-divr-model.

A key challenge for Deep Neural Network (DNN) algorithms is their vulnerability to adversarial attacks. Inherently non-deterministic compute substrates, such as those based on Analog In-Memory Computing (AIMC), have been speculated to provide significant adversarial robustness when performing DNN inference. In this paper, we experimentally validate this conjecture for the first time on an AIMC chip based on Phase Change Memory (PCM) devices. We demonstrate higher adversarial robustness against different types of adversarial attacks when implementing an image classification network. Additional robustness is also observed when performing hardware-in-the-loop attacks, for which the attacker is assumed to have full access to the hardware. A careful study of the various noise sources indicate that a combination of stochastic noise sources (both recurrent and non-recurrent) are responsible for the adversarial robustness and that their type and magnitude disproportionately effects this property. Finally, it is demonstrated, via simulations, that when a much larger transformer network is used to implement a Natural Language Processing (NLP) task, additional robustness is still observed.

Kernel functions are vital ingredients of several machine learning algorithms, but often incur significant memory and computational costs. We introduce an approach to kernel approximation in machine learning algorithms suitable for mixed-signal Analog In-Memory Computing (AIMC) architectures. Analog In-Memory Kernel Approximation addresses the performance bottlenecks of conventional kernel-based methods by executing most operations in approximate kernel methods directly in memory. The IBM HERMES Project Chip, a state-of-the-art phase-change memory based AIMC chip, is utilized for the hardware demonstration of kernel approximation. Experimental results show that our method maintains high accuracy, with less than a 1% drop in kernel-based ridge classification benchmarks and within 1% accuracy on the Long Range Arena benchmark for kernelized attention in Transformer neural networks. Compared to traditional digital accelerators, our approach is estimated to deliver superior energy efficiency and lower power consumption. These findings highlight the potential of heterogeneous AIMC architectures to enhance the efficiency and scalability of machine learning applications.

We study concavity properties of positive solutions to the Logarithmic Schr\"odinger equation $-\Delta u=u\, \log u^2$ in a general convex domain with Dirichlet conditions. To this aim, we analyse the auxiliary Lane-Emden problems $-\Delta u = \sigma\, (u^q-u)$ and build, for any $\sigma>0$ and $q>1$, solutions $u_q$ such that $u_q^{(1-q)/2}$ is convex. By choosing $\sigma_q=2/(1-q)$ and letting $q \to 1^+$ we eventually construct a solution $u$ of the Logarithmic Schr\"odinger equation such that $\log u$ is concave. This seems one of the few attempts in studying concavity properties for superlinear, sign changing sources. To get the result, we both make inspections on the constant rank theorem and develop Liouville theorems on convex epigraphs, which might be useful in other frameworks.

We study the following nonlinear Schr\"odinger equation and we look for normalized solutions $(u,\mu)\in H^1({\bf R}^N)\times{\bf R}$ for a given $m>0$ and $N\geq 2$ \[ -\Delta u + \mu u = g(u)\quad \text{in}\ {\bf R}^N, \qquad \frac{1}{2}\int_{{\bf R}^N} u^2 dx = m. \] We assume that $g$ has an $L^2$-critical growth, both at the origin and at infinity. That is, for $p=1+\frac{4}{N}$, $g(s)=|s|^{p-1}s +h(s)$, $h(s)=o(|s|^p)$ as $s\sim 0$ and $s\sim\infty$. The $L^2$-critical exponent $p$ is very special for this problem; in the power case $g(s) = |s|^{p-1}s$ a solution exists only for the specific mass $m=m_1$, where $m_1=\frac{1}{2}\int_{{\bf R}^N}\omega_1^2\, dx$ is the mass of a least energy solution $\omega_1$ of $-\Delta \omega+\omega=\omega^p$ in ${\bf R}^N$. We prove the existence of a positive solution for $m=m_1$ when $h$ has a sublinear growth at infinity, i.e., $h(s)=o(s)$ as $s\sim\infty$. In contrast, we show non-existence results for $h(s)\not=o(s)$ ($s\sim 0$) under a suitable monotonicity condition., Comment: 59 pages

We report the discovery of eight new recurrent changing-look (CL) active galactic nuclei (AGNs), including seven re-brightening turn-off AGNs and one fading turn-on AGN. These systems are valuable for placing constraints on the duration of dim and bright states, which may be linked to the AGN duty cycle or disk instability. Long-term optical light curve analysis reveals that many objects in our sample exhibit a prolonged plateau during the dim states lasting 4 to 7 years, with gradual turn-on/off process. We observe no significant difference between the turn-on and turn-off timescales, and this timescale is broadly consistent with the heating/cooling front propagation timescale. The comparison between optical and infrared variations supports that these transitions are driven by changes in accretion disk emission rather than dust obscuration. Our discovery significantly increases the previously identified recurrent CL AGN sample from eleven objects to nineteen, demonstrating that some AGNs can enter dormancy and reawaken on timescales of a few years, which provides useful information for understanding AGN episodic accretion., Comment: 12 pages, 4 figures; submitted

The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensitive workflows, AI-HPC convergence, multi-facility workflows, heterogeneous HPC environments, user experience, and FAIR computational workflows. The integration of AI and exascale computing has revolutionized scientific workflows, enabling higher-fidelity models and complex, time-sensitive processes, while introducing challenges in managing heterogeneous environments and multi-facility data dependencies. The rise of large language models is driving computational demands to zettaflop scales, necessitating modular, adaptable systems and cloud-service models to optimize resource utilization and ensure reproducibility. Multi-facility workflows present challenges in data movement, curation, and overcoming institutional silos, while diverse hardware architectures require integrating workflow considerations into early system design and developing standardized resource management tools. The summit emphasized improving user experience in workflow systems and ensuring FAIR workflows to enhance collaboration and accelerate scientific discovery. Key recommendations include developing standardized metrics for time-sensitive workflows, creating frameworks for cloud-HPC integration, implementing distributed-by-design workflow modeling, establishing multi-facility authentication protocols, and accelerating AI integration in HPC workflow management. The summit also called for comprehensive workflow benchmarks, workflow-specific UX principles, and a FAIR workflow maturity model, highlighting the need for continued collaboration in addressing the complex challenges posed by the convergence of AI, HPC, and multi-facility research environments.

We introduce nvTorchCam, an open-source library under the Apache 2.0 license, designed to make deep learning algorithms camera model-independent. nvTorchCam abstracts critical camera operations such as projection and unprojection, allowing developers to implement algorithms once and apply them across diverse camera models--including pinhole, fisheye, and 360 equirectangular panoramas, which are commonly used in automotive and real estate capture applications. Built on PyTorch, nvTorchCam is fully differentiable and supports GPU acceleration and batching for efficient computation. Furthermore, deep learning models trained for one camera type can be directly transferred to other camera types without requiring additional modification. In this paper, we provide an overview of nvTorchCam, its functionality, and present various code examples and diagrams to demonstrate its usage. Source code and installation instructions can be found on the nvTorchCam GitHub page at https://github.com/NVlabs/nvTorchCam., Comment: Source code and installation instructions are available at https://github.com/NVlabs/nvTorchCam

While vision transformers are able to solve a wide variety of computer vision tasks, no pre-training method has yet demonstrated the same scaling laws as observed in language models. Autoregressive models show promising results, but are commonly trained on images that are cropped or transformed into square images, which distorts or destroys information present in the input. To overcome this limitation, we propose NARAIM, a vision model pre-trained with an autoregressive objective that uses images in their native aspect ratio. By maintaining the native aspect ratio, we preserve the original spatial context, thereby enhancing the model's ability to interpret visual information. In our experiments, we show that maintaining the aspect ratio improves performance on a downstream classification task., Comment: Accepted to NeurIPS, see https://openreview.net/forum?id=7Iuh8VWU66

Diffusion models have achieved unprecedented performance in image generation, yet they suffer from slow inference due to their iterative sampling process. To address this, early-exiting has recently been proposed, where the depth of the denoising network is made adaptive based on the (estimated) difficulty of each sampling step. Here, we discover an interesting "phase transition" in the sampling process of current adaptive diffusion models: the denoising network consistently exits early during the initial sampling steps, until it suddenly switches to utilizing the full network. Based on this, we propose accelerating generation by employing a shallower denoising network in the initial sampling steps and a deeper network in the later steps. We demonstrate empirically that our dual-backbone approach, DuoDiff, outperforms existing early-exit diffusion methods in both inference speed and generation quality. Importantly, DuoDiff is easy to implement and complementary to existing approaches for accelerating diffusion., Comment: Accepted to NeurIPS, see https://openreview.net/forum?id=G7E4tNmmHD

To achieve accurate and unbiased predictions, Machine Learning (ML) models rely on large, heterogeneous, and high-quality datasets. However, this could raise ethical and legal concerns regarding copyright and authorization aspects, especially when information is gathered from the Internet. With the rise of generative models, being able to track data has become of particular importance, especially since they may (un)intentionally replicate copyrighted contents. Therefore, this work proposes Copyright Audit via Prompts generation (CAP), a framework for automatically testing whether an ML model has been trained with unauthorized data. Specifically, we devise an approach to generate suitable keys inducing the model to reveal copyrighted contents. To prove its effectiveness, we conducted an extensive evaluation campaign on measurements collected in four IoT scenarios. The obtained results showcase the effectiveness of CAP, when used against both realistic and synthetic datasets.

We analyze the problem of locating a public facility on a line in a society where agents have either single-peaked or single-dipped preferences. We consider the domain analyzed in Alcalde-Unzu et al. (2024), where the type of preference of each agent is public information, but the location of her peak/dip as well as the rest of the preference are unknown. We characterize all strategy-proof and type-anonymous rules on this domain. Building on existing results, we provide a two-step characterization": first, the median between the peaks and a collection of fixed values is computed (Moulin, 1980), resulting in either a single alternative or a pair of contiguous alternatives. If the outcome of the median is a pair, we apply a double-quota majority method" in the second step to choose between the two alternatives in the pair (Moulin, 1983). We also show the additional conditions that type-anonymity imposes on the strategy-proof rules characterized by Alcalde-Unzu et al. (2024). Finally, we show the equivalence between the two characterizations.