Search

With the increase in the number of parameters in large language models, the process of pre-training and fine-tuning increasingly demands larger volumes of GPU memory. A significant portion of this memory is typically consumed by the optimizer state. To overcome this challenge, recent approaches such as low-rank adaptation (LoRA (Hu et al., 2021)), low-rank gradient projection (GaLore (Zhao et al., 2024)), and blockwise optimization (BAdam (Luo et al., 2024)) have been proposed. However, in all these algorithms, the $\textit{effective rank of the weight updates remains low-rank}$, which can lead to a substantial loss of information from the gradient. This loss can be critically important, especially during the pre-training stage. In this paper, we introduce $\texttt{FRUGAL}$ ($\textbf{F}$ull-$\textbf{R}$ank $\textbf{U}$pdates with $\textbf{G}$r$\textbf{A}$dient sp$\textbf{L}$itting), a new memory-efficient optimization framework. $\texttt{FRUGAL}$ leverages gradient splitting to perform low-dimensional updates using advanced algorithms (such as Adam), while updates along the remaining directions are executed via state-free methods like SGD or signSGD (Bernstein et al., 2018). Our framework can be integrated with various low-rank update selection techniques, including GaLore and BAdam. We provide theoretical convergence guarantees for our framework when using SGDM for low-dimensional updates and SGD for state-free updates. Additionally, our method consistently outperforms concurrent approaches across various fixed memory budgets, achieving state-of-the-art results in pre-training and fine-tuning tasks while balancing memory efficiency and performance metrics.

Time delays in both galaxy- and cluster-scale strong gravitational lenses have recently attracted a lot of attention in the context of the Hubble tension. Future wide-field cadenced surveys, such as the LSST, are anticipated to discover strong lenses across various scales. We generate mock catalogs of strongly lensed QSOs and SNe on galaxy-, group-, and cluster-scales based on a halo model that incorporates dark matter halos, galaxies, and subhalos. For the upcoming LSST survey, we predict that approximately 3500 lensed QSOs and 200 lensed SNe with resolved multiple images will be discovered. Among these, about 80 lensed QSOs and 10 lensed SNe will have maximum image separations larger than 10 arcsec, which roughly correspond to cluster-scale strong lensing. We find that adopting the Chabrier stellar IMF instead of the fiducial Salpeter IMF reduces the predicted number of strong lenses approximately by half, while the distributions of lens and source redshifts and image separations are not significantly changed. In addition to mock catalogs of multiple-image lens systems, we create mock catalogs of highly magnified systems, including both multiple-image and single-image systems. We find that such highly magnified systems are typically produced by massive galaxies, but non-negligible fraction of them are located in the outskirt of galaxy groups and clusters. Furthermore, we compare subsamples of our mock catalogs with lensed QSO samples constructed from the SDSS and Gaia to find that our mock catalogs with the fiducial Salpeter IMF reproduce the observation quite well. In contrast, our mock catalogs with the Chabrier IMF predict a significantly smaller number of lensed QSOs compared with observations, which adds evidence that the stellar IMF of massive galaxies is Salpeter-like. Our python code SL-Hammocks as well as the mock catalogs are made available online. (abridged), Comment: 25 pages, 20 figures, 3 tables, the code available at https://github.com/LSSTDESC/SL-Hammocks and the mock catalogs available at https://github.com/LSST-strong-lensing/data_public

Primordial black holes (PBH) can arise in a wide range of scenarios, from inflation to first-order phase transitions. Light PBHs, such as those produced during preheating or at the GUT scale, could induce an early matter-dominated phase given a moderate initial abundance. During the early matter-domination, the growth of initial PBH density perturbations can trigger collapse on horizon scales, producing much heavier PBHs. While the remaining original PBHs evaporate and reheat the Universe, these massive reformed PBHs survive for an extended period of time, producing potentially observable signatures at the present. We study this PBH reformation scenario and show that those reformed PBHs can emit significant quantities of gamma rays detectable by the next generation of experiments. The rapid reheating after matter domination generates a coincident stochastic gravitational wave background, which could be within range of the upcoming CMB-S4 experiment. The PBH reformation scenario provides an intriguing possibility of decoupling the current PBH population and the initial formation mechanism from early Universe physics, while providing opportunities for observation through multi-messenger astronomy., Comment: 8 pages, 5 figures

Uncertainty Quantification (UQ) is crucial for reliable image segmentation. Yet, while the field sees continual development of novel methods, a lack of agreed-upon benchmarks limits their systematic comparison and evaluation: Current UQ methods are typically tested either on overly simplistic toy datasets or on complex real-world datasets that do not allow to discern true uncertainty. To unify both controllability and complexity, we introduce Arctique, a procedurally generated dataset modeled after histopathological colon images. We chose histopathological images for two reasons: 1) their complexity in terms of intricate object structures and highly variable appearance, which yields challenging segmentation problems, and 2) their broad prevalence for medical diagnosis and respective relevance of high-quality UQ. To generate Arctique, we established a Blender-based framework for 3D scene creation with intrinsic noise manipulation. Arctique contains 50,000 rendered images with precise masks as well as noisy label simulations. We show that by independently controlling the uncertainty in both images and labels, we can effectively study the performance of several commonly used UQ methods. Hence, Arctique serves as a critical resource for benchmarking and advancing UQ techniques and other methodologies in complex, multi-object environments, bridging the gap between realism and controllability. All code is publicly available, allowing re-creation and controlled manipulations of our shipped images as well as creation and rendering of new scenes., Comment: 13 pages, 4 figures

We propose a compressive yet effective mesh representation, Blocked and Patchified Tokenization (BPT), facilitating the generation of meshes exceeding 8k faces. BPT compresses mesh sequences by employing block-wise indexing and patch aggregation, reducing their length by approximately 75\% compared to the original sequences. This compression milestone unlocks the potential to utilize mesh data with significantly more faces, thereby enhancing detail richness and improving generation robustness. Empowered with the BPT, we have built a foundation mesh generative model training on scaled mesh data to support flexible control for point clouds and images. Our model demonstrates the capability to generate meshes with intricate details and accurate topology, achieving SoTA performance on mesh generation and reaching the level for direct product usage., Comment: Homepage: https://whaohan.github.io/bpt , Code: https://github.com/whaohan/bpt

Despite the importance of shape perception in human vision, early neural image classifiers relied less on shape information for object recognition than other (often spurious) features. While recent research suggests that current large Vision-Language Models (VLMs) exhibit more reliance on shape, we find them to still be seriously limited in this regard. To quantify such limitations, we introduce IllusionBench, a dataset that challenges current cutting-edge VLMs to decipher shape information when the shape is represented by an arrangement of visual elements in a scene. Our extensive evaluations reveal that, while these shapes are easily detectable by human annotators, current VLMs struggle to recognize them, indicating important avenues for future work in developing more robust visual perception systems. The full dataset and codebase are available at: \url{https://arshiahemmat.github.io/illusionbench/}

Optical emission lines across the Small Magellanic Cloud (SMC) have been measured from multiple fields using the Australian National University (ANU) 2.3m telescope with the Wide-Field Spectrograph (WiFeS). Interpolated maps of the gas-phase metallicity, extinction, H$\alpha$ radial velocity and H$\alpha$ velocity dispersion have been made from these measurements. There is a metallicity gradient from the centre to the north of the galaxy of ~-0.095 dex/kpc with a shallower metallicity gradient from the centre to the south of the galaxy of ~-0.013 dex/kpc. There is an extinction gradient of ~-0.086 E(B-V)/kpc from the centre going north and shallower going from the centre to the south of ~-0.0089 E(B-V)/kpc. The SMC eastern arm has lower extinction than the main body. The radial velocity of the gas from the H$\alpha$ line and the HI line have been compared across the SMC. In general there is good agreement between the two measurements, though there are a few notable exceptions. Both show a region that has different radial velocity to the bulk motion of the SMC in the southern western corner by at least 16 kms$^{-1}$. The velocity dispersion from H$\alpha$ and HI across the SMC have also been compared, with the H$\alpha$ velocity dispersion usually the higher of the two. The eastern arm of the SMC generally has lower velocity dispersion than the SMC's main body. These measurements enable a detailed examination of the SMC, highlighting its nature as a disrupted satellite galaxy., Comment: Accepted for publication in MNRAS

We address the question of how to connect predictions by hydrodynamic models of how sea lice move in water to observable measures that count the number of lice on each fish in a cage in the water. This question is important for management and regulation of aquacultural practice that tries to maximise food production and minimise risk to the environment. We do this through a simple rule-based model of interaction between sea lice and caged fish. The model is simple: sea lice can attach and detach from a fish. The model has a novel feature, encoding what is known as a master equation producing a time-series of distributions of lice on fish that one might expect to find if a cage full of fish were placed at any given location. To demonstrate how this works, and to arrive at a rough estimate of the interaction rates, we fit a simplified version of the model with three free parameters to publicly available data about an experiment with sentinel cages in Loch Linnhe in Scotland. Our construction, coupled to the hydrodynamic models driven by surveillance data from industrial farms, quantifies the environmental impact as: what would the infection burden look like in a notional cage at any location and how does it change with time?

Over the past few years, a growing number of data platforms have emerged, including data commons, data repositories, and databases containing biomedical, environmental, social determinants of health and other data relevant to improving health outcomes. With the growing number of data platforms, interoperating multiple data platforms to form data meshes, data fabrics and other types of data ecosystems reduces data silos, expands data use, and increases the potential for new discoveries. In this paper, we introduce ten principles, which we call pillars, for data meshes. The goals of the principles are 1) to make it easier, faster, and more uniform to set up a data mesh from multiple data platforms; and, 2) to make it easier, faster, and more uniform, for a data platform to join one or more data meshes. The hope is that the greater availability of data through data meshes will accelerate research and that the greater uniformity of meshes will lower the cost of developing meshes and connecting a data platform to them., Comment: 10 pages, 1 figure

In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations., Comment: 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

We outline a practical scheme for converting between three commonly used sets of phases to describe the trajectories of extreme mass ratio inspirals; quasi-Keplerian angles, Mino time action-angles, and Boyer-Lindquist time action-angles (as utilised by the FastEMRIWaveform package). Conversion between Boyer-Lindquist time action angles and quasi-Keplerian angles is essential for the construction of a source frame for adiabatic inspirals that can be related to the source frames used by other gravitational wave source modelling techniques. While converting from quasi-Keplerian angles to Boyer-Lindquist time action angles via Mino time action-angles can be done analytically, the same does not hold for the converse, and so we make use of an efficient numerical root-finding method. We demonstrate the efficacy of our scheme by comparing two calculations for an eccentric and inclined geodesic orbit in Kerr spacetime using two different sets of orbital angles. We have made our implementations available in Mathematica, C, and Python., Comment: 17 pages, 5 figures. Comments welcome

This study explores the potential of off-the-shelf Vision-Language Models (VLMs) for high-level robot planning in the context of autonomous navigation. Indeed, while most of existing learning-based approaches for path planning require extensive task-specific training/fine-tuning, we demonstrate how such training can be avoided for most practical cases. To do this, we introduce Select2Plan (S2P), a novel training-free framework for high-level robot planning which completely eliminates the need for fine-tuning or specialised training. By leveraging structured Visual Question-Answering (VQA) and In-Context Learning (ICL), our approach drastically reduces the need for data collection, requiring a fraction of the task-specific data typically used by trained models, or even relying only on online data. Our method facilitates the effective use of a generally trained VLM in a flexible and cost-efficient way, and does not require additional sensing except for a simple monocular camera. We demonstrate its adaptability across various scene types, context sources, and sensing setups. We evaluate our approach in two distinct scenarios: traditional First-Person View (FPV) and infrastructure-driven Third-Person View (TPV) navigation, demonstrating the flexibility and simplicity of our method. Our technique significantly enhances the navigational capabilities of a baseline VLM of approximately 50% in TPV scenario, and is comparable to trained models in the FPV one, with as few as 20 demonstrations.

Research in vision and language has made considerable progress thanks to benchmarks such as COCO. COCO captions focused on unambiguous facts in English; ArtEmis introduced subjective emotions and ArtELingo introduced some multilinguality (Chinese and Arabic). However we believe there should be more multilinguality. Hence, we present ArtELingo-28, a vision-language benchmark that spans $\textbf{28}$ languages and encompasses approximately $\textbf{200,000}$ annotations ($\textbf{140}$ annotations per image). Traditionally, vision research focused on unambiguous class labels, whereas ArtELingo-28 emphasizes diversity of opinions over languages and cultures. The challenge is to build machine learning systems that assign emotional captions to images. Baseline results will be presented for three novel conditions: Zero-Shot, Few-Shot and One-vs-All Zero-Shot. We find that cross-lingual transfer is more successful for culturally-related languages. Data and code are provided at www.artelingo.org., Comment: 9 pages, Accepted at EMNLP 24, for more details see www.artelingo.org

Large language models (LLMs) have shown remarkable progress in encoding clinical knowledge and responding to complex medical queries with appropriate clinical reasoning. However, their applicability in subspecialist or complex medical settings remains underexplored. In this work, we probe the performance of AMIE, a research conversational diagnostic AI system, in the subspecialist domain of breast oncology care without specific fine-tuning to this challenging domain. To perform this evaluation, we curated a set of 50 synthetic breast cancer vignettes representing a range of treatment-naive and treatment-refractory cases and mirroring the key information available to a multidisciplinary tumor board for decision-making (openly released with this work). We developed a detailed clinical rubric for evaluating management plans, including axes such as the quality of case summarization, safety of the proposed care plan, and recommendations for chemotherapy, radiotherapy, surgery and hormonal therapy. To improve performance, we enhanced AMIE with the inference-time ability to perform web search retrieval to gather relevant and up-to-date clinical knowledge and refine its responses with a multi-stage self-critique pipeline. We compare response quality of AMIE with internal medicine trainees, oncology fellows, and general oncology attendings under both automated and specialist clinician evaluations. In our evaluations, AMIE outperformed trainees and fellows demonstrating the potential of the system in this challenging and important domain. We further demonstrate through qualitative examples, how systems such as AMIE might facilitate conversational interactions to assist clinicians in their decision making. However, AMIE's performance was overall inferior to attending oncologists suggesting that further research is needed prior to consideration of prospective uses.

We explore the applicability of the transcorrelated method to the elements in the second row of the periodic table. We use transcorrelated Hamiltonians in conjunction with full configuration interaction quantum Monte Carlo and coupled cluster techniques to obtain total energies and ionisation potentials, investigating their dependence on the nature and size of the basis sets used. Transcorrelation accelerates convergence to the complete basis set limit relative to conventional approaches, and chemically accurate results can generally be obtained with the cc-pVTZ basis, even with a frozen Ne core in the post-Hartree--Fock treatment., Comment: 11 pages, 5 figures

This document is comprised of a collection of updated preliminary parameters for the key parts of the muon collider. The updated preliminary parameters follow on from the October 2023 Tentative Parameters Report. Particular attention has been given to regions of the facility that are believed to hold greater technical uncertainty in their design and that have a strong impact on the cost and power consumption of the facility. The data is collected from a collaborative spreadsheet and transferred to overleaf.

Multimodal Retrieval Augmented Generation (mRAG) plays an important role in mitigating the "hallucination" issue inherent in multimodal large language models (MLLMs). Although promising, existing heuristic mRAGs typically predefined fixed retrieval processes, which causes two issues: (1) Non-adaptive Retrieval Queries. (2) Overloaded Retrieval Queries. However, these flaws cannot be adequately reflected by current knowledge-seeking visual question answering (VQA) datasets, since the most required knowledge can be readily obtained with a standard two-step retrieval. To bridge the dataset gap, we first construct Dyn-VQA dataset, consisting of three types of "dynamic" questions, which require complex knowledge retrieval strategies variable in query, tool, and time: (1) Questions with rapidly changing answers. (2) Questions requiring multi-modal knowledge. (3) Multi-hop questions. Experiments on Dyn-VQA reveal that existing heuristic mRAGs struggle to provide sufficient and precisely relevant knowledge for dynamic questions due to their rigid retrieval processes. Hence, we further propose the first self-adaptive planning agent for multimodal retrieval, OmniSearch. The underlying idea is to emulate the human behavior in question solution which dynamically decomposes complex multimodal questions into sub-question chains with retrieval action. Extensive experiments prove the effectiveness of our OmniSearch, also provide direction for advancing mRAG. The code and dataset will be open-sourced at https://github.com/Alibaba-NLP/OmniSearch.

Large language models (LLMs) pre-trained on massive corpora have demonstrated impressive few-shot learning capability on many NLP tasks. Recasting an NLP task into a text-to-text generation task is a common practice so that generative LLMs can be prompted to resolve it. However, performing document-level relation extraction (DocRE) tasks with generative LLM models is still challenging due to the structured output format of DocRE, which complicates the conversion to plain text. Limited information available in few-shot samples and prompt instructions induce further difficulties and challenges in relation extraction for mentioned entities in a document. In this paper, we represent the structured output as a graph-style triplet rather than natural language expressions and leverage generative LLMs for the DocRE task. Our approach, the Graph-DPEP framework is grounded in the reasoning behind triplet explanation thoughts presented in natural language. In this framework, we first introduce a ``decomposed-plug" method for performing the generation from LLMs over prompts with type-space decomposition to alleviate the burden of distinguishing all relation types. Second, we employ a verifier for calibrating the generation and identifying overlooked query entity pairs. Third, we develop "ensemble-play", reapplying generation on the entire type list by leveraging the reasoning thoughts embedded in a sub-graph associated with the missing query pair to address the missingness issue. Through extensive comparisons with existing prompt techniques and alternative Language Models (LLMs), our framework demonstrates superior performance on publicly available benchmarks in experiments.

Due to the significant time and effort required for handcrafting translations, most manga never leave the domestic Japanese market. Automatic manga translation is a promising potential solution. However, it is a budding and underdeveloped field and presents complexities even greater than those found in standard translation due to the need to effectively incorporate visual elements into the translation process to resolve ambiguities. In this work, we investigate to what extent multimodal large language models (LLMs) can provide effective manga translation, thereby assisting manga authors and publishers in reaching wider audiences. Specifically, we propose a methodology that leverages the vision component of multimodal LLMs to improve translation quality and evaluate the impact of translation unit size, context length, and propose a token efficient approach for manga translation. Moreover, we introduce a new evaluation dataset -- the first parallel Japanese-Polish manga translation dataset -- as part of a benchmark to be used in future research. Finally, we contribute an open-source software suite, enabling others to benchmark LLMs for manga translation. Our findings demonstrate that our proposed methods achieve state-of-the-art results for Japanese-English translation and set a new standard for Japanese-Polish., Comment: COLING 2025

Multilingual LLMs have achieved remarkable benchmark performance, but we find they continue to underperform on non-Latin script languages across contemporary LLM families. This discrepancy arises from the fact that LLMs are pretrained with orthographic scripts, which are dominated by Latin characters that obscure their shared phonology with non-Latin scripts. We propose leveraging phonemic transcriptions as complementary signals to induce script-invariant representations. Our study demonstrates that integrating phonemic signals improves performance across both non-Latin and Latin languages, with a particularly significant impact on closing the performance gap between the two. Through detailed experiments, we show that phonemic and orthographic scripts retrieve distinct examples for in-context learning (ICL). This motivates our proposed Mixed-ICL retrieval strategy, where further aggregation leads to our significant performance improvements for both Latin script languages (up to 12.6%) and non-Latin script languages (up to 15.1%) compared to randomized ICL retrieval.

Intense femtosecond x-ray pulses produced by an x-ray free-electron laser can trigger irreversible structural transitions in crystalline solids. For instance, irradiation of diamond can lead to graphitization and, at higher deposited doses, to amorphization. Our Monte Carlo simulations of irradiated diamond under realistic experimental conditions demonstrate that triggering graphitization or other phase transitions with hard x-ray photons can be challenging due to the ballistic escape of photoelectrons out of the beam focus. Decisive parameter here is the photoelectron range in proportion to the focal beam size. For future experiments on x-ray-induced transitions, such dedicated simulations of ballistic transport preceding the beamtime will be necessary. They can predict experimental conditions under which the desired distribution of the absorbed x-ray dose in the irradiated solid can be achieved., Comment: 15 pages, 2 figures, 1 table. Submitted to Phys. Rev. B

High harmonic generation is a resource of extremely broad frequency combs of ultrashort light pulses. The non-classical nature of this new quantum source has been recently evidenced in semiconductors by showing that high harmonic generation generates multimode squeezed states of light. Applications in quantum information science require the knowledge of the mode structure of the created states, defining how the quantum properties distribute over the spectral modes. To achieve that, an effective Schmidt decomposition of the reduced photonic state is performed on a tripartite harmonic set by simultaneously measuring the second- and third-order intensity correlation function. The Schmidt number is estimated which indicates an almost single-mode structure for each harmonic, a useful resource in quantum technology. By modelling our data with a displaced squeezed state, we retrieve the dependencies of the measured correlation as a function of the high harmonic driving laser intensity. The effective high-harmonic mode distribution is retrieved, and the strength of the contributing squeezing modes is estimated. Additionally, we demonstrate a significant violation of a Cauchy-Schwarz-type inequality for three biseparable partitions by multiple standard deviations. Our results confirm non-classicality of the high-harmonic generation process in semiconductors. The source operates at room temperature with compact lasers, and it could become a useful resource for future applications in quantum technologies., Comment: 25 pages, 9 figures

Intelligent agents designed for interactive environments face significant challenges in text-based games, a domain that demands complex reasoning and adaptability. While agents based on large language models (LLMs) using self-reflection have shown promise, they struggle when initially successful and exhibit reduced effectiveness when using smaller LLMs. We introduce Sweet&Sour, a novel approach that addresses these limitations in existing reflection methods by incorporating positive experiences and managed memory to enrich the context available to the agent at decision time. Our comprehensive analysis spans both closed- and open-source LLMs and demonstrates the effectiveness of Sweet&Sour in improving agent performance, particularly in scenarios where previous approaches fall short., Comment: To appear at NeurIPS 2024 Language Gamification workshop

A comprehensive framework of modeling covariance in angular streaking experiments is presented. Within the impulsive streaking regime, the displacement of electron momentum distribution (MD) provides a tight connection between the dressing-free MD and the dressed MD. Such connection establishes universal structures in the composition of streaking covariance that are common across different MDs, regardless of their exact shape. Building on this robust framework, we have developed methods for retrieving temporal information from angular streaking measurements. By providing a detailed understanding of the covariance structure in angular streaking experiments, our work enables more accurate and robust temporal measurements in a wide range of experimental scenarios., Comment: 22 pages, 13 figures

In molecular systems, the ultrafast motion of electrons initiates the process of chemical change. Tracking this electronic motion across molecules requires coupling attosecond time resolution to atomic-scale spatial sensitivity. In this work, we employ a pair of attosecond x-ray pulses from an x-ray free-electron laser to follow electron motion resulting from the sudden removal of an electron from a prototypical aromatic system, para-aminophenol. X-ray absorption enables tracking this motion with atomic-site specificity. Our measurements are compared with state-of-the-art computational modeling, reproducing the observed response across multiple timescales. Sub-femtosecond dynamics are assigned to states undergoing non-radiative decay, while few-femtosecond oscillatory motion is associated with electronic wavepacket motion in stable cation states, that will eventually couple to nuclear motion. Our work provides insight on the ultrafast charge motion preceding and initiating chemical transformations in moderately complex systems, and provides a powerful benchmark for computational models of ultrafast charge motion in matter.

Modern applications demand high performance and cost efficient database management systems (DBMSs). Their workloads may be diverse, ranging from online transaction processing to analytics and decision support. The cloud infrastructure enables disaggregation of monolithic DBMSs into components that facilitate software-hardware co-design. This is realized using pools of hardware resources, i.e., CPUs, GPUs, memory, FPGA, NVM, etc., connected using high-speed networks. This disaggregation trend is being adopted by cloud DBMSs because hardware re-provisioning can be achieved by simply invoking software APIs. Disaggregated DBMSs separate processing from storage, enabling each to scale elastically and independently. They may disaggregate compute usage based on functionality, e.g., compute needed for writes from compute needed for queries and compute needed for compaction. They may also use disaggregated memory, e.g., for intermediate results in a shuffle or for remote caching. The DBMS monitors the characteristics of a workload and dynamically assembles its components that are most efficient and cost effective for the workload. This paper is a summary of a panel session that discussed the capability, challenges, and opportunities of these emerging DBMSs and disaggregated hardware systems., Comment: This paper appeared in the {\em Performance Evaluation and Benchmarking} - 14th TPC Technology Conference, TPCTC 2022, Sydney, NSW, Australia, September 5, 2022, Revised Selected Papers. Lecture Notes in Computer Science 13860, Springer 2023, ISBN 978-3-031-29575-1

We introduce the notion of a move graph, that is, a directed graph whose vertex set is a $\mathbb Z$-module $\mathbb Z_n^m$, and whose arc set is uniquely determined by the action $M\!:\!\mathbb Z_n^m\to \mathbb Z_n^m$ where $M$ is an $m\times m$ matrix with integer entries. We study the manner in which properties of move graphs differ when one varies the choice of cyclic group $\mathbb Z_n$. Our principal focus is on a special family of such graphs, which we refer to as ``sub-add move graphs.''

In ill-posed imaging inverse problems, there can exist many hypotheses that fit both the observed measurements and prior knowledge of the true image. Rather than returning just one hypothesis of that image, posterior samplers aim to explore the full solution space by generating many probable hypotheses, which can later be used to quantify uncertainty or construct recoveries that appropriately navigate the perception/distortion trade-off. In this work, we propose a fast and accurate posterior-sampling conditional generative adversarial network (cGAN) that, through a novel form of regularization, aims for correctness in the posterior mean as well as the trace and K principal components of the posterior covariance matrix. Numerical experiments demonstrate that our method outperforms contemporary cGANs and diffusion models in imaging inverse problems like denoising, large-scale inpainting, and accelerated MRI recovery. The code for our model can be found here: https://github.com/matt-bendel/pcaGAN., Comment: To appear at NeurIPS 2024

Virtual Production (VP) technologies have continued to improve the flexibility of on-set filming and enhance the live concert experience. The core technology of VP relies on high-resolution, high-brightness LED panels to playback/render video content. There are a number of technical challenges to effective deployment e.g. image tile synchronisation across the panels, cross panel colour balancing and compensating for colour fluctuations due to changes in camera angles. Given the complexity and potential quality degradation, the industry prefers "pristine" or lossless compressed source material for displays, which requires significant storage and bandwidth. Modern lossy compression standards like AV1 or H.265 could maintain the same quality at significantly lower bitrates and resource demands. There is yet no agreed methodology for assessing the impact of these standards on quality when the VP scene is recorded in-camera. We present a methodology to assess this impact by comparing lossless and lossy compressed footage displayed through VP screens and recorded in-camera. We assess the quality impact of HAP/NotchLC/Daniel2 and AV1/HEVC/H.264 compression bitrates from 2 Mb/s to 2000 Mb/s with various GOP sizes. Several perceptual quality metrics are then used to automatically evaluate in-camera picture quality, referencing the original uncompressed source content through the LED wall. Our results show that we can achieve the same quality with hybrid codecs as with intermediate encoders at orders of magnitude less bitrate and storage requirements., Comment: SMPTE Media Summit Paper on use of Compression in Virtual Production from TCD and Disguise

We present an explicit construction of a relativistic quantum computing architecture using a variational quantum circuit approach that is shown to allow for universal quantum computing. The variational quantum circuit consists of tunable single-qubit rotations and entangling gates that are implemented successively. The single qubit rotations are parameterized by the proper time intervals of the qubits' trajectories and can be tuned by varying their relativistic motion in spacetime. The entangling layer is mediated by a relativistic quantum field instead of through direct coupling between the qubits. Within this setting, we give a prescription for how to use quantum field-mediated entanglement and manipulation of the relativistic motion of qubits to obtain a universal gate set, for which compact non-perturbative expressions that are valid for general spacetimes are also obtained. We also derive a lower bound on the channel fidelity that shows the existence of parameter regimes in which all entangling operations are effectively unitary, despite the noise generated from the presence of a mediating quantum field. Finally, we consider an explicit implementation of the quantum Fourier transform with relativistic qubits., Comment: 5 pages + appendices, 1 figure - revtex4-2

Electrochemical biosensors ("E-AB" or "E-DNA" type sensors) that utilize square-wave voltammetry originated in academic labs with a few standard experimental configurations for the electrochemical cell and data analysis. We report here on adaptations of these approaches that are friendly to novice scientists such as those in undergraduate laboratories. These approaches utilize commercially available components, low volumes, work over extended periods and enable facile analysis using a custom excel sheet.

Any algebraic connection on a vector bundle on a smooth complex algebraic curve determines an irregular class and in turn a fission tree at each puncture. The fission trees are the discrete data classifying the admissible deformation classes. Here we explain how to count the fission trees with given slope and number of leaves, in the untwisted case. This also leads to a clearer picture of the ``periodic table'' of the atoms that play the role of building blocks in 2d gauge theory., Comment: 23 pages, 12 figures, 4 tables

Despite the success of Instruction Tuning (IT) in training large language models (LLMs) to perform arbitrary user-specified tasks, these models often still leverage spurious or biased features learned from their training data, leading to undesired behaviours when deploying them in new contexts. In this work, we introduce Focus Instruction Tuning (FIT), which trains LLMs to condition their responses by focusing on specific features whilst ignoring others, leading to different behaviours based on what features are specified. Across several experimental settings, we show that focus-tuned models can be adaptively steered by focusing on different features at inference-time: for instance, robustness can be improved by focusing on task-causal features and ignoring spurious features, and social bias can be mitigated by ignoring demographic categories. Furthermore, FIT can steer behaviour in new contexts, generalising under distribution shift and to new unseen features at inference time, and thereby facilitating more robust, fair, and controllable LLM applications in real-world environments., Comment: 28pages, 14 figures

This report describes our systems submitted for the DCASE2024 Task 3 challenge: Audio and Audiovisual Sound Event Localization and Detection with Source Distance Estimation (Track B). Our main model is based on the audio-visual (AV) Conformer, which processes video and audio embeddings extracted with ResNet50 and with an audio encoder pre-trained on SELD, respectively. This model outperformed the audio-visual baseline of the development set of the STARSS23 dataset by a wide margin, halving its DOAE and improving the F1 by more than 3x. Our second system performs a temporal ensemble from the outputs of the AV-Conformer. We then extended the model with features for distance estimation, such as direct and reverberant signal components extracted from the omnidirectional audio channel, and depth maps extracted from the video frames. While the new system improved the RDE of our previous model by about 3 percentage points, it achieved a lower F1 score. This may be caused by sound classes that rarely appear in the training set and that the more complex system does not detect, as analysis can determine. To overcome this problem, our fourth and final system consists of an ensemble strategy combining the predictions of the other three. Many opportunities to refine the system and training strategy can be tested in future ablation experiments, and likely achieve incremental performance gains for this audio-visual task.

Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model (TabDAR) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order, hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18% to 45% on eight metrics across three distinct aspects.

We propose a novel dust battery mechanism for generating seed magnetic fields in the early universe, in which charged dust grains are radiatively accelerated, inducing strong electric currents that subsequently generate magnetic fields. Our analysis demonstrates that this process is effective even at very low metallicities (approximately $ \sim 10^{-5} Z_\odot$), and capable of producing seed fields with significant amplitudes of $B \sim \rm \mu G$ around luminous sources over timescales of years to Myr and across spatial scales ranging from au to kpc. Crucially, we find that this mechanism is generically $\sim10^8$ times more effective than the radiatively-driven electron battery or Biermann battery in relatively cool gas ($\ll 10^{5}\,$K), including both neutral and ionized gas. Furthermore, our results suggest that, to first order, dissipation effects do not appear to significantly impede this process, and that it can feasibly generate coherent seed fields on macroscopically large ISM scales (much larger than turbulent dissipation scales or electron mean-free-paths in the ISM). These seed fields could then be amplified by subsequent dynamo actions to the observed magnetic fields in galaxies. Additionally, we propose a sub-grid model for integration into cosmological simulations, and the required electric-field expressions for magnetohydrodynamic-particle-in-a-cell (MHD-PIC) simulations that explicitly model dust dynamics. Finally, we explore the broad applicability of this mechanism across different scales and conditions, emphasizing its robustness compared to other known battery mechanisms., Comment: 19 pages, 2 figures, 1 appendix. Submitted to ApJ

Sub-dimensional localization, also known as directional localization, arises when 2d electrons are subject to a periodic potential and incommensurate magnetic field which cause them to become exponentially localized along one crystal axis, while remaining extended in the orthogonal direction. We establish that sub-dimensional localization is robust to the presence of electronic interactions, which are known to generically destabilize single-particle localization. Consequently we predict that sub-dimensional localization may occur in moire materials, where it is manifest as a total absence of conductivity in the localized (transverse) direction, with finite conductivity in the extended (longitudinal) direction. We find the electrons form longitudinal charge density waves which are incommensurate with the moire potential, allowing them to slide and hence conduct in the longitudinal direction. The conductivity remains prohibited in the transverse direction due an emergent charge dipole moment conservation law., Comment: 7 pages

Synchrotron X-ray emission has been detected from nearly a dozen young supernova remnants (SNRs). X-rays of synchrotron origin exhibit linear polarization in a regular, non-randomly oriented magnetic field. The significant polarized X-ray emission from four such SNRs has already been reported on the basis of observations with the Imaging X-ray Polarimetry Explorer (IXPE). The magnetic-field structure as derived from IXPE observations is radial for Cassiopeia A, Tycho's SNR, and SN 1006, and tangential for RX J1713.7-3946. The latter together with the recent detection of a tangential magnetic field in SNR 1E 0102.2-7219 by the Australia Telescope Compact Array in the radio band shows that tangential magnetic fields can also be present in young SNRs. Thus, the dichotomy in polarization between young and middle-aged SNRs (radial magnetic fields in young SNRs, but tangential magnetic fields in middle-aged SNRs), previously noticed in the radio band, deserves additional attention. The present analysis of IXPE observations determines, for the first time, a magnetic-field structure in the northwestern rim of Vela Jr, also known as RX J0852.0-4622, and provides a new example of a young SNR with a tangential magnetic field., Comment: Accepted for publication in Astronomy and Astrophysics

We consider a susceptible-infected-susceptible (SIS) epidemic model in which a large group of individuals decide whether to adopt partially effective protection without being aware of their individual infection status. Each individual receives a signal which conveys noisy information about its infection state, and then decides its action to maximize its expected utility computed using its posterior probability of being infected conditioned on the received signal. We first derive the static signal which minimizes the infection level at the stationary Nash equilibrium under suitable assumptions. We then formulate an optimal control problem to determine the optimal dynamic signal that minimizes the aggregate infection level along the solution trajectory. We compare the performance of the dynamic signaling scheme with the optimal static signaling scheme, and illustrate the advantage of the former through numerical simulations.

One of the most challenging forms of misinformation involves the out-of-context (OOC) use of images paired with misleading text, creating false narratives. Existing AI-driven detection systems lack explainability and require expensive fine-tuning. We address these issues with MAD-Sherlock: a Multi-Agent Debate system for OOC Misinformation Detection. MAD-Sherlock introduces a novel multi-agent debate framework where multimodal agents collaborate to assess contextual consistency and request external information to enhance cross-context reasoning and decision-making. Our framework enables explainable detection with state-of-the-art accuracy even without domain-specific fine-tuning. Extensive ablation studies confirm that external retrieval significantly improves detection accuracy, and user studies demonstrate that MAD-Sherlock boosts performance for both experts and non-experts. These results position MAD-Sherlock as a powerful tool for autonomous and citizen intelligence applications.

X-ray polarization is a unique new probe of the particle acceleration in astrophysical jets made possible through the Imaging X-ray Polarimetry Explorer. Here we report on the first dense X-ray polarization monitoring campaign on the blazar Mrk 421. Our observations were accompanied by an even denser radio and optical polarization campaign. We find significant short-timescale variability in both X-ray polarization degree and angle, including a $\sim90^\circ$ angle rotation about the jet axis. We attribute this to random variations of the magnetic field, consistent with the presence of turbulence but also unlikely to be explained by turbulence alone. At the same time, the degree of lower-energy polarization is significantly lower and shows no more than mild variability. Our campaign provides further evidence for a scenario in which energy-stratified shock-acceleration of relativistic electrons, combined with a turbulent magnetic field, is responsible for optical to X-ray synchrotron emission in blazar jets., Comment: 23 pages, including 8 pages of appendices. 12 figures, 3 tables. Submitted to ApJ

GPT-4o is an autoregressive omni model that accepts as input any combination of text, audio, image, and video, and generates any combination of text, audio, and image outputs. It's trained end-to-end across text, vision, and audio, meaning all inputs and outputs are processed by the same neural network. GPT-4o can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds, which is similar to human response time in conversation. It matches GPT-4 Turbo performance on text in English and code, with significant improvement on text in non-English languages, while also being much faster and 50\% cheaper in the API. GPT-4o is especially better at vision and audio understanding compared to existing models. In line with our commitment to building AI safely and consistent with our voluntary commitments to the White House, we are sharing the GPT-4o System Card, which includes our Preparedness Framework evaluations. In this System Card, we provide a detailed look at GPT-4o's capabilities, limitations, and safety evaluations across multiple categories, focusing on speech-to-speech while also evaluating text and image capabilities, and measures we've implemented to ensure the model is safe and aligned. We also include third-party assessments on dangerous capabilities, as well as discussion of potential societal impacts of GPT-4o's text and vision capabilities.

The design and control of atomic-scale spin structures constitute major challenges for spin-based quantum technology platforms, including quantum dots, color centers, and molecular spins. Here, we showcase a strategy for designing the quantum properties of molecular spin qubits by combining tip-assisted on-surface assembly with electron spin resonance scanning tunneling microscopy (ESR-STM): We fabricate magnetic dimer complexes that consist of an iron phthalocyanine (FePc) molecule and an organometallic half-sandwich complex formed by the FePc ligand and an attached iron atom, Fe(C6H6). The total complex forms a mixed-spin (1/2,1) quantum ferrimagnet with a well-separated correlated ground state doublet, which we utilize for coherent control. As a result of the correlation, the quantum ferrimagnet shows an improved spin lifetime (>1.5 ${\mu}$s) as it is partially protected against inelastic electron scattering. Lastly, the ferrimagnet units also enable intermolecular coupling, that can be used to realize both ferromagnetic or antiferromagnetic structures. Thus, quantum ferrimagnets provide a versatile platform to improve coherent control in general and to study complex magnetic interactions., Comment: 17 pages, 4 figures

We study the identification and estimation of heterogeneous, intertemporal treatment effects (TE) when potential outcomes depend on past treatments. First, applying a dynamic panel data model to observed outcomes, we show that instrument-based GMM estimators, such as Arellano and Bond (1991), converge to a non-convex (negatively weighted) aggregate of TE plus non-vanishing trends. We then provide restrictions on sequential exchangeability (SE) of treatment and TE heterogeneity that reduce the GMM estimand to a convex (positively weighted) aggregate of TE. Second, we introduce an adjusted inverse-propensity-weighted (IPW) estimator for a new notion of average treatment effect (ATE) over past observed treatments. Third, we show that when potential outcomes are generated by dynamic panel data models with homogeneous TE, such GMM estimators converge to causal parameters (even when SE is generically violated without conditioning on individual fixed effects). Finally, we motivate SE and compare it with parallel trends (PT) in various settings with observational data (when treatments are dynamic, rational choices under learning) or experimental data (when treatments are sequentially randomized).

Large Language Models (LLMs) often hallucinate, producing unfaithful or factually incorrect outputs by misrepresenting the provided context or incorrectly recalling internal knowledge. Recent studies have identified specific attention heads within the Transformer architecture, known as retrieval heads, responsible for extracting relevant contextual information. We hypothesise that masking these retrieval heads can induce hallucinations and that contrasting the outputs of the base LLM and the masked LLM can reduce hallucinations. To this end, we propose Decoding by Contrasting Retrieval Heads (DeCoRe), a novel training-free decoding strategy that amplifies information found in the context and model parameters. DeCoRe mitigates potentially hallucinated responses by dynamically contrasting the outputs of the base LLM and the masked LLM, using conditional entropy as a guide. Our extensive experiments confirm that DeCoRe significantly improves performance on tasks requiring high contextual faithfulness, such as summarisation (XSum by 18.6%), instruction following (MemoTrap by 10.9%), and open-book question answering (NQ-Open by 2.4% and NQ-Swap by 5.5%).

Graph neural networks (GNNs) have become the dominant solution for learning on graphs, the typical non-Euclidean structures. Conventional GNNs, constructed with the Artificial Neuron Network (ANN), have achieved impressive performance at the cost of high computation and energy consumption. In parallel, spiking GNNs with brain-like spiking neurons are drawing increasing research attention owing to the energy efficiency. So far, existing spiking GNNs consider graphs in Euclidean space, ignoring the structural geometry, and suffer from the high latency issue due to Back-Propagation-Through-Time (BPTT) with the surrogate gradient. In light of the aforementioned issues, we are devoted to exploring spiking GNN on Riemannian manifolds, and present a Manifold-valued Spiking GNN (MSG). In particular, we design a new spiking neuron on geodesically complete manifolds with the diffeomorphism, so that BPTT regarding the spikes is replaced by the proposed differentiation via manifold. Theoretically, we show that MSG approximates a solver of the manifold ordinary differential equation. Extensive experiments on common graphs show the proposed MSG achieves superior performance to previous spiking GNNs and energy efficiency to conventional GNNs., Comment: Accepted by NeurIPS 2024, 30 pages

Real-world applications of reinforcement learning often involve environments where agents operate on complex, high-dimensional observations, but the underlying (''latent'') dynamics are comparatively simple. However, outside of restrictive settings such as small latent spaces, the fundamental statistical requirements and algorithmic principles for reinforcement learning under latent dynamics are poorly understood. This paper addresses the question of reinforcement learning under $\textit{general}$ latent dynamics from a statistical and algorithmic perspective. On the statistical side, our main negative result shows that most well-studied settings for reinforcement learning with function approximation become intractable when composed with rich observations; we complement this with a positive result, identifying latent pushforward coverability as a general condition that enables statistical tractability. Algorithmically, we develop provably efficient observable-to-latent reductions -- that is, reductions that transform an arbitrary algorithm for the latent MDP into an algorithm that can operate on rich observations -- in two settings: one where the agent has access to hindsight observations of the latent dynamics [LADZ23], and one where the agent can estimate self-predictive latent models [SAGHCB20]. Together, our results serve as a first step toward a unified statistical and algorithmic theory for reinforcement learning under latent dynamics.

Water at negative pressures can remain in a metastable state for a surprisingly long time before it reaches equilibrium by cavitation, i.e. by the formation of vapor bubbles. The wide spread of experimentally measured cavitation pressures depending on water purity, surface contact angle and surface quality implicates the relevance of water cavitation in bulk, at surfaces and at surface defects for different systems. We formulate a kinetic model that includes all three different cavitation pathways and determine the needed nucleation attempt frequencies in bulk, at surfaces and at defects from atomistic molecular dynamics simulations. Our model reveals that cavitation occurs in pure bulk water only for defect-free hydrophilic surfaces with wetting contact angles below 50{\deg} to 60{\deg} and at pressures of the order of $-$100 MPa, depending only slightly on system size and observation time. Cavitation on defect-free surfaces occurs only for higher contact angles, with the typical cavitation pressure rising to about $-$30 MPa for very hydrophobic surfaces. Nanoscopic hydrophobic surface defects act as very efficient cavitation nuclei and can dominate the cavitation kinetics in a macroscopic system. In fact, a nanoscopic defect that hosts a pre-existing vapor bubble can raise the critical cavitation pressure much further. Our results explain the wide variation of experimentally observed cavitation pressures in synthetic and biological systems and highlight the importance of surface and defect mechanisms for the nucleation of metastable systems.

In this work, we present conception and study of gallium nitride (GaN) nanostructures on a gallium arsenide (GaAs) substrate with (111)A orientation. The nanostructures were designed by GaN droplet epitaxy and studied in-situ by X-ray photoelectron spectroscopy and ex-situ by atomic force microscopy, scanning electron microscopy and transmission electron microscopy. These studies were coupled with kinetic Monte Carlo simulations to precisely understand the phenomena occurring during the nitridation and to find the optimum conditions for complete nitridation of gallium droplets. The HRTEM observation showed a cubic (zinc blende) crystal structure of the GaN nanodots for a nitridation at 300{\deg}C. Ramping the temperature from 100{\deg}C to 350{\deg}C during droplet nitridation enabled to obtain a very high density (>1011cm-2) of GaN nanodots with the zinc blende crystallinity.

Model-free reinforcement learning (RL) is inherently a reactive method, operating under the assumption that it starts with no prior knowledge of the system and entirely depends on trial-and-error for learning. This approach faces several challenges, such as poor sample efficiency, generalization, and the need for well-designed reward functions to guide learning effectively. On the other hand, controllers based on complete system dynamics do not require data. This paper addresses the intermediate situation where there is not enough model information for complete controller design, but there is enough to suggest that a model-free approach is not the best approach either. By carefully decoupling known and unknown information about the system dynamics, we obtain an embedded controller guided by our partial model and thus improve the learning efficiency of an RL-enhanced approach. A modular design allows us to deploy mainstream RL algorithms to refine the policy. Simulation results show that our method significantly improves sample efficiency compared with standard RL methods on continuous control tasks, and also offers enhanced performance over traditional control approaches. Experiments on a real ground vehicle also validate the performance of our method, including generalization and robustness., Comment: Accepted to IROS 2024