Search

Neural networks often favor shortcut heuristics based on surface-level patterns. As one example, language models (LMs) behave like n-gram models early in training. However, to correctly apply grammatical rules, LMs must rely on hierarchical syntactic representations instead of n-grams. In this work, we use cases studies of English grammar to explore how latent structure in training data drives models toward improved out-of-distribution (OOD) generalization.We then investigate how data composition can lead to inconsistent OOD behavior across random seeds and to unstable training dynamics. Our results show that models stabilize in their OOD behavior only when they fully commit to either a surface-level linear rule or a hierarchical rule. The hierarchical rule, furthermore, is induced by grammatically complex sequences with deep embedding structures, whereas the linear rule is induced by simpler sequences. When the data contains a mix of simple and complex examples, potential rules compete; each independent training run either stabilizes by committing to a single rule or remains unstable in its OOD behavior. These conditions lead `stable seeds' to cluster around simple rules, forming bimodal performance distributions across seeds. We also identify an exception to the relationship between stability and generalization: models which memorize patterns from low-diversity training data can overfit stably, with different rules for memorized and unmemorized patterns. Our findings emphasize the critical role of training data in shaping generalization patterns and how competition between data subsets contributes to inconsistent generalization outcomes across random seeds. Code is available at https://github.com/sunnytqin/concept_comp.git.

Modern large language models use a fixed tokenizer to effectively compress text drawn from a source domain. However, applying the same tokenizer to a new target domain often leads to inferior compression, more costly inference, and reduced semantic alignment. To address this deficiency, we introduce Sparse Sinkhorn Token Translation (S2T2). S2T2 trains a tailored tokenizer for the target domain and learns to translate between target and source tokens, enabling more effective reuse of the pre-trained next-source-token predictor. In our experiments with finetuned English language models, S2T2 improves both the perplexity and the compression of out-of-domain protein sequences, outperforming direct finetuning with either the source or target tokenizer. In addition, we find that token translations learned for smaller, less expensive models can be directly transferred to larger, more powerful models to reap the benefits of S2T2 at lower cost.

Empirical auditing has emerged as a means of catching some of the flaws in the implementation of privacy-preserving algorithms. Existing auditing mechanisms, however, are either computationally inefficient requiring multiple runs of the machine learning algorithms or suboptimal in calculating an empirical privacy. In this work, we present a tight and efficient auditing procedure and analysis that can effectively assess the privacy of mechanisms. Our approach is efficient; similar to the recent work of Steinke, Nasr, and Jagielski (2023), our auditing procedure leverages the randomness of examples in the input dataset and requires only a single run of the target mechanism. And it is more accurate; we provide a novel analysis that enables us to achieve tight empirical privacy estimates by using the hypothesized $f$-DP curve of the mechanism, which provides a more accurate measure of privacy than the traditional $\epsilon,\delta$ differential privacy parameters. We use our auditing procure and analysis to obtain empirical privacy, demonstrating that our auditing procedure delivers tighter privacy estimates.

The Mixture-of-Experts (MoE) architecture enables a significant increase in the total number of model parameters with minimal computational overhead. However, it is not clear what performance tradeoffs, if any, exist between MoEs and standard dense transformers. In this paper, we show that as we increase the number of experts (while fixing the number of active parameters), the memorization performance consistently increases while the reasoning capabilities saturate. We begin by analyzing the theoretical limitations of MoEs at reasoning. We prove that there exist graph problems that cannot be solved by any number of experts of a certain width; however, the same task can be easily solved by a dense model with a slightly larger width. On the other hand, we find that on memory-intensive tasks, MoEs can effectively leverage a small number of active parameters with a large number of experts to memorize the data. We empirically validate these findings on synthetic graph problems and memory-intensive closed book retrieval tasks. Lastly, we pre-train a series of MoEs and dense transformers and evaluate them on commonly used benchmarks in math and natural language. We find that increasing the number of experts helps solve knowledge-intensive tasks, but fails to yield the same benefits for reasoning tasks.

The quest for radio signals from technologically-advanced extraterrestrial intelligence has traditionally concentrated on the vicinity of 1.4 GHz. In this paper, we extend the search to unprecedented territories, detailing our extensive observations at 6 GHz and initiating the first-ever survey at 18 GHz with the Sardinia Radio Telescope (SRT). Our strategy entailed rigorous observation sessions, totaling 36 hours, directed towards the Galactic Center and 72 selected TESS targets-making this the most comprehensive high-frequency technosignature search to date. Our narrowband signal search found no definitive evidence of drifting signals that could suggest an extraterrestrial origin from the surveyed regions. Nevertheless, our efforts have enabled us to set new constraints on the presence of radio emissions from approximately $5\times 10^{5}$ stars, establishing an isotropic radiated power limit of $1.8\times 10^{19} W$. We also provide a comparative analysis of the 'hits' recorded across both frequencies to highlight the significance of pursuing technosignature searches at higher frequencies, where the spectral landscape is less congested and more conducive to detection., Comment: Submitted to Acta Astronautica

Bayesian optimization (BO) is a powerful framework to optimize black-box expensive-to-evaluate functions via sequential interactions. In several important problems (e.g. drug discovery, circuit design, neural architecture search, etc.), though, such functions are defined over large $\textit{combinatorial and unstructured}$ spaces. This makes existing BO algorithms not feasible due to the intractable maximization of the acquisition function over these domains. To address this issue, we propose $\textbf{GameOpt}$, a novel game-theoretical approach to combinatorial BO. $\textbf{GameOpt}$ establishes a cooperative game between the different optimization variables, and selects points that are game $\textit{equilibria}$ of an upper confidence bound acquisition function. These are stable configurations from which no variable has an incentive to deviate$-$ analog to local optima in continuous domains. Crucially, this allows us to efficiently break down the complexity of the combinatorial domain into individual decision sets, making $\textbf{GameOpt}$ scalable to large combinatorial spaces. We demonstrate the application of $\textbf{GameOpt}$ to the challenging $\textit{protein design}$ problem and validate its performance on four real-world protein datasets. Each protein can take up to $20^{X}$ possible configurations, where $X$ is the length of a protein, making standard BO methods infeasible. Instead, our approach iteratively selects informative protein configurations and very quickly discovers highly active protein variants compared to other baselines.

Point cloud completion aims to recover the complete 3D shape of an object from partial observations. While approaches relying on synthetic shape priors achieved promising results in this domain, their applicability and generalizability to real-world data are still limited. To tackle this problem, we propose a self-supervised framework, namely RealDiff, that formulates point cloud completion as a conditional generation problem directly on real-world measurements. To better deal with noisy observations without resorting to training on synthetic data, we leverage additional geometric cues. Specifically, RealDiff simulates a diffusion process at the missing object parts while conditioning the generation on the partial input to address the multimodal nature of the task. We further regularize the training by matching object silhouettes and depth maps, predicted by our method, with the externally estimated ones. Experimental results show that our method consistently outperforms state-of-the-art methods in real-world point cloud completion.

Comparing datasets is a fundamental task in machine learning, essential for various learning paradigms; from evaluating train and test datasets for model generalization to using dataset similarity for detecting data drift. While traditional notions of dataset distances offer principled measures of similarity, their utility has largely been assessed through prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, where predictions serve as inputs for downstream optimization tasks, model performance is measured through decision regret minimization rather than prediction error minimization. In this work, we (i) show that traditional dataset distances, which rely solely on feature and label dimensions, lack informativeness in the PtO context, and (ii) propose a new dataset distance that incorporates the impacts of downstream decisions. Our results show that this decision-aware dataset distance effectively captures adaptation success in PtO contexts, providing a PtO adaptation bound in terms of dataset distance. Empirically, we show that our proposed distance measure accurately predicts transferability across three different PtO tasks from the literature.

The 408 MHz Haslam map is widely used as a low-frequency anchor for the intensity and morphology of Galactic synchrotron emission. Multi-frequency, multi-experiment fits show evidence of spatial variation and curvature in the synchrotron frequency spectrum, but there are also poorly-understood gain factors between experiments. We perform a Bayesian model comparison across a range of scenarios, using fits that include recent spectroscopic observations at $\sim 1$~GHz by MeerKAT. A large uncorrected gain factor of about 60\% in the Haslam data is strongly preferred, partly undermining its use as a reference template.

Weight-ensembles are formed when the parameters of multiple neural networks are directly averaged into a single model. They have demonstrated generalization capability in-distribution (ID) and out-of-distribution (OOD) which is not completely understood, though they are thought to successfully exploit functional diversity allotted by each distinct model. Given a collection of models, it is also unclear which combination leads to the optimal weight-ensemble; the SOTA is a linear-time ``greedy" method. We introduce two novel weight-ensembling approaches to study the link between performance dynamics and the nature of how each method decides to use apply the functionally diverse components, akin to diversity-encouragement in the prediction-ensemble literature. We develop a visualization tool to explain how each algorithm explores various domains defined via pairwise-distances to further investigate selection and algorithms' convergence. Empirical analyses shed perspectives which reinforce how high-diversity enhances weight-ensembling while qualifying the extent to which diversity alone improves accuracy. We also demonstrate that sampling positionally distinct models can contribute just as meaningfully to improvements in a weight-ensemble., Comment: Published at the ICML 2024 (Vienna, Austria) Workshop on Foundation Models in the Wild

Siril is a powerful open-source software package designed for the preprocessing and post-processing of astronomical images. It is particularly well-suited for astrophotography enthusiasts and professional astronomers alike. Siril provides advanced tools for tasks such as image stacking, calibration, registration, and enhancement, enabling users to produce high-quality images of celestial objects. The version discussed here is the development branch 1.4, which, while not officially released, is available for testing and includes the latest features and improvements., Comment: 4 pages, 1 figure: short article in the form requested by the JOSS guideline

AI-generated synthetic media, also called Deepfakes, have significantly influenced so many domains, from entertainment to cybersecurity. Generative Adversarial Networks (GANs) and Diffusion Models (DMs) are the main frameworks used to create Deepfakes, producing highly realistic yet fabricated content. While these technologies open up new creative possibilities, they also bring substantial ethical and security risks due to their potential misuse. The rise of such advanced media has led to the development of a cognitive bias known as Impostor Bias, where individuals doubt the authenticity of multimedia due to the awareness of AI's capabilities. As a result, Deepfake detection has become a vital area of research, focusing on identifying subtle inconsistencies and artifacts with machine learning techniques, especially Convolutional Neural Networks (CNNs). Research in forensic Deepfake technology encompasses five main areas: detection, attribution and recognition, passive authentication, detection in realistic scenarios, and active authentication. This paper reviews the primary algorithms that address these challenges, examining their advantages, limitations, and future prospects.

We present results from MeerKAT single-dish HI intensity maps, the final observations to be performed in L-band in the MeerKAT Large Area Synoptic Survey (MeerKLASS) campaign. The observations represent the deepest single-dish HI intensity maps to date, produced from 41 repeated scans over $236\,{\rm deg}^2$, providing 62 hours of observational data for each of the 64 dishes before flagging. By introducing an iterative self-calibration process, the estimated thermal noise of the reconstructed maps is limited to ${\sim}\,1.21\,$mK ($1.2\,\times$ the theoretical noise level). This thermal noise will be sub-dominant relative to the HI fluctuations on large scales ($k\,{\lesssim}\,0.15\,h\,\text{Mpc}^{-1}$), which demands upgrades to power spectrum analysis techniques, particularly for covariance estimation. In this work, we present the improved MeerKLASS analysis pipeline, validating it on both a suite of mock simulations and a small sample of overlapping spectroscopic galaxies from the Galaxy And Mass Assembly (GAMA) survey. Despite only overlapping with ${\sim}\,25\%$ of the MeerKLASS deep field, and a conservative approach to covariance estimation, we still obtain a ${>}\,4\,\sigma$ detection of the cross-power spectrum between the intensity maps and the 2269 galaxies at the narrow redshift range $0.39\,{<}\,z\,{<}\,0.46$. We briefly discuss the HI auto-power spectrum from this data, the detection of which will be the focus of follow-up work. For the first time with MeerKAT single-dish intensity maps, we also present evidence of HI emission from stacking the maps onto the positions of the GAMA galaxies., Comment: 29 pages, 27 figures. See Fig.14 and 15 for auto-HI and HI-galaxy cross power spectra. Fig 19 and 20 for stacking detections. Submitted to MNRAS

Designing high-quality indoor 3D scenes is important in many practical applications, such as room planning or game development. Conventionally, this has been a time-consuming process which requires both artistic skill and familiarity with professional software, making it hardly accessible for layman users. However, recent advances in generative AI have established solid foundation for democratizing 3D design. In this paper, we propose a pioneering approach for text-based 3D room design. Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. With an additional text prompt the users can change the appearance of the entire scene or of individual objects in it. Built using in-context learning, CAD model retrieval and 3D-Gaussian-Splatting-based stylization, our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices. Our project page is available at https://sceneteller.github.io/., Comment: ECCV'24 camera-ready version

MoS2 is one of the most investigated and promising transition-metal dichalcogenides. Its popularity stems from the interesting properties of the monolayer phase, which can serve as the fundamental block for numerous applications. In this paper, we propose an atomistic perspective on the modulation of thermal transport properties in monolayer MoS2 through strategic defect engineering, specifically the introduction of sulfur vacancies. Using a combination of molecular dynamics simulations and lattice dynamics calculations, we show how various distributions of sulfur vacancies -- ranging from random to periodically arranged configurations -- affect its thermal conductivity. Notably, we observe that certain periodic arrangements restore the thermal conductivity of the pristine system, due to a minimized interaction between acoustic and optical phonons facilitated by the imposed superperiodicity. This research deepens the understanding of phononic heat transport in two-dimensional materials and introduces a different point-of-view for phonon engineering in nanoscale devices, offering a pathway to enhance device performance and longevity through tailored thermal management strategies.

Optimal Transport (OT) has recently emerged as a powerful framework for learning minimal-displacement maps between distributions. The predominant approach involves a neural parametrization of the Monge formulation of OT, typically assuming the same space for both distributions. However, the setting across ``incomparable spaces'' (e.g., of different dimensionality), corresponding to the Gromov- Wasserstein distance, remains underexplored, with existing methods often imposing restrictive assumptions on the cost function. In this paper, we present a novel neural formulation of the Gromov-Monge (GM) problem rooted in one of its fundamental properties: invariance to strong isomorphisms. We operationalize this property by decomposing the learnable OT map into two components: (i) an approximate strong isomorphism between the source distribution and an intermediate reference distribution, and (ii) a GM-optimal map between this reference and the target distribution. Our formulation leverages and extends the Monge gap regularizer of Uscidda & Cuturi (2023) to eliminate the need for complex architectural requirements of other neural OT methods, yielding a simple but practical method that enjoys favorable theoretical guarantees. Our preliminary empirical results show that our framework provides a promising approach to learn OT maps across diverse spaces., Comment: ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling

Everlasting contact between language communities leads to constant changes in languages over time, and gives rise to language varieties and dialects. However, the communities speaking non-standard language are often overlooked by non-inclusive NLP technologies. Recently, there has been a surge of interest in studying diatopic and diachronic changes in dialect NLP, but there is currently no research exploring the intersection of both. Our work aims to fill this gap by systematically reviewing diachronic and diatopic papers from a unified perspective. In this work, we critically assess nine tasks and datasets across five dialects from three language families (Slavic, Romance, and Germanic) in both spoken and written modalities. The tasks covered are diverse, including corpus construction, dialect distance estimation, and dialect geolocation prediction, among others. Moreover, we outline five open challenges regarding changes in dialect use over time, the reliability of dialect datasets, the importance of speaker characteristics, limited coverage of dialects, and ethical considerations in data collection. We hope that our work sheds light on future research towards inclusive computational methods and datasets for language varieties and dialects., Comment: LChange24 Camera Ready

We study the linear response of four dimensional magnetized black holes and regular topological stars arising from dimensional compactification of Einstein-Maxwell theory in five dimensions. We consider both radial and nonradial perturbations and study the stability of these solutions, both in the frequency and in the time domain. Due to the presence of magnetic fluxes in the background, axial (i.e., odd-parity) gravitational perturbations are coupled to polar (i.e., even-parity) electromagnetic perturbations (Type-I sector) whereas polar gravitational and scalar perturbations are coupled to axial electromagnetic ones (Type-II sector). We provide a comprehensive analytical and numerical study of the radial perturbations and of the Type I sector, finding no evidence of linear instabilities (besides the already known Gregory-Laflamme instability of black strings occurring only in a certain range of the parameters), even despite the fact that the effective potential for radial perturbations of topological stars is negative and divergent near the inner boundary. Ultracompact topological stars exhibit long-lived trapped modes that give rise to echoes in the time-domain response. Their prompt response is very similar to that of the corresponding black hole with comparable charge-to-mass ratio. This provides a concrete realization of ultracompact objects arising from a well-defined theory. The numerical analysis of the Type-II sector will appear in a companion paper., Comment: 23 pages, 11 figures. Comments and clarifications added, to be published in PRD

Large language models (LLMs) have shown remarkable potential for problem solving, with open source models achieving increasingly impressive performance on benchmarks measuring areas from logical reasoning to mathematical ability. Ensembling models can further improve capabilities across a variety of domains. However, conventional methods of combining models at inference time such as shallow fusion necessitate a shared vocabulary and tokenization, and alternatives like fine-tuning for domain-specific performance are both time consuming and computationally expensive. We therefore present an inference-time ensembling algorithm aimed at "averaging" outputs from multiple LLMs and illustrate its improved performance across multiple domains compared to its constituent models alone. Character-wise ensemble decoding, CharED, finds the marginal distribution of each character for an individual model and performs a weighted average to generate an output, character by character. In coding, math, and toxicity benchmarks, we find our proposed model able to combine complimentary strengths of multiple LLMs, regardless of vocabulary, tokenization, or model size., Comment: 9 pages, 4 figures

The potential of transformer-based LLMs risks being hindered by privacy concerns due to their reliance on extensive datasets, possibly including sensitive information. Regulatory measures like GDPR and CCPA call for using robust auditing tools to address potential privacy issues, with Membership Inference Attacks (MIA) being the primary method for assessing LLMs' privacy risks. Differently from traditional MIA approaches, often requiring computationally intensive training of additional models, this paper introduces an efficient methodology that generates \textit{noisy neighbors} for a target sample by adding stochastic noise in the embedding space, requiring operating the target model in inference mode only. Our findings demonstrate that this approach closely matches the effectiveness of employing shadow models, showing its usability in practical privacy auditing scenarios.

This second paper presents an in-depth analysis of the composition of the planetary material that has been accreted onto seven white dwarfs with circumstellar dust and gas emission discs with abundances reported in Paper I. The white dwarfs are accreting planetary bodies with a wide range of oxygen, carbon, and sulfur volatile contents, including one white dwarf that shows the most enhanced sulfur abundance seen to date. Three white dwarfs show tentative evidence (2-3$\sigma$) of accreting oxygen-rich material, potentially from water-rich bodies, whilst two others are accreting dry, rocky material. One white dwarf is accreting a mantle-rich fragment of a larger differentiated body, whilst two white dwarfs show an enhancement in their iron abundance and could be accreting core-rich fragments. Whilst most planetary material accreted by white dwarfs display chondritic or bulk Earth-like compositions, these observations demonstrate that core-mantle differentiation, disruptive collisions, and the accretion of core-mantle differentiated material are important. Less than one percent of polluted white dwarfs host both observable circumstellar gas and dust. It is unknown whether these systems are experiencing an early phase in the disruption and accretion of planetary bodies, or alternatively if they are accreting larger planetary bodies. From this work there is no substantial evidence for significant differences in the accreted refractory abundance ratios for those white dwarfs with or without circumstellar gas, but there is tentative evidence for those with circumstellar gas discs to be accreting more water rich material which may suggest that volatiles accrete earlier in a gas-rich phase., Comment: Accepted for publication in MNRAS

Data $\textit{quality}$ is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good" training data. However, a major challenge in achieving this goal is the observation that distillation approaches, which rely on sophisticated but mostly disparate methods to generate synthetic data, have little in common with each other. In this work, we highlight a largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance of state-of-the-art distillation methods is not the specific techniques used to generate synthetic data but rather the use of soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain $\textit{structured information}$ to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establish an empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available at https://github.com/sunnytqin/no-distillation.

This paper explores the application of deep learning (DL) techniques to strong motion records for single-station epicenter localization. Often underutilized in seismology-related studies, strong motion records offer a potential wealth of information about seismic events. We investigate whether DL-based methods can effectively leverage this data for accurate epicenter localization. Our study introduces AFAD-1218, a collection comprising more than 36,000 strong motion records sourced from Turkey. To utilize the strong motion records represented in either the time or the frequency domain, we propose two neural network architectures: deep residual network and temporal convolutional networks. Through extensive experimentation, we demonstrate the efficacy of DL approaches in extracting meaningful insights from these records, showcasing their potential for enhancing seismic event analysis and localization accuracy. Notably, our findings highlight significant reductions in prediction error achieved through the exclusion of low signal-to-noise ratio records, both in nationwide experiments and regional transfer-learning scenarios. Overall, this research underscores the promise of DL techniques in harnessing strong motion records for improved seismic event characterization and localization.

Radio Frequency Interference (RFI) is emitted from various sources, terrestrial or orbital, and create a nuisance for ground-based 21cm experiments. In particular, single-dish 21cm intensity mapping experiments will be highly susceptible to contamination from these sources due to its wide primary beam and sensitivity. This work aims to simulate the contamination effects emitted from orbital sources in the Radio Navigational Satellite System within the 1100-1350 MHz frequency. This simulation can be split into two parts: (I) satellite positioning, emission power, and beam response on the telescope and (II) fitting of the satellite signal to data in order to improve the original model. We use previously observed single dish MeerKAT L-band data which needs to be specially calibrated to include data contaminated by satellite-based RFI. We find that due to non-linearity effects, it becomes non-trivial to fit the satellite power. However, when masking regions where this non-linearity is problematic, we can recreate the satellite contamination with high accuracy around its peak frequencies. The simulation can predict satellite movements and signal for past and future observations, which can help in RFI avoidance and testing novel cleaning methods. The predicted signal from simulations sits below the noise in the target cosmology window for the L-band (970 - 1015 MHz) making it difficult to confirm any out-of-band emission from satellites. However, a power spectrum analysis shows that such signal can still contaminate the 21cm power spectrum at these frequencies. In our simulations, this contamination overwhelms the auto-power spectrum but still allows for a clean detection of the signal in cross-correlations with mild foreground cleaning. Whether such contamination does exist one will require further characterization of the satellite signals far away from their peak frequencies., Comment: 20 pages, 27 figures, 4 tables, Comments welcome

As large language models (LLMs) have gained popularity for a variety of use cases, making them adaptable and controllable has become increasingly important, especially for user-facing applications. While the existing literature on LLM adaptation primarily focuses on finding a model (or models) that optimizes a single predefined objective, here we focus on the challenging case where the model must dynamically adapt to diverse -- and often changing -- user preferences. For this, we leverage adaptation methods based on linear weight interpolation, casting them as continuous multi-domain interpolators that produce models with specific prescribed generation characteristics on-the-fly. Specifically, we use low-rank updates to fine-tune a base model to various different domains, yielding a set of anchor models with distinct generation profiles. Then, we use the weight updates of these anchor models to parametrize the entire (infinite) class of models contained within their convex hull. We empirically show that varying the interpolation weights yields predictable and consistent change in the model outputs with respect to all of the controlled attributes. We find that there is little entanglement between most attributes and identify and discuss the pairs of attributes for which this is not the case. Our results suggest that linearly interpolating between the weights of fine-tuned models facilitates predictable, fine-grained control of model outputs with respect to multiple stylistic characteristics simultaneously., Comment: 20 pages, 22 figures

Reference resolution is an important problem, one that is essential to understand and successfully handle context of different kinds. This context includes both previous turns and context that pertains to non-conversational entities, such as entities on the user's screen or those running in the background. While LLMs have been shown to be extremely powerful for a variety of tasks, their use in reference resolution, particularly for non-conversational entities, remains underutilized. This paper demonstrates how LLMs can be used to create an extremely effective system to resolve references of various types, by showing how reference resolution can be converted into a language modeling problem, despite involving forms of entities like those on screen that are not traditionally conducive to being reduced to a text-only modality. We demonstrate large improvements over an existing system with similar functionality across different types of references, with our smallest model obtaining absolute gains of over 5% for on-screen references. We also benchmark against GPT-3.5 and GPT-4, with our smallest model achieving performance comparable to that of GPT-4, and our larger models substantially outperforming it., Comment: Accepted at SIGDIAL 2024 (Oral presentation)

The chemical composition of an extrasolar planet is fundamental to its formation, evolution and habitability. In this study, we explore a new way to measure the chemical composition of the building blocks of extrasolar planets, by measuring the gas composition of the disrupted planetesimals around white dwarf stars. As a first attempt, we used the photo-ionization code Cloudy to model the circumstellar gas emission around a white dwarf Gaia J0611$-$6931 under some simplified assumptions. We found most of the emission lines are saturated and the line ratios approaching the ratios of thermal emission; therefore only lower limits to the number density can be derived. Silicon is the best constrained element in the circumstellar gas and we derived a lower limit of 10$^{10.3}$ cm$^{-3}$. In addition, we placed a lower limit on the total amount of gas to be 1.8 $\times$ 10$^{19}$ g. Further study is needed to better constrain the parameters of the gas disk and connect it to other white dwarfs with circumstellar gas absorption., Comment: 14 pages, 10 figures, 4 table, accepted for publication in AJ

What does a neural network learn when training from a task-specific dataset? Synthesizing this knowledge is the central idea behind Dataset Distillation, which recent work has shown can be used to compress large datasets into a small set of input-label pairs ($\textit{prototypes}$) that capture essential aspects of the original dataset. In this paper, we make the key observation that existing methods distilling into explicit prototypes are very often suboptimal, incurring in unexpected storage cost from distilled labels. In response, we propose $\textit{Distributional Dataset Distillation}$ (D3), which encodes the data using minimal sufficient per-class statistics and paired with a decoder, we distill dataset into a compact distributional representation that is more memory-efficient compared to prototype-based methods. To scale up the process of learning these representations, we propose $\textit{Federated distillation}$, which decomposes the dataset into subsets, distills them in parallel using sub-task experts and then re-aggregates them. We thoroughly evaluate our algorithm on a three-dimensional metric and show that our method achieves state-of-the-art results on TinyImageNet and ImageNet-1K. Specifically, we outperform the prior art by $6.9\%$ on ImageNet-1K under the storage budget of 2 images per class.