Search

Large language models (LLMs) are essential in natural language processing (NLP) but are costly in data collection, pre-training, fine-tuning, and inference. Task-specific small language models (SLMs) offer a cheaper alternative but lack robustness and generalization. This paper proposes LEGO, a novel technique to extract SLMs from an LLM and recombine them. Using state-of-the-art LLM pruning strategies, we can create task- and user-specific SLM building blocks that are efficient for fine-tuning and inference while also preserving user data privacy. LEGO utilizes Federated Learning and a novel aggregation scheme for the LLM reconstruction, maintaining robustness without high costs and preserving user data privacy. We experimentally demonstrate the versatility of LEGO, showing its ability to enable model heterogeneity and mitigate the effects of data heterogeneity while maintaining LLM robustness.

As virtual agents become increasingly prevalent in human-computer interaction, generating realistic and contextually appropriate gestures in real-time remains a significant challenge. While neural rendering techniques have made substantial progress with static scripts, their applicability to human-computer interactions remains limited. To address this, we introduce Large Body Language Models (LBLMs) and present LBLM-AVA, a novel LBLM architecture that combines a Transformer-XL large language model with a parallelized diffusion model to generate human-like gestures from multimodal inputs (text, audio, and video). LBLM-AVA incorporates several key components enhancing its gesture generation capabilities, such as multimodal-to-pose embeddings, enhanced sequence-to-sequence mapping with redefined attention mechanisms, a temporal smoothing module for gesture sequence coherence, and an attention-based refinement module for enhanced realism. The model is trained on our large-scale proprietary open-source dataset Allo-AVA. LBLM-AVA achieves state-of-the-art performance in generating lifelike and contextually appropriate gestures with a 30% reduction in Fr\'echet Gesture Distance (FGD), and a 25% improvement in Fr\'echet Inception Distance compared to existing approaches.

The scarcity of high-quality, multimodal training data severely hinders the creation of lifelike avatar animations for conversational AI in virtual environments. Existing datasets often lack the intricate synchronization between speech, facial expressions, and body movements that characterize natural human communication. To address this critical gap, we introduce Allo-AVA, a large-scale dataset specifically designed for text and audio-driven avatar gesture animation in an allocentric (third person point-of-view) context. Allo-AVA consists of $\sim$1,250 hours of diverse video content, complete with audio, transcripts, and extracted keypoints. Allo-AVA uniquely maps these keypoints to precise timestamps, enabling accurate replication of human movements (body and facial gestures) in synchronization with speech. This comprehensive resource enables the development and evaluation of more natural, context-aware avatar animation models, potentially transforming applications ranging from virtual reality to digital assistants.

Superconductor-normal material-superconductor (SNS) junctions are being integrated into microwave circuits for fundamental and applied research goals. The short junction limit is a common simplifying assumption for experiments with SNS junctions, but this limit constrains how small the nonlinearity of the microwave circuit can be. Here, we show that a finite length of the weak link strongly suppresses the nonlinearity compared to its zero-length limit -- the suppression can be up to a factor of ten even when the length remains shorter than the induced coherence length. We tie this behavior to the nonanalytic dependence of nonlinearity on length, which the critical current does not exhibit. Further, we identify additional experimentally observable consequences of nonzero length, and we conjecture that anharmonicity is bounded between zero and a maximally negative value for any non-interacting Josephson junction in the presence of time-reversal symmetry. We promote SNS junction length as a useful parameter for designing weakly nonlinear microwave circuits.

CORSIKA 8 is a new framework for simulations of particle cascades in air and dense media implemented in modern C++17, based on past experience with existing codes, in particular CORSIKA 7. The flexible and modular structure of the project allows the development of independent modules that can produce a fully customizable particle shower simulation. The radio module in particular is designed to treat the electric field calculation and its propagation through complex media to each observer location in an autonomous and flexible way. It already allows for the simultaneous simulation of the radio emission calculated with two independent time-domain formalisms, the "Endpoint formalism" as previously implemented in CoREAS and the "ZHS" algorithm as ported from ZHAireS. The design acts as the baseline interface for current and future development for the simulation of radio emission from particle showers in standard and complex scenarios, such as cross-media showers penetrating from air into ice. In this work, we present the design and implementation of the radio module in CORSIKA 8, along with validation studies and a direct comparison of the radio emission from air showers simulated with CORSIKA 8, CORSIKA 7 and ZHAireS. We also present the impact of simulation details such as the step size of simulated particle tracks on radio-emission simulations and perform a direct comparison of the "Endpoints" and "ZHS" formalisms for the same underlying air showers. Finally, we present an in-depth comparison of CORSIKA 8 and CORSIKA 7 for optimum simulation settings and discuss the relevance of observed differences in light of reconstruction efforts for the energy and mass of cosmic rays., Comment: 17 pages

Retrieval augmented generation (RAG) systems augment how knowledge language models are by integrating external information sources such as Wikipedia, internal documents, scientific papers, or the open internet. RAG systems that rely on the open internet as their knowledge source have to contend with the complexities of human-generated content. Human communication extends much deeper than just the words rendered as text. Intent, tonality, and connotation can all change the meaning of what is being conveyed. Recent real-world deployments of RAG systems have shown some difficulty in understanding these nuances of human communication. One significant challenge for these systems lies in processing sarcasm. Though the Large Language Models (LLMs) that make up the backbone of these RAG systems are able to detect sarcasm, they currently do not always use these detections for the subsequent processing of text. To address these issues, in this paper, we synthetically generate sarcastic passages from Natural Question's Wikipedia retrieval corpus. We then test the impact of these passages on the performance of both the retriever and reader portion of the RAG pipeline. We introduce a prompting system designed to enhance the model's ability to interpret and generate responses in the presence of sarcasm, thus improving overall system performance. Finally, we conduct ablation studies to validate the effectiveness of our approach, demonstrating improvements in handling sarcastic content within RAG systems.

An enhancement in the activity of the early young Sun resulting in a high charged particle flux has been invoked to explain excesses in spallation-induced nuclides in primitive planetary materials. Astronomical observations of energetic outbursts of young stellar objects (YSOs) also support the idea of an active young Sun. However, the early solar cosmic-ray (SCR) flux has not been well constrained. Here we use measured concentrations of SCR-produced nuclides that formed and are preserved in meteoritical hibonite and spinel, some of the oldest solids formed in the Solar System, and physical models for dust transport in the early protoplanetary disk to determine the magnitude of the early SCR flux. We focus our attention on cosmogenic neon which cannot have been inherited from precursors and can only be produced in situ in solids. Our modeled effective exposure time to SCRs for these solids is very short on the order of years. This indicates the early SCR flux recorded in refractory mineral hibonite was up to ~7 orders of magnitude higher than the contemporary level. Our flux estimate is consistent with the >10^5 enhanced flux inferred from astronomical observations of greatly enhanced flare activities of YSOs., Comment: 7 Figures, Accepted in ApJ

A common approach for sequence tagging tasks based on contextual word representations is to train a machine learning classifier directly on these embedding vectors. This approach has two shortcomings. First, such methods consider single input sequences in isolation and are unable to put an individual embedding vector in relation to vectors outside the current local context of use. Second, the high performance of these models relies on fine-tuning the embedding model in conjunction with the classifier, which may not always be feasible due to the size or inaccessibility of the underlying feature-generation model. It is thus desirable, given a collection of embedding vectors of a corpus, i.e., a datastore, to find features of each vector that describe its relation to other, similar vectors in the datastore. With this in mind, we introduce complexity measures of the local topology of the latent space of a contextual language model with respect to a given datastore. The effectiveness of our features is demonstrated through their application to dialogue term extraction. Our work continues a line of research that explores the manifold hypothesis for word embeddings, demonstrating that local structure in the space carved out by word embeddings can be exploited to infer semantic properties., Comment: Accepted as a long paper to SIGDIAL 2024. 9 pages, 2 figures, 3 tables

Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or proprietary, without retraining the model has become an important capability. This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model, significantly outperforming the discriminatory abilities of the previous state-of-the-art. A dual method called LEARN is also proposed for targeted knowledge addition. Results show LEARN can match the fine-tuning accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar tasks., Comment: 11 pages, 2 Figures

Emotions are indispensable in human communication, but are often overlooked in task-oriented dialogue (ToD) modelling, where the task success is the primary focus. While existing works have explored user emotions or similar concepts in some ToD tasks, none has so far included emotion modelling into a fully-fledged ToD system nor conducted interaction with human or simulated users. In this work, we incorporate emotion into the complete ToD processing loop, involving understanding, management, and generation. To this end, we extend the EmoWOZ dataset (Feng et al., 2022) with system affective behaviour labels. Through interactive experimentation involving both simulated and human users, we demonstrate that our proposed framework significantly enhances the user's emotional experience as well as the task success., Comment: Accepted by SIGDIAL 2024

State-of-the-art task-oriented dialogue systems typically rely on task-specific ontologies for fulfilling user queries. The majority of task-oriented dialogue data, such as customer service recordings, comes without ontology and annotation. Such ontologies are normally built manually, limiting the application of specialised systems. Dialogue ontology construction is an approach for automating that process and typically consists of two steps: term extraction and relation extraction. In this work, we focus on relation extraction in a transfer learning set-up. To improve the generalisation, we propose an extension to the decoding mechanism of large language models. We adapt Chain-of-Thought (CoT) decoding, recently developed for reasoning problems, to generative relation extraction. Here, we generate multiple branches in the decoding space and select the relations based on a confidence threshold. By constraining the decoding to ontology terms and relations, we aim to decrease the risk of hallucination. We conduct extensive experimentation on two widely used datasets and find improvements in performance on target ontology for source fine-tuned and one-shot prompted large language models., Comment: Accepted to appear at SIGDIAL 2024. 9 pages, 4 figures

As the nation transitions from responding to the height of the COVID-19 pandemic to navigating its ongoing impact on youth and their communities, K-12 students continue to experience mental health and wellbeing challenges. In the fall of 2021, Redwood City School District (RCSD) placed a full-time, district-employed mental health counselor in each of its twelve schools in order to bolster district capacity to support mental health and wellbeing. This study aims to identify strategic opportunities for understanding and advancing the counseling program's impact as it moves into its third year. With these goals in mind, this research brief will describe findings related to three questions: (1) What is the role of RCSD's school-based mental health counselors?; (2) What are early signals or "lead indicators" of the value or impact of the counseling program?; and (3) What considerations might further improve the program as it moves into Year 3? To answer these questions, the authors conducted 19 interviews with district administrators, site administrators, and mental health counselors; and 22 participant observations of meetings related to the counseling program and student services more broadly. While the research team employed qualitative data analysis software (Dedoose 9.0.90) to facilitate the coding of qualitative data, the analysis of coded data followed a more traditional, iterative approach, moving between deductive and inductive reasoning. This research brief summarizes key findings and offers considerations for leveraging the counseling program to strengthen a district-wide, coordinated system of support for mental health and wellbeing and developing a theory-based evaluation of impact during Year 3 of program implementation. [This work is generously funded by Stanford's Office of the Provost through the Office of Community Engagement.]

We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems by distilling behaviors from auxiliary RL tasks. AuxDistill achieves this by concurrently carrying out multi-task RL with auxiliary tasks, which are easier to learn and relevant to the main task. A weighted distillation loss transfers behaviors from these auxiliary tasks to solve the main task. We demonstrate that AuxDistill can learn a pixels-to-actions policy for a challenging multi-stage embodied object rearrangement task from the environment reward without demonstrations, a learning curriculum, or pre-trained skills. AuxDistill achieves $2.3 \times$ higher success than the previous state-of-the-art baseline in the Habitat Object Rearrangement benchmark and outperforms methods that use pre-trained skills and expert demonstrations.

An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed of text, equations, figures, and tables, SIMMC methods must be developed specifically for each component to support the depth of inquiry and interactions required by research scientists. This work introduces Conversational Papers (cPAPERS), a dataset of conversational question-answer pairs from reviews of academic papers grounded in these paper components and their associated references from scientific documents available on arXiv. We present a data collection strategy to collect these question-answer pairs from OpenReview and associate them with contextual information from LaTeX source files. Additionally, we present a series of baseline approaches utilizing Large Language Models (LLMs) in both zero-shot and fine-tuned configurations to address the cPAPERS dataset., Comment: 14 pages, 1 figure

This paper introduces Interactive Tables (iTBLS), a dataset of interactive conversations situated in tables from scientific articles. This dataset is designed to facilitate human-AI collaborative problem-solving through AI-powered multi-task tabular capabilities. In contrast to prior work that models interactions as factoid QA or procedure synthesis, iTBLS broadens the scope of interactions to include mathematical reasoning, natural language manipulation, and expansion of existing tables from natural language conversation by delineating interactions into one of three tasks: interpretation, modification, or generation. Additionally, the paper presents a suite of baseline approaches to iTBLS, utilizing zero-shot prompting and parameter-efficient fine-tuning for different computing situations. We also introduce a novel multi-step approach and show how it can be leveraged in conjunction with parameter-efficient fine-tuning to achieve the state-of-the-art on iTBLS; outperforming standard parameter-efficient fine-tuning by up to 15% on interpretation, 18% on modification, and 38% on generation., Comment: 14 pages, 4 figures

Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables is ensuring their syntactic validity. Prior approaches address this challenge by including additional parameters in the Transformer's attention mechanism to attend to specific rows and column headers. In contrast to this single-stage method, this paper presents a two-stage approach called Generative Tables (gTBLS). The first stage infers table structure (row and column headers) from the text. The second stage formulates questions using these headers and fine-tunes a causal language model to answer them. Furthermore, the gTBLS approach is amenable to the utilization of pre-trained Large Language Models in a zero-shot configuration, presenting a solution for table generation in situations where fine-tuning is not feasible. gTBLS improves prior approaches by up to 10% in BERTScore on the table construction task and up to 20% on the table content generation task of the E2E, WikiTableText, WikiBio, and RotoWire datasets., Comment: 12 pages, 1 figure

Wind turbines operate in the atmospheric boundary layer (ABL), where Coriolis effects are present. As wind turbines with larger rotor diameters are deployed, the wake structures that they create in the ABL also increase in length. Contemporary utility-scale wind turbines operate at rotor diameter-based Rossby numbers, the nondimensional ratio between inertial and Coriolis forces, of O(100) where Coriolis effects become increasingly relevant. Coriolis forces provide a direct forcing on the wake, but also affect the ABL base flow, which indirectly influences wake evolution. These effects may constructively or destructively interfere because both the magnitude and sign of the direct and indirect Coriolis effects depend on the Rossby number, turbulence, and buoyancy effects in the ABL. Using large eddy simulations, we investigate wake evolution over a wide range of Rossby numbers relevant to offshore wind turbines. Through an analysis of the streamwise and lateral momentum budgets, we show that Coriolis effects have a small impact on the wake recovery rate, but Coriolis effects induce significant wake deflections which can be parsed into two regimes. For high Rossby numbers (weak Coriolis forcing), wakes deflect clockwise in the northern hemisphere. By contrast, for low Rossby numbers (strong Coriolis forcing), wakes deflect anti-clockwise. Decreasing the Rossby number results in increasingly anti-clockwise wake deflections. The transition point between clockwise and anti-clockwise deflection depends on the direct Coriolis forcing, pressure gradients, and turbulent fluxes in the wake. At a Rossby number of 125, Coriolis deflections are comparable to wake deflections induced by 17{\deg} of yaw-misalignment., Comment: 40 pages, 20 figures, 1 table

This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three use cases of LLMs: interactions over web content, code, and mathematical texts. By analyzing attention distance, dispersion, and interdependency across these domains, we highlight the unique challenges posed by conversational data. Notably, conversations require nuanced handling of long-term contextual relationships and exhibit higher complexity through their attention patterns. Our findings reveal that while language models exhibit domain-specific attention behaviors, there is a significant gap in their ability to specialize in human conversations. Through detailed attention entropy analysis and t-SNE visualizations, we demonstrate the need for models trained with a diverse array of high-quality conversational data to enhance understanding and generation of human-like dialogue. This research highlights the importance of domain specialization in language models and suggests pathways for future advancement in modeling human conversational nuances.

Attosecond photoemission or photoionization delays are a unique probe of the structure and the electronic dynamics of matter. However, spectral congestion and spatial delocalization of valence electron wave functions set fundamental limits to the complexity of systems that can be studied and the information that can be retrieved, respectively. Using attosecond X-ray pulses from LCLS, we demonstrate the key advantages of measuring core-level delays: the photoelectron spectra remain atom-like, the measurements become element specific and the observed scattering dynamics originate from a point-like source. We exploit these unique features to reveal the effects of electronegativity and symmetry on attosecond scattering dynamics by measuring the photoionization delays between N-1s and C-1s core shells of a series of aromatic azabenzene molecules. Remarkably, the delays systematically increase with the number of nitrogen atoms in the molecule and reveal multiple resonances. We identify two previously unknown mechanisms regulating the associated attosecond dynamics, namely the enhanced confinement of the trapped wavefunction with increasing electronegativity of the atoms and the decrease of the coupling strength among the photoemitted partial waves with increasing symmetry. This study demonstrates the unique opportunities opened by measurements of core-level photoionization delays for unravelling attosecond electron dynamics in complex matter.

Dense passage retrieval (DPR) is the first step in the retrieval augmented generation (RAG) paradigm for improving the performance of large language models (LLM). DPR fine-tunes pre-trained networks to enhance the alignment of the embeddings between queries and relevant textual data. A deeper understanding of DPR fine-tuning will be required to fundamentally unlock the full potential of this approach. In this work, we explore DPR-trained models mechanistically by using a combination of probing, layer activation analysis, and model editing. Our experiments show that DPR training decentralizes how knowledge is stored in the network, creating multiple access pathways to the same information. We also uncover a limitation in this training style: the internal knowledge of the pre-trained model bounds what the retrieval model can retrieve. These findings suggest a few possible directions for dense retrieval: (1) expose the DPR training process to more knowledge so more can be decentralized, (2) inject facts as decentralized representations, (3) model and incorporate knowledge uncertainty in the retrieval process, and (4) directly map internal model knowledge to a knowledge base.

AI systems cannot exist without data. Now that AI models (data science and AI) have matured and are readily available to apply in practice, most organizations struggle with the data infrastructure to do so. There is a growing need for data engineers that know how to prepare data for AI systems or that can setup enterprise-wide data architectures for analytical projects. But until now, the data engineering part of AI engineering has not been getting much attention, in favor of discussing the modeling part. In this paper we aim to change this by perform a mapping study on data engineering for AI systems, i.e., AI data engineering. We found 25 relevant papers between January 2019 and June 2023, explaining AI data engineering activities. We identify which life cycle phases are covered, which technical solutions or architectures are proposed and which lessons learned are presented. We end by an overall discussion of the papers with implications for practitioners and researchers. This paper creates an overview of the body of knowledge on data engineering for AI. This overview is useful for practitioners to identify solutions and best practices as well as for researchers to identify gaps., Comment: Preprint, accepted for CAIN24

We present an integrated magneto-photonic device for all-optical switching of non-volatile multi-bit spintronic memory. The bits are based on stand-alone magneto-tunnel junctions which are perpendicularly magnetized with all-optically switchable free layers, coupled onto photonic crystal nanobeam cavities on an indium phosphide based platform. This device enables switching of the magnetization state of the bits by locally increasing the power absorption of light at resonance with the cavity. We design an add/drop network of cavities to grant random access to multiple bits via a wavelength-division multiplexing scheme. Based on a three-dimensional finite-difference time-domain method, we numerically illustrate a compact device capable of switching and accessing 8 bits in different cavities with a 5 nm wavelength spacing in the conventional (C) telecommunication band. Our multi-bit device holds promise as a new paradigm for developing an ultrafast photonically-addressable spintronic memory and may also empower novel opportunities for photonically-driven spintronic-based neuromorphic computing., Comment: 12 pages, 6 figures, 1 table

Formation, maintenance and physiology of high-density protein-enriched organized nanodomains, first observed in electron microscopy images, remains challenging to investigate due to their small sizes. However, these regions regulate molecular trafficking, assembly and sorting required for higher cell functions, such as communication or plastic changes. Over the past ten years, super-resolution single-particle trajectories (SPTs) have been used to sample these sub-cellular environments at a nanometer resolution for both membrane and soluble proteins. We present here data analysis developments and algorithms that convert high-throughput molecular trajectories into maps of molecular density, diffusion and local drift organization. These approaches transform intrinsic trajectory properties into statistics of the underlying cellular organization. The automatic identification of large numbers of high-density regions allows quantifying their boundary location and organization, their stability over time and their ability to transiently retain molecules. To conclude recent automated algorithms can now be used to extract biophysical parameters of sub-cellular nanodomains over a large amount of trajectories., Comment: 6 figs

Supervised neural approaches are hindered by their dependence on large, meticulously annotated datasets, a requirement that is particularly cumbersome for sequential tasks. The quality of annotations tends to deteriorate with the transition from expert-based to crowd-sourced labelling. To address these challenges, we present CAMEL (Confidence-based Acquisition Model for Efficient self-supervised active Learning), a pool-based active learning framework tailored to sequential multi-output problems. CAMEL possesses two core features: (1) it requires expert annotators to label only a fraction of a chosen sequence, and (2) it facilitates self-supervision for the remainder of the sequence. By deploying a label correction mechanism, CAMEL can also be utilised for data cleaning. We evaluate CAMEL on two sequential tasks, with a special emphasis on dialogue belief tracking, a task plagued by the constraints of limited and noisy datasets. Our experiments demonstrate that CAMEL significantly outperforms the baselines in terms of efficiency. Furthermore, the data corrections suggested by our method contribute to an overall improvement in the quality of the resulting datasets., Comment: Accepted at TACL

Fluxons in a superconducting loop can be coherently coupled by quantum phase slips occurring at a weak link such as a Josephson junction. If Cooper pair tunneling at the junction occurs through a resonant level, $2\pi$ quantum phase slips are suppressed, and fluxons are predominantly coupled by $4\pi$ quantum phase slips. We analyze this scenario by computing the coupling between fluxons as the level is brought into resonance with the superconducting condensate. The results indicate that the $4\pi$-dominated regime can be observed directly in the transition spectrum for circuit parameters typical of a fluxonium qubit. We also show that, if the inductive energy of the loop is much smaller than the plasma frequency of the junction, the low-energy Hamiltonian of the circuit is dual to that of a topological superconducting island. These findings can inform experiments on bifluxon qubits as well as the design of novel types of protected qubits.