Search

A massive black hole can develop a dark-matter overdensity, and the dark matter changes the evolution of a stellar-mass compact object inspiraling around the massive black hole through the dense dark-matter environment. Specifically, dynamical friction speeds up the inspiral of the compact object and causes feedback on the dark-matter distribution. These intermediate mass-ratio inspirals with dark matter are a source of gravitational waves (GWs), and the waves can dephase significantly from an equivalent system in vacuum. Prior work has shown that this dephasing needs to be modeled to detect the GWs from these systems with LISA (the Laser Interferometer Space Antenna); it also showed that the density and distribution of dark matter can be inferred from a GW measurement. In this paper, we study whether the parametrized post-Einsteinian (ppE) framework can be used to infer the presence of dark matter in these systems. We confirm that if vacuum waveform templates are used to model the GWs from an inspiral in a dark-matter halo, then the resulting parameter estimation is biased. We then apply the ppE framework to determine whether it can reduce the parameter-estimation biases, and we find that adding one ppE phase term to a waveform template eliminates the parameter-estimation biases (statistical errors become larger than the systematic ones), but the effective post-Newtonian order in the ppE framework must be specified without uncertainties. When the post-Newtonian order has uncertainty, we find that the systematic errors on the ppE and the binary's parameters exceed the statistical errors. Thus, the simplest ppE framework would not give unbiased results for these systems, and a further extension of it, or dedicated parameter estimation with gravitational waveforms that include dark-matter effects would be needed., Comment: 16 pages, 8 figures

We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit (e.g., a word) is an affine function of its in-context information content. We first observe that surprisal is only one out of many potential ways that a contextual predictor can be derived from a language model. Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency. Moreover, both PMI and surprisal are correlated with frequency. This means that neither PMI nor surprisal contains information about context alone. In response to this, we propose a technique where we project surprisal onto the orthogonal complement of frequency, yielding a new contextual predictor that is uncorrelated with frequency. Our experiments show that the proportion of variance in reading times explained by context is a lot smaller when context is represented by the orthogonalized predictor. From an interpretability standpoint, this indicates that previous studies may have overstated the role that context has in predicting reading times., Comment: EMNLP 2024

We report a proof-of-principle lattice QCD simulation of the electric and magnetic polarizabilities for a neutral pion in the four-point function method. The results are based on the same quenched Wilson ensembles on a $24^3\times 48$ lattice at $\beta=6.0$ with pion mass from 1100 to 370 MeV previously used for a charged pion. For electric polarizability, the results are largely consistent with those from the background field method and ChPT. In contrast, there are significant differences for magnetic polarizability among the four-point function method, the background field method, and ChPT. The situation points to the potentially important role of disconnected diagrams for a neutral pion. We elucidate a transparent quark decomposition in the four-point function method that can be used to shed light on the issue., Comment: 11 pages, 9 figures, 1 table. This version matches the one published in NPB in content. arXiv admin note: text overlap with arXiv:2307.08620

Density reconstruction from X-ray projections is an important problem in radiography with key applications in scientific and industrial X-ray computed tomography (CT). Often, such projections are corrupted by unknown sources of noise and scatter, which when not properly accounted for, can lead to significant errors in density reconstruction. In the setting of this problem, recent deep learning-based methods have shown promise in improving the accuracy of density reconstruction. In this article, we propose a deep learning-based encoder-decoder framework wherein the encoder extracts robust features from noisy/corrupted X-ray projections and the decoder reconstructs the density field from the features extracted by the encoder. We explore three options for the latent-space representation of features: physics-inspired supervision, self-supervision, and no supervision. We find that variants based on self-supervised and physicsinspired supervised features perform better over a range of unknown scatter and noise. In extreme noise settings, the variant with self-supervised features performs best. After investigating further details of the proposed deep-learning methods, we conclude by demonstrating that the newly proposed methods are able to achieve higher accuracy in density reconstruction when compared to a traditional iterative technique.

Fairness in AI and ML systems is increasingly linked to the proper treatment and recognition of data workers involved in training dataset development. Yet, those who collect and annotate the data, and thus have the most intimate knowledge of its development, are often excluded from critical discussions. This exclusion prevents data annotators, who are domain experts, from contributing effectively to dataset contextualization. Our investigation into the hiring and engagement practices of 52 data work requesters on platforms like Amazon Mechanical Turk reveals a gap: requesters frequently hold naive or unchallenged notions of worker identities and capabilities and rely on ad-hoc qualification tasks that fail to respect the workers' expertise. These practices not only undermine the quality of data but also the ethical standards of AI development. To rectify these issues, we advocate for policy changes to enhance how data annotation tasks are designed and managed and to ensure data workers are treated with the respect they deserve., Comment: Accepted for publication at AIES 2024

We represent the outermost shear interface of an eddy by a circular vortex sheet in two dimensions, and provide a new proof of linear instability via the Birkhoff-Rott equation. Like planar vortex sheets, circular sheets are found to be susceptible to a violent short-wave instability known as the Kelvin-Helmholtz instability, with some modifications due to vortex sheet geometry. This result is in agreement with the classical derivation of (Moore 1974, Saffman 1992), but our modern approach provides greater clarity. We go on to show that the linear evolution problem can develop a singularity from analytic initial data in a time proportional to the square of the vortex sheet radius. Numerical evidence is presented that suggests this linear instability captures the wave-breaking mechanism observed in nonlinear point vortex simulations. Based on these results, we hypothesize that the Kelvin-Helmholtz instability can contribute to the development of secondary instability for eddies in two-dimensional turbulent flow., Comment: arXiv admin note: text overlap with arXiv:2211.03585

A trained attention-based transformer network can robustly recover the complex topologies given by the Richtmyer-Meshkoff instability from a sequence of hydrodynamic features derived from radiographic images corrupted with blur, scatter, and noise. This approach is demonstrated on ICF-like double shell hydrodynamic simulations. The key component of this network is a transformer encoder that acts on a sequence of features extracted from noisy radiographs. This encoder includes numerous self-attention layers that act to learn temporal dependencies in the input sequences and increase the expressiveness of the model. This approach is demonstrated to exhibit an excellent ability to accurately recover the Richtmyer-Meshkov instability growth rates, even despite the gas-metal interface being greatly obscured by radiographic noise.

Calls for engagement with the public in Artificial Intelligence (AI) research, development, and governance are increasing, leading to the use of surveys to capture people's values, perceptions, and experiences related to AI. In this paper, we critically examine the state of human participant surveys associated with these topics. Through both a reflexive analysis of a survey pilot spanning six countries and a systematic literature review of 44 papers featuring public surveys related to AI, we explore prominent perspectives and methodological nuances associated with surveys to date. We find that public surveys on AI topics are vulnerable to specific Western knowledge, values, and assumptions in their design, including in their positioning of ethical concepts and societal values, lack sufficient critical discourse surrounding deployment strategies, and demonstrate inconsistent forms of transparency in their reporting. Based on our findings, we distill provocations and heuristic questions for our community, to recognize the limitations of surveys for meeting the goals of engagement, and to cultivate shared principles to design, deploy, and interpret surveys cautiously and responsibly., Comment: To appear in 7th AAAI Conference on AI, Ethics, and Society (AIES)

Generalization capabilities, or rather a lack thereof, is one of the most important unsolved problems in the field of robot learning, and while several large scale efforts have set out to tackle this problem, unsolved it remains. In this paper, we hypothesize that learning temporal action abstractions using latent variable models (LVMs), which learn to map data to a compressed latent space and back, is a promising direction towards low-level skills that can readily be used for new tasks. Although several works have attempted to show this, they have generally been limited by architectures that do not faithfully capture shareable representations. To address this we present Quantized Skill Transformer (QueST), which learns a larger and more flexible latent encoding that is more capable of modeling the breadth of low-level skills necessary for a variety of tasks. To make use of this extra flexibility, QueST imparts causal inductive bias from the action sequence data into the latent space, leading to more semantically useful and transferable representations. We compare to state-of-the-art imitation learning and LVM baselines and see that QueST's architecture leads to strong performance on several multitask and few-shot learning benchmarks. Further results and videos are available at https://quest-model.github.io/, Comment: Keywords: Behavior Clonning, Action Quantization, Self Supervised Skill Abstraction, Few-shot Imitation Learning

National Science Foundation (NSF) funded Research Experiences for Undergraduates (REUs) are explicitly intended to reach minoritized students in STEM and those who have few research opportunities. Many undergraduates are encouraged to seek them out, but their actual efficacy is not well-established, and the out-of-state travel required for many attendees may prove a significant barrier for the very students REUs wish to reach. We interviewed physics graduate students who attended REUs as undergrauates, focusing on how the REUs benefitted them, barriers they faced attending REUs, and their relationship with their REU mentors. Interviewees reported benefits that aligned with the NSF goals: skills, enculturation, and knowledge they had not received in their undergraduate institutions. They also reported financial barriers they faced which they were able to overcome due to their financial privilege. Participants also reported widely varying experiences with their mentors. Some mentors did and some did not meet their mentees where they were at in their career and skill levels. Some students did not know how to approach their mentors with their questions or needs.

Experimental measurements of plasma and neutral profiles across the pedestal are used in conjunction with 2D edge modeling to examine pedestal stiffness in Alcator C-Mod H-mode plasmas. Experiments on Alcator C-Mod observed pedestal degradation and loss in confinement below a critical value of net power crossing the separatrix, $P_\mathrm{net} = P_\mathrm{net}^\mathrm{crit} \approx 2.3$ MW. New analysis of ionization and particle flux profiles reveal saturation of the pedestal electron density, $n_{e}^\mathrm{ped}$ despite continuous increases in ionization throughout the pedestal, inversely related to $P_\mathrm{net}$. A limit to the pedestal $\nabla n_{e}$ emerges as the particle flux, $\Gamma_{D}$ continues to grow, implying increases in the effective particle diffusivity, $D_\mathrm{eff}$. This is well-correlated with the separatrix collisionality, $\nu^{*}_\mathrm{sep}$ and a turbulence control parameter, $\alpha_{t}$, implying a possible transition in type of turbulence. The transition is well correlated with the experimentally observed value of $P_\mathrm{net}^\mathrm{crit}$. SOLPS-ITER modeling is performed for select discharges from the power scan, constrained with experimental electron and neutral densities, measured at the outer midplane. The modeling confirms general growth in $D_\mathrm{eff}$, consistent with experimental findings, and additionally suggests even larger growth in $\chi_{e}$ at the same $P_\mathrm{net}^\mathrm{crit}$.

At the high $n_{e}$ proposed for high-field fusion reactors, it is uncertain whether ionization, as opposed to plasma transport, will be most influential in determining $n_{e}$ at the pedestal and separatrix. A database of Alcator C-Mod discharges is analyzed to evaluate the impact of source modification via cryopumping. The database contains similarly-shaped H-modes at fixed $I_{p} =$ 0.8 MA and $B_{t} =$ 5.4 T, spanning a large range in $P_\mathrm{net}$ and ionization. Measurements from an edge Thomson Scattering system are combined with those from a midplane-viewing Ly$_{\alpha}$ camera to evaluate changes to $n_{e}$ and $T_{e}$ in response to changes to ionization rates, $S_\mathrm{ion}$. $n_{e}^\mathrm{sep}$ and $T_{e}^\mathrm{ped}$ are found to be most sensitive to changes to $S_\mathrm{ion}^\mathrm{sep}$, as opposed to $n_{e}^\mathrm{ped}$ and $T_{e}^\mathrm{sep}$. Dimensionless quantities, namely $\alpha_\mathrm{MHD}$ and $\nu^{*}$, are found to regulate attainable pedestal values. Select discharges at different values of $P_\mathrm{net}$ and in different pumping configurations are analyzed further using SOLPS-ITER. It is determined that changes to plasma transport coefficients are required to self-consistently model both plasma and neutral edge dynamics. Pumping is found to modify the poloidal distribution of atomic neutral density, $n_{0}$, along the separatrix, increasing $n_{0}$ at the active X-point. Opaqueness to neutrals from high $n_{e}$ in the divertor is found to play a role in mediating neutral penetration lengths and hence, the poloidal distribution of neutrals along the separatrix. Pumped discharges thus require a larger particle diffusion coefficient than that inferred purely from 1D experimental profiles at the outer midplane.

As quantum technologies transition out of the research lab and into commercial applications, it becomes important to better prepare students to enter this new and evolving workforce. To work towards this goal of preparing physics students for a career in the quantum industry, a senior capstone course called "Quantum Forge" was created at the University of Colorado Boulder. This course aims to provide students a hands-on quantum experience and prepare them to enter the quantum workforce directly after their undergraduate studies. Some of the course's goals are to have students understand what comprises the quantum industry and have them feel confident they could enter the industry if desired. To understand to what extent these goals are achieved, we followed the first cohort of Quantum Forge students through their year in the course in order to understand their perceptions of the quantum industry including what it is, whether they feel that they could be successful in it, and whether or not they want to participate in it. The results of this work can assist educators in optimizing the design of future quantum-industry-focused courses and programs to better prepare students to be a part of this burgeoning industry.

Tokamak plasmas with strong negative triangularity (NT) shaping typically exhibit fundamentally different edge behavior than conventional L-mode or H-mode plasmas. Over the entire DIII-D database, plasmas with sufficiently negative triangularity are found to be inherently free of edge localized modes (ELMs), even at injected powers well above the predicted L-H power threshold. A critical triangularly ($\delta_\mathrm{crit}\simeq-0.15$), consistent with inherently ELM-free operation is identified, beyond which access to the second stability region for infinite-$n$ ballooning modes closes on DIII-D. It is also possible to close access to this region, and thereby prevent an H-mode transition, at weaker average triangularities ($\delta\lesssim\delta_\mathrm{crit}$) provided that at least one of the two x-points is still sufficiently negative. Enhanced low field side magnetic fluctuations during ELM-free operation are consistent with additional turbulence limiting the NT edge gradient. Despite the reduced upper limit on the pressure gradient imposed by ballooning stability, NT plasmas are able to support small pedestals and are typically characterized by an enhancement of edge pressure gradients beyond those found in traditional L-mode plasmas. Further, the pressure gradient inside of this small pedestal is unusually steep, allowing access to high core performance that is competitive with other ELM-free regimes previously achieved on DIII-D. Since ELM-free operation in NT is linked directly to the magnetic geometry, NT fusion pilot plants are predicted to maintain advantageous edge conditions even in burning plasma regimes, potentially eliminating reactor core-integration issues caused by ELMs.

The ability to build and leverage world models is essential for a general-purpose AI agent. Testing such capabilities is hard, in part because the building blocks of world models are ill-defined. We present Elements of World Knowledge (EWOK), a framework for evaluating world modeling in language models by testing their ability to use knowledge of a concept to match a target text with a plausible/implausible context. EWOK targets specific concepts from multiple knowledge domains known to be vital for world modeling in humans. Domains range from social interactions (help/hinder) to spatial relations (left/right). Both, contexts and targets are minimal pairs. Objects, agents, and locations in the items can be flexibly filled in enabling easy generation of multiple controlled datasets. We then introduce EWOK-CORE-1.0, a dataset of 4,374 items covering 11 world knowledge domains. We evaluate 20 openweights large language models (1.3B--70B parameters) across a battery of evaluation paradigms along with a human norming study comprising 12,480 measurements. The overall performance of all tested models is worse than human performance, with results varying drastically across domains. These data highlight simple cases where even large models fail and present rich avenues for targeted research on LLM world modeling capabilities., Comment: 21 pages (11 main), 7 figures. Authors Anna Ivanova, Aalok Sathe, Benjamin Lipkin contributed equally

The path of tokamak fusion and ITER is maintaining high-performance plasma to produce sufficient fusion power. This effort is hindered by the transient energy burst arising from the instabilities at the boundary of high-confinement plasmas. The application of 3D magnetic perturbations is the method in ITER and possibly in future fusion power plants to suppress this instability and avoid energy busts damaging the device. Unfortunately, the conventional use of the 3D field in tokamaks typically leads to degraded fusion performance and an increased risk of other plasma instabilities, two severe issues for reactor implementation. In this work, we present an innovative 3D field optimization, exploiting machine learning, real-time adaptability, and multi-device capabilities to overcome these limitations. This integrated scheme is successfully deployed on DIII-D and KSTAR tokamaks, consistently achieving reactor-relevant core confinement and the highest fusion performance without triggering damaging instabilities or bursts while demonstrating ITER-relevant automated 3D optimization for the first time. This is enabled both by advances in the physics understanding of self-organized transport in the plasma edge and by advances in machine-learning technology, which is used to optimize the 3D field spectrum for automated management of a volatile and complex system. These findings establish real-time adaptive 3D field optimization as a crucial tool for ITER and future reactors to maximize fusion performance while simultaneously minimizing damage to machine components.

Palm oil production has been identified as one of the major drivers of deforestation for tropical countries. To meet supply chain objectives, commodity producers and other stakeholders need timely information of land cover dynamics in their supply shed. However, such data are difficult to obtain from suppliers who may lack digital geographic representations of their supply sheds and production locations. Here we present a "community model," a machine learning model trained on pooled data sourced from many different stakeholders, to develop a specific land cover probability map, in this case a semi-global oil palm map. An advantage of this method is the inclusion of varied inputs, the ability to easily update the model as new training data becomes available and run the model on any year that input imagery is available. Inclusion of diverse data sources into one probability map can help establish a shared understanding across stakeholders on the presence and absence of a land cover or commodity (in this case oil palm). The model predictors are annual composites built from publicly available satellite imagery provided by Sentinel-1, Sentinel-2, and ALOS DSM. We provide map outputs as the probability of palm in a given pixel, to reflect the uncertainty of the underlying state (palm or not palm). The initial version of this model provides global accuracy estimated to be approximately 90% (at 0.5 probability threshold) from spatially partitioned test data. This model, and resulting oil palm probability map products are useful for accurately identifying the geographic footprint of palm cultivation. Used in conjunction with timely deforestation information, this palm model is useful for understanding the risk of continued oil palm plantation expansion in sensitive forest areas., Comment: v0

Monograph "B. Grechuk, Polynomial Diophantine equations. A systematic approach" suggests solving Diophantine equations systematically in certain order. Many hundreds of the equations are left to the reader. Here, we provide complete solutions to all these equations. The difficulties of solved equations range from elementary to research level. In the last section, we present a summary table of all solved equations ordered by their size, which makes them easy to find. As a result, this document may be used as a database of solved Diophantine equations of small size, to which more difficult equations can be reduced.

After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the competition rules will be different. The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques. Second, we are relaxing the rules around pretraining data, and will now allow participants to construct their own datasets provided they stay within the 100M-word or 10M-word budget. Third, we introduce a multimodal vision-and-language track, and will release a corpus of 50% text-only and 50% image-text multimodal data as a starting point for LM model training. The purpose of this CfP is to provide rules for this year's challenge, explain these rule changes and their rationale in greater detail, give a timeline of this year's competition, and provide answers to frequently asked questions from last year's challenge.

Cut meshes are a type of mesh that is formed by allowing embedded boundaries to "cut" a simple underlying mesh resulting in a hybrid mesh of cut and standard elements. While cut meshes can allow complex boundaries to be represented well regardless of the mesh resolution, their arbitrarily shaped and sized cut elements can present issues such as the small cell problem, where small cut elements can result in a severely restricted CFL condition. State redistribution, a technique developed by Berger and Giuliani [1], can be used to address the small cell problem. In this work, we pair state redistribution with a high-order discontinuous Galerkin scheme that is $L_2$ energy stable for arbitrary quadrature. We prove that state redistribution can be added to a provably $L_2$ energy stable discontinuous Galerkin method on a cut mesh without damaging the scheme's $L_2$ stability. We numerically verify the high order accuracy and stability of our scheme on two-dimensional wave propagation problems.

With the upcoming AI regulations (e.g., EU AI Act) and rapid advancements in generative AI, new challenges emerge in the area of Human-Centered Responsible Artificial Intelligence (HCR-AI). As AI becomes more ubiquitous, questions around decision-making authority, human oversight, accountability, sustainability, and the ethical and legal responsibilities of AI and their creators become paramount. Addressing these questions requires a collaborative approach. By involving stakeholders from various disciplines in the 2\textsuperscript{nd} edition of the HCR-AI Special Interest Group (SIG) at CHI 2024, we aim to discuss the implications of regulations in HCI research, develop new theories, evaluation frameworks, and methods to navigate the complex nature of AI ethics, steering AI development in a direction that is beneficial and sustainable for all of humanity., Comment: 6 pages

Prompt-based interfaces for Large Language Models (LLMs) have made prototyping and building AI-powered applications easier than ever before. However, identifying potential harms that may arise from AI applications remains a challenge, particularly during prompt-based prototyping. To address this, we present Farsight, a novel in situ interactive tool that helps people identify potential harms from the AI applications they are prototyping. Based on a user's prompt, Farsight highlights news articles about relevant AI incidents and allows users to explore and edit LLM-generated use cases, stakeholders, and harms. We report design insights from a co-design study with 10 AI prototypers and findings from a user study with 42 AI prototypers. After using Farsight, AI prototypers in our user study are better able to independently identify potential harms associated with a prompt and find our tool more useful and usable than existing resources. Their qualitative feedback also highlights that Farsight encourages them to focus on end-users and think beyond immediate harms. We discuss these findings and reflect on their implications for designing AI prototyping experiences that meaningfully engage with AI harms. Farsight is publicly accessible at: https://PAIR-code.github.io/farsight., Comment: Accepted to CHI 2024 (Best Paper, Honorable Mention). 40 pages, 19 figures, 5 tables. For a demo video, see https://youtu.be/BlSFbGkOlHk. For a live demo, visit https://PAIR-code.github.io/farsight. The source code is available at https://github.com/PAIR-code/farsight

Pursuing new, commonsense approaches to education reform and work-family policies, from childcare and early education through higher education and workforce flexibility, will foster the conditions for family flourishing and increase birth rates for married couples. Affordable childcare from a variety of providers, including at-home options, access to high-quality K-12 education that reflects a family's values, a higher education landscape that provides numerous routes to upward mobility, and flexible work arrangements will give couples the confidence that they can have the number of children they desire. Public policy should support families as they form and grow, and can do so through reducing government regulatory overreach, directly funding students instead of institutions, and by removing barriers to a flexible workforce. Ultimately, these reforms will support couples as they raise their children to become productive members of a free society.

In lattice QCD the calculation of disconnected quark loops from the trace of the inverse quark matrix has large noise variance. A multilevel Monte Carlo method is proposed for this problem that uses different degree polynomials on a multilevel system. The polynomials are developed from the GMRES algorithm for solving linear equations. To reduce orthogonalization expense, the highest degree polynomial is a composite or double polynomial found with a polynomial preconditioned GMRES iteration. Matrix deflation is used in three different ways: in the Monte Carlo levels, in the main solves, and in the deflation of the highest level double polynomial. A numerical comparison with optimized Hutchinson is performed on a quenched \(24^4\) lattice. The results demonstrate that the new Multipolynomial Monte Carlo method can significantly improve the trace computation for matrices that have a difficult spectrum due to small eigenvalues.}, Comment: To be published in Proceedings of Science, 40th International Symposium on Lattice Field Theory (Lattice 2023)

Zipf (1935) posited that wordforms are optimized to minimize utterances' communicative costs. Under the assumption that cost is given by an utterance's length, he supported this claim by showing that words' lengths are inversely correlated with their frequencies. Communicative cost, however, can be operationalized in different ways. Piantadosi et al. (2011) claim that cost should be measured as the distance between an utterance's information rate and channel capacity, which we dub the channel capacity hypothesis (CCH) here. Following this logic, they then proposed that a word's length should be proportional to the expected value of its surprisal (negative log-probability in context). In this work, we show that Piantadosi et al.'s derivation does not minimize CCH's cost, but rather a lower bound, which we term CCH-lower. We propose a novel derivation, suggesting an improved way to minimize CCH's cost. Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio. Experimentally, we compare these three communicative cost functions: Zipf's, CCH-lower , and CCH. Across 13 languages and several experimental settings, we find that length is better predicted by frequency than either of the other hypotheses. In fact, when surprisal's expectation, or expectation plus variance-to-mean ratio, is estimated using better language models, it leads to worse word length predictions. We take these results as evidence that Zipf's longstanding hypothesis holds., Comment: Published at EMNLP 2023

Training on multiple modalities of input can augment the capabilities of a language model. Here, we ask whether such a training regime can improve the quality and efficiency of these systems as well. We focus on text--audio and introduce Whisbert, which is inspired by the text--image approach of FLAVA (Singh et al., 2022). In accordance with Babylm guidelines (Warstadt et al., 2023), we pretrain Whisbert on a dataset comprising only 100 million words plus their corresponding speech from the word-aligned version of the People's Speech dataset (Galvez et al., 2021). To assess the impact of multimodality, we compare versions of the model that are trained on text only and on both audio and text simultaneously. We find that while Whisbert is able to perform well on multimodal masked modeling and surpasses the Babylm baselines in most benchmark tasks, it struggles to optimize its complex objective and outperform its text-only Whisbert baseline., Comment: Published at the BabyLM Challenge, a shared task co-sponsored by CMCL 2023 and CoNLL 2023, hosted by EMNLP 2023