Search

To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training., Comment: accepted to Conference on Natural Language Learning 2024 (https://www.conll.org/)

From CMB polarization data alone we reconstruct the CMB lensing power spectrum, comparable in overall constraining power to previous temperature-based reconstructions, and an unlensed E-mode power spectrum. The observations, taken in 2019 and 2020 with the South Pole Telescope (SPT) and the SPT-3G camera, cover 1500 deg$^2$ at 95, 150, and 220 GHz with arcminute resolution and roughly 4.9$\mu$K-arcmin coadded noise in polarization. The power spectrum estimates, together with systematic parameter estimates and a joint covariance matrix, follow from a Bayesian analysis using the Marginal Unbiased Score Expansion (MUSE) method. The E-mode spectrum at $\ell>2000$ and lensing spectrum at $L>350$ are the most precise to date. Assuming the $\Lambda$CDM model, and using only these SPT data and priors on $\tau$ and absolute calibration from Planck, we find $H_0=66.81\pm0.81$ km/s/Mpc, comparable in precision to the Planck determination and in 5.4$\sigma$ tension with the most precise $H_0$ inference derived via the distance ladder. We also find $S_8=0.850\pm0.017$, providing further independent evidence of a slight tension with low-redshift structure probes. The $\Lambda$CDM model provides a good simultaneous fit to the combined Planck, ACT, and SPT data, and thus passes a powerful test. Combining these CMB datasets with BAO observations, we find that the effective number of neutrino species, spatial curvature, and primordial helium fraction are consistent with standard model values, and that the 95% confidence upper limit on the neutrino mass sum is 0.075 eV. The SPT data are consistent with the somewhat weak preference for excess lensing power seen in Planck and ACT data relative to predictions of the $\Lambda$CDM model. We also detect at greater than 3$\sigma$ the influence of non-linear evolution in the CMB lensing power spectrum and discuss it in the context of the $S_8$ tension.(abridged), Comment: 28 pages, 21 figures + appendices

The detection of satellite thermal emission at millimeter wavelengths is presented using data from the 3rd-Generation receiver on the South Pole Telescope (SPT-3G). This represents the first reported detection of thermal emission from artificial satellites at millimeter wavelengths. Satellite thermal emission is shown to be detectable at high signal-to-noise on timescales as short as a few tens of milliseconds. An algorithm for downloading orbital information and tracking known satellites given observer constraints and time-ordered observatory pointing is described. Consequences for cosmological surveys and short-duration transient searches are discussed, revealing that the integrated thermal emission from all large satellites does not contribute significantly to the SPT-3G survey intensity map. Measured satellite positions are found to be discrepant from their two-line element (TLE) derived ephemerides up to several arcminutes which may present a difficulty in cross-checking or masking satellites from short-duration transient searches.

We investigate zero-shot cross-lingual news sentiment detection, aiming to develop robust sentiment classifiers that can be deployed across multiple languages without target-language training data. We introduce novel evaluation datasets in several less-resourced languages, and experiment with a range of approaches including the use of machine translation; in-context learning with large language models; and various intermediate training regimes including a novel task objective, POA, that leverages paragraph-level information. Our results demonstrate significant improvements over the state of the art, with in-context learning generally giving the best performance, but with the novel POA approach giving a competitive alternative with much lower computational overhead. We also show that language similarity is not in itself sufficient for predicting the success of cross-lingual transfer, but that similarity in semantic content and structure can be equally important., Comment: The first two authors share equal contribution

Despite being operational for only a short time, the Einstein Probe mission has already significantly advanced the study of rapid variability in the soft X-ray sky. We report the discovery of luminous and variable radio emission from the Einstein Probe fast X-ray transient EP240414a, the second such source with a radio counterpart. The radio emission at $3\,\rm{GHz}$ peaks at $\sim30$ days post explosion and with a spectral luminosity $\sim2\times10^{30}\,\rm{erg}\,\rm{s}^{-1}\,\rm{Hz}^{-1}$, similar to what is seen from long gamma-ray bursts, and distinct from other extra-galactic transients including supernovae and tidal disruption events, although we cannot completely rule out emission from engine driven stellar explosions e.g. the fast blue optical transients. An equipartition analysis of our radio data reveals that an outflow with at least a moderate bulk Lorentz factor ($\Gamma\gtrsim1.6$) with a minimum energy of $\sim10^{48}\,\rm{erg}$ is required to explain our observations. The apparent lack of reported gamma-ray counterpart to EP240414a could suggest that an off-axis or choked jet could be responsible for the radio emission, although a low luminosity gamma-ray burst may have gone undetected. Our observations are consistent with the hypothesis that a significant fraction of extragalactic fast X-ray transients are associated with the deaths of massive stars., Comment: 11 pages, 6 figures, submitted to ApJL

Planet-planet occultations (PPOs) occur when one exoplanet occults another exoplanet in the same system as seen from the Earth's vantage point. PPOs may provide a unique opportunity to observe radio "spillover" from extraterrestrial intelligences' (ETIs) radio transmissions or radar being transmitted from the further exoplanet towards the nearer one for the purposes of communication or scientific exploration. Planetary systems with many tightly packed, low-inclination planets, such as TRAPPIST-1, are predicted to have frequent PPOs. Here, the narrowband technosignature search code turboSETI was used in combination with the newly developed NbeamAnalysis filtering pipeline to analyze 28 hours of beamformed data taken with the Allen Telescope Array (ATA) during late October and early November 2022, from 0.9--9.3~GHz, targeting TRAPPIST-1. During this observing window, 7 possible PPO events were predicted using the NbodyGradient code. The filtering pipeline reduced the original list of 25 million candidate signals down to 6 million by rejecting signals that were not sky-localized and, from these, identified a final list of 11127 candidate signals above a power law cutoff designed to segregate signals by their attenuation and morphological similarity between beams. All signals were plotted for visual inspection, 2264 of which were found to occur during PPO windows. We report no detection of signals of non-human origin, with upper limits calculated for each PPO event exceeding EIRPs of 2.17--13.3 TW for minimally drifting signals and 40.8--421 TW in the maximally drifting case. This work constitutes the longest single-target radio SETI search of TRAPPIST-1 to date.

Introducing superconductivity in topological materials can lead to innovative electronic phases and device functionalities. Here, we present a new strategy for quantum engineering of superconducting junctions in moire materials through direct, on-chip, and fully encapsulated 2D crystal growth. We achieve robust and designable superconductivity in Pd-metalized twisted bilayer molybdenum ditelluride (MoTe2) and observe anomalous superconducting effects in high-quality junctions across ~ 20 moire cells. Surprisingly, the junction develops enhanced, instead of weakened, superconducting behaviors, exhibiting fluctuations to a higher critical magnetic field compared to its adjacent Pd7MoTe2 superconductor. Additionally, the critical current further exhibits a striking V-shaped minimum at zero magnetic field. These features are unexpected in conventional Josephson junctions and indeed absent in junctions of natural bilayer MoTe2 created using the same approach. We discuss implications of these observations, including the possible formation of mixed even- and odd-parity superconductivity at the moire junctions. Our results also demonstrate a pathway to engineer and investigate superconductivity in fractional Chern insulators., Comment: 25 pages, 5 main article figures and 10 supplementary figures

We present radio observations of the long-duration gamma-ray burst (GRB) 221009A which has become known to the community as the Brightest Of All Time or the BOAT. Our observations span the first 475 days post-burst and three orders of magnitude in observing frequency, from 0.15 to 230GHz. By combining our new observations with those available in the literature, we have the most detailed radio data set in terms of cadence and spectral coverage of any GRB to date, which we use to explore the spectral and temporal evolution of the afterglow. By testing a series of phenomenological models, we find that three separate synchrotron components best explain the afterglow. The high temporal and spectral resolution allows us to conclude that standard analytical afterglow models are unable to explain the observed evolution of GRB 221009A. We explore where the discrepancies between the observations and the models are most significant and place our findings in the context of the most well-studied GRB radio afterglows to date. Our observations are best explained by three synchrotron emitting regions which we interpret as a forward shock, a reverse shock and an additional shock potentially from a cocoon or wider outflow. Finally, we find that our observations do not show any evidence of any late-time spectral or temporal changes that could result from a jet break but note that any lateral structure could significantly affect a jet break signature., Comment: 15 pages, 8 figures. Accepted to MNRAS

Large semantic knowledge bases are grounded in factual knowledge. However, recent approaches to dense text representations (i.e. embeddings) do not efficiently exploit these resources. Dense and robust representations of documents are essential for effectively solving downstream classification and retrieval tasks. This work demonstrates that injecting embedded information from knowledge bases can augment the performance of contemporary Large Language Model (LLM)-based representations for the task of text classification. Further, by considering automated machine learning (AutoML) with the fused representation space, we demonstrate it is possible to improve classification accuracy even if we use low-dimensional projections of the original representation space obtained via efficient matrix factorization. This result shows that significantly faster classifiers can be achieved with minimal or no loss in predictive performance, as demonstrated using five strong LLM baselines on six diverse real-life datasets. The code is freely available at \url{https://github.com/bkolosk1/bablfusion.git}., Comment: Accepted at the 2024 Discovery Science Conference, oral presentation track

We present the detection and characterization of fluctuations in linearly polarized emission from the atmosphere above the South Pole. These measurements make use of Austral winter survey data from the SPT-3G receiver on the South Pole Telescope in three frequency bands centered at 95, 150, and 220 GHz. We use the cross-correlation between detectors to produce an unbiased estimate of the power in Stokes I, Q, and U parameters on large angular scales. Our results are consistent with the polarized signal being produced by the combination of Rayleigh scattering of thermal radiation from the ground and thermal emission from a population of horizontally aligned ice crystals with an anisotropic distribution described by Kolmogorov turbulence. The signal is most significant at large angular scales, high observing frequency, and low elevation angle. Polarized atmospheric emission has the potential to significantly impact observations on the large angular scales being targeted by searches for inflationary B-mode CMB polarization. We present the distribution of measured angular power spectrum amplitudes in Stokes Q and I for 4 years of winter observations, which can be used to simulate the impact of atmospheric polarization and intensity fluctuations at the South Pole on a specified experiment and observation strategy. For the SPT-3G data, downweighting the small fraction of significantly contaminated observations is an effective mitigation strategy. In addition, we present a strategy for further improving sensitivity on large angular scales where maps made in the 220 GHz band are used to measure and subtract the polarized atmosphere signal from the 150 GHz band maps. In observations with the SPT-3G instrument at the South Pole, the polarized atmospheric signal is a well-understood and sub-dominant contribution to the measured noise after implementing the mitigation strategies described here., Comment: 32 pages, 28 figures

Clays are a broad class of ubiquitous layered materials. Their specific chemophysical properties are intimately connected to their molecular structure, featuring repeating patterns broken by substitutions. Molecular dynamics simulations can provide insight into the mechanisms leading to the emergent properties of these layered materials, however up to now idealised clay structures have been simulated to make the modelling process tractable. We present ClayCode, software facilitating the modelling of clay systems closely resembling experimentally determined structures. By comparing a realistic model to a commonly used montmorillonite clay model, we demonstrate that idealised models feature noticeably different ionic adsorption patterns. We then present an application of ClayCode to the study the competitive barium and sodium adsorption on Wyoming montmorillonite, Georgia kaolinite, and Montana illite, of interest in the context of nuclear waste disposal.

The growing impact of climate change on coastal areas, particularly active but fragile regions, necessitates collaboration among diverse stakeholders and disciplines to formulate effective environmental protection policies. We introduce a novel specialized corpus comprising 2,491 sentences from 410 scientific abstracts concerning coastal areas, for the Automatic Term Extraction (ATE) and Classification (ATC) tasks. Inspired by the ARDI framework, focused on the identification of Actors, Resources, Dynamics and Interactions, we automatically extract domain terms and their distinct roles in the functioning of coastal systems by leveraging monolingual and multilingual transformer models. The evaluation demonstrates consistent results, achieving an F1 score of approximately 80\% for automated term extraction and F1 of 70\% for extracting terms and their labels. These findings are promising and signify an initial step towards the development of a specialized Knowledge Base dedicated to coastal areas.

Dehumanisation involves the perception and or treatment of a social group's members as less than human. This phenomenon is rarely addressed with computational linguistic techniques. We adapt a recently proposed approach for English, making it easier to transfer to other languages and to evaluate, introducing a new sentiment resource, the use of zero-shot cross-lingual valence and arousal detection, and a new method for statistical significance testing. We then apply it to study attitudes to migration expressed in Slovene newspapers, to examine changes in the Slovene discourse on migration between the 2015-16 migration crisis following the war in Syria and the 2022-23 period following the war in Ukraine. We find that while this discourse became more negative and more intense over time, it is less dehumanising when specifically addressing Ukrainian migrants compared to others., Comment: The first authors have contributted equally. Accepted at LREC-COLING

For assessing various performance indicators of companies, the focus is shifting from strictly financial (quantitative) publicly disclosed information to qualitative (textual) information. This textual data can provide valuable weak signals, for example through stylistic features, which can complement the quantitative data on financial performance or on Environmental, Social and Governance (ESG) criteria. In this work, we use various multi-task learning methods for financial text classification with the focus on financial sentiment, objectivity, forward-looking sentence prediction and ESG-content detection. We propose different methods to combine the information extracted from training jointly on different tasks; our best-performing method highlights the positive effect of explicitly adding auxiliary task predictions as features for the final target task during the multi-task training. Next, we use these classifiers to extract textual features from annual reports of FTSE350 companies and investigate the link between ESG quantitative scores and these features., Comment: Accepted at MIDAS Workshop at ECML-PKDD 2022

In this paper, we focus on the detection of semantic changes in Slovene, a less resourced Slavic language with two million speakers. Detecting and tracking semantic changes provides insights into the evolution of the language caused by changes in society and culture. Recently, several systems have been proposed to aid in this study, but all depend on manually annotated gold standard datasets for evaluation. In this paper, we present the first Slovene dataset for evaluating semantic change detection systems, which contains aggregated semantic change scores for 104 target words obtained from more than 3000 manually annotated sentence pairs. We evaluate several existing semantic change detection methods on this dataset and also propose a novel approach based on optimal transport that improves on the existing state-of-the-art systems with an error reduction rate of 22.8%.

Software supply chain attacks, which exploit the build process or artifacts used in the process of building a software product, are increasingly of concern. To combat these attacks, one must be able to check that every artifact that a software product depends on does not contain vulnerabilities. In this paper, we introduce OmniBOR, (Universal Bill of Receipts) a minimalistic scheme for build tools to create an artifact dependency graph which can be used to track every software artifact incorporated into a built software product. We present the architecture of OmniBOR, the underlying data representations, and two implementations that produce OmniBOR data and embed an OmniBOR Identifier into built software, including a compiler-based approach and one based on tracing the build process. We demonstrate the efficacy of this approach on benchmarks including a Linux distribution for applications such as Common Vulnerabilities and Exposures (CVE) detection and software bill of materials (SBOM) computation.

In an era marked by a rapid increase in scientific publications, researchers grapple with the challenge of keeping pace with field-specific advances. We present the `AHAM' methodology and a metric that guides the domain-specific \textbf{adapt}ation of the BERTopic topic modeling framework to improve scientific text analysis. By utilizing the LLaMa2 generative language model, we generate topic definitions via one-shot learning by crafting prompts with the \textbf{help} of domain experts to guide the LLM for literature mining by \textbf{asking} it to model the topic names. For inter-topic similarity evaluation, we leverage metrics from language generation and translation processes to assess lexical and semantic similarity of the generated topics. Our system aims to reduce both the ratio of outlier topics to the total number of topics and the similarity between topic definitions. The methodology has been assessed on a newly gathered corpus of scientific papers on literature-based discovery. Through rigorous evaluation by domain experts, AHAM has been validated as effective in uncovering intriguing and novel insights within broad research areas. We explore the impact of domain adaptation of sentence-transformers for the task of topic \textbf{model}ing using two datasets, each specialized to specific scientific domains within arXiv and medarxiv. We evaluate the impact of data size, the niche of adaptation, and the importance of domain adaptation. Our results suggest a strong interaction between domain adaptation and topic modeling precision in terms of outliers and topic definitions., Comment: Submitted to IDA 2024

FRB 20220912A is a repeating Fast Radio Burst (FRB) that was discovered in Fall 2022 and remained highly active for several months. We report the detection of 35 FRBs from 541 hours of follow-up observations of this source using the recently refurbished Allen Telescope Array, covering 1344 MHz of bandwidth primarily centered at 1572 MHz. All 35 FRBs were detected in the lower half of the band with non-detections in the upper half and covered fluences from 4-431 Jy-ms (median$=$48.27 Jy-ms). We find consistency with previous repeater studies for a range of spectrotemporal features including: bursts with downward frequency drifting over time; a positive correlation between bandwidth and center frequency; and a decrease in sub-burst duration over time. We report an apparent decrease in the center frequency of observed bursts over the 2 months of the observing campaign (corresponding to a drop of $6.21\pm 0.76$ MHz per day). We predict a cut-off fluence for FRB 20220912A of $F_\textrm{max}\lesssim 10^4$ Jy-ms, for this source to be consistent with the all-sky rate, and find that FRB 20220912A significantly contributed to the all-sky FRB rate at a level of a few percent for fluences of $\sim$100 Jy-ms. Finally, we investigate characteristic timescales and sub-burst periodicities and find a) a median inter-subburst timescale of 5.82$\pm$1.16 ms in the multi-component bursts and b) no evidence of strict periodicity even in the most evenly-spaced multi-component burst in the sample. Our results demonstrate the importance of wideband observations of FRBs, and provide an important set of observational parameters against which to compare FRB progenitor and emission mechanism models., Comment: 17 pages, 8 figures, 2 tables, accepted to MNRAS

Image-to-Image translation (I2I) is a subtype of Machine Learning (ML) that has tremendous potential in applications where two domains of images and the need for translation between the two exist, such as the removal of fog. For example, this could be useful for autonomous vehicles, which currently struggle with adverse weather conditions like fog. However, datasets for I2I tasks are not abundant and typically hard to acquire. Here, we introduce STEREOFOG, a dataset comprised of $10,067$ paired fogged and clear images, captured using a custom-built device, with the purpose of exploring I2I's potential in this domain. It is the only real-world dataset of this kind to the best of our knowledge. Furthermore, we apply and optimize the pix2pix I2I ML framework to this dataset. With the final model achieving an average Complex Wavelet-Structural Similarity (CW-SSIM) score of $0.76$, we prove the technique's suitability for the problem., Comment: 7 pages, 7 figures, for associated dataset and Supplement file, see https://github.com/apoll2000/stereofog

The central topic of this letter is to show that light-matter hybridization not only gives rise to novel dynamic responses but can also modify intermolecular interactions and induce new structural order. Using the van der Waals (vdW) system in an optical cavity as an example, we predict the effects of interference and collectivity in cavity-induced many-body dispersion interactions. Specifically, the leading order correction due to cavity-induced quantum fluctuations leads to 3-body and 4-body vdW interactions, which can align intermolecular vectors and are not pairwise additive. In addition, the cavity-induced dipole leads to a single-molecule energy shift that aligns individual molecules, and a pair-wise interaction that scales as $R^{-3}$ instead of the standard $R^{-6}$ distance scaling. The coefficients of all these cavity-induced corrections depend on the cavity frequency and are renormalized by the effective Rabi frequency, which in turn depends on the particle density. Finally, we study the interaction of the vdW system in a cavity with an external object and find a significant enhancement in the interaction range due to modified distance scaling laws. These theoretical predictions suggest the possibility of cavity-induced nematic or smectic order and may provide an essential clue to understanding intriguing phenomena observed in optical cavities, such as strongly-modified ground-state reactivity, ion transport and solvent polarity.