Search

This study investigates the internal reasoning mechanism of language models during symbolic multi-step reasoning, motivated by the question of whether chain-of-thought (CoT) outputs are faithful to the model's internals. Specifically, we inspect when they internally determine their answers, particularly before or after CoT begins, to determine whether models follow a post-hoc "think-to-talk" mode or a step-by-step "talk-to-think" mode of explanation. Through causal probing experiments in controlled arithmetic reasoning tasks, we found systematic internal reasoning patterns across models; for example, simple subproblems are solved before CoT begins, and more complicated multi-hop calculations are performed during CoT.

Evaluating the quality of free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to evaluate how LLMs rate explanations. We observed that larger models outputted labels that maintained or increased the inter-annotator agreement, suggesting that they are within the expected variance between human raters. However, their correlation with majority-voted human ratings varied across different quality aspects, indicating that they are not a complete replacement. In turn, using LLMs as a supplement to a smaller group of human raters in some cases improved the correlation with the original majority labels. However, the effect was limited to cases where human raters were scarce, and an additional human rater had a more pronounced effect in all cases. Overall, we recommend against using LLMs as a complete replacement for human raters but encourage using them in configurations that end with targeted human involvement. Data available here: https://github.com/a-brassard/ACORN, Comment: 18 pages, 7 figures, accepted to COLM 2024. Data available here: https://github.com/a-brassard/ACORN

In this paper, we describe the development of a communication support system that detects erroneous translations to facilitate crosslingual communications due to the limitations of current machine chat translation methods. We trained an error detector as the baseline of the system and constructed a new Japanese-English bilingual chat corpus, BPersona-chat, which comprises multiturn colloquial chats augmented with crowdsourced quality ratings. The error detector can serve as an encouraging foundation for more advanced erroneous translation detection systems.

Neural reasoning accuracy improves when generating intermediate reasoning steps. However, the source of this improvement is yet unclear. Here, we investigate and factorize the benefit of generating intermediate steps for symbolic reasoning. Specifically, we decompose the reasoning strategy w.r.t. step granularity and chaining strategy. With a purely symbolic numerical reasoning dataset (e.g., A=1, B=3, C=A+3, C?), we found that the choice of reasoning strategies significantly affects the performance, with the gap becoming even larger as the extrapolation length becomes longer. Surprisingly, we also found that certain configurations lead to nearly perfect performance, even in the case of length extrapolation. Our results indicate the importance of further exploring effective strategies for neural reasoning models., Comment: This paper is accepted as the findings at EACL 2023, and the earlier version (non-archival) of this work got the Best Paper Award in the Student Research Workshop of AACL 2022

Compositionality is a pivotal property of symbolic reasoning. However, how well recent neural models capture compositionality remains underexplored in the symbolic reasoning tasks. This study empirically addresses this question by systematically examining recently published pre-trained seq2seq models with a carefully controlled dataset of multi-hop arithmetic symbolic reasoning. We introduce a skill tree on compositionality in arithmetic symbolic reasoning that defines the hierarchical levels of complexity along with three compositionality dimensions: systematicity, productivity, and substitutivity. Our experiments revealed that among the three types of composition, the models struggled most with systematicity, performing poorly even with relatively simple compositions. That difficulty was not resolved even after training the models with intermediate reasoning steps., Comment: accepted by EACL 2023

In the framework of our extensive modeling study of the spectral evolution of white dwarfs, we present here a new set of detailed calculations of the transport of residual hydrogen in helium-rich white dwarfs. First, we investigate the so-called float-up process at high effective temperature, whereby the upward diffusion of trace hydrogen leads to the formation of a hydrogen atmosphere. We examine the dependence of this phenomenon on the initial hydrogen abundance and on the strength of the radiative wind that opposes gravitational settling. Combined with our empirical knowledge of spectral evolution, our simulations provide new quantitative constraints on the hydrogen content of the hot helium-dominated white dwarf population. Then, we study the outcome of the so-called convective dilution process at low effective temperature, whereby the superficial hydrogen layer is mixed within the underlying helium-rich envelope. In stark contrast with previous works on convective dilution, we demonstrate that, under reasonable assumptions, our models successfully reproduce the observed atmospheric composition of cool DBA stars, thereby solving one of the most important problems of spectral evolution theory. This major improvement is due to our self-consistent modeling of the earlier float-up process, which predicts the existence of a massive hydrogen reservoir underneath the thin superficial layer. We argue that the trace hydrogen detected at the surface of DBA white dwarfs is, in most cases, of primordial origin rather than the result of external accretion., Comment: 21 pages, 9 figures, accepted for publication in The Astrophysical Journal

Quantum key distribution (QKD) protocols aim at allowing two parties to generate a secret shared key. While many QKD protocols have been proven unconditionally secure in theory, practical security analyses of experimental QKD implementations typically do not take into account all possible loopholes, and practical devices are still not fully characterized for obtaining tight and realistic key rates. We present a simple method of computing secure key rates for any practical implementation of discrete-variable QKD (which can also apply to measurement-device-independent QKD), initially in the single-qubit lossless regime, and we rigorously prove its unconditional security against any possible attack. We hope our method becomes one of the standard tools used for analysing, benchmarking, and standardizing all practical realizations of QKD., Comment: 25 pages

Language models (LMs) have been used in cognitive modeling as well as engineering studies -- they compute information-theoretic complexity metrics that simulate humans' cognitive load during reading. This study highlights a limitation of modern neural LMs as the model of choice for this purpose: there is a discrepancy between their context access capacities and that of humans. Our results showed that constraining the LMs' context access improved their simulation of human reading behavior. We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs' context access might enhance their cognitive plausibility., Comment: Accepted by EMNLP2022 (main long)

We continue our comprehensive theoretical investigation of the spectral evolution of white dwarfs based on sophisticated simulations of element transport. In this paper, we focus on the transformation of PG 1159 stars into DO/DB white dwarfs due to the gravitational settling of heavy elements, and then into DQ white dwarfs through the convective dredge-up of carbon. We study the impact of several physical parameters on the evolution of the surface carbon abundance over a wide range of effective temperature. In the hot PG 1159 and DO phases, our calculations confirm that the temperature of the PG 1159-to-DO transition depends sensitively on the stellar mass and the wind mass-loss rate. We show that measured carbon abundances of DOZ white dwarfs are mostly accounted for by our models, with the notable exception of the coolest DOZ stars. In the cooler DB and DQ phases, the predicted atmospheric composition is strongly influenced by the stellar mass, the thickness of the envelope, the initial carbon content, the efficiency of convective overshoot, and the presence of residual hydrogen. We demonstrate that, under reasonable assumptions, our simulations reproduce very well the observed carbon abundance pattern of DQ stars, which thus allows us to constrain the extent of the overshoot region in cool helium-rich white dwarfs. We also argue that our calculations naturally explain a number of recent empirical results, such as the relative excess of low-mass DQ stars and the presence of trace hydrogen and/or carbon at the surface of most DC and DZ stars., Comment: 23 pages, 15 figures, accepted for publication in The Astrophysical Journal

We present Semi-Structured Explanations for COPA (COPA-SSE), a new crowdsourced dataset of 9,747 semi-structured, English common sense explanations for Choice of Plausible Alternatives (COPA) questions. The explanations are formatted as a set of triple-like common sense statements with ConceptNet relations but freely written concepts. This semi-structured format strikes a balance between the high quality but low coverage of structured data and the lower quality but high coverage of free-form crowdsourcing. Each explanation also includes a set of human-given quality ratings. With their familiar format, the explanations are geared towards commonsense reasoners operating on knowledge graphs and serve as a starting point for ongoing work on improving such systems. The dataset is available at https://github.com/a-brassard/copa-sse., Comment: 6 pages, 6 figures, LREC 2022. Data available at https://github.com/a-brassard/copa-sse

White dwarf stars are subject to various element transport mechanisms that can cause their surface composition to change radically as they cool, a phenomenon known as spectral evolution. In this paper, we undertake a comprehensive theoretical investigation of the spectral evolution of white dwarfs. First, we introduce STELUM, a new implementation of the stellar evolutionary code developed at the Universit\'e de Montr\'eal. We provide a thorough description of the physical content and numerical techniques of the code, covering the treatment of both stellar evolution and chemical transport. Then, we present two state-of-the-art numerical simulations of element transport in evolving white dwarfs. Atomic diffusion, convective mixing, and mass loss are considered simultaneously as time-dependent diffusive processes and are fully coupled to the cooling. We first model the PG 1159$-$DO$-$DB$-$DQ evolutionary channel: a helium-, carbon-, and oxygen-rich PG 1159 star transforms into a pure-helium DB white dwarf due to gravitational settling, and then into a helium-dominated, carbon-polluted DQ white dwarf through convective dredge-up. We also compute for the first time the full DO$-$DA$-$DC evolutionary channel: a helium-rich DO white dwarf harboring residual hydrogen becomes a pure-hydrogen DA star through the float-up process, and then a helium-dominated, hydrogen-bearing DC star due to convective mixing. We demonstrate that our results are in excellent agreement with available empirical constraints. In particular, our DO$-$DA$-$DC simulation perfectly reproduces the lower branch of the bifurcation observed in the Gaia color-magnitude diagram, which can therefore be interpreted as a signature of spectral evolution., Comment: 23 pages, 6 figures, accepted for publication in The Astrophysical Journal

DA-type white dwarfs account for 80% of all white dwarfs and represent, for most of them, the ultimate outcome of the typical evolution of low-to-intermediate mass stars. Their internal chemical stratification is strongly marked by passed, often uncertain, stellar evolution processes that occurred during the helium (core and shell) burning phases, i.e., from the horizontal branch through AGB and post-AGB stages. Pulsating white dwarfs, in particular the "cool" DA-type ZZ Ceti variables, offer an outstanding opportunity to dig into these stars by fully exploiting their asteroseismic potential. With our most recent tools dedicated to that purpose, we show that a complete cartography of the stratification of the main constituents of a white dwarf can be inferred, leading in particular to strong constraints on the C/O core structure produced by the processes mentioned above. This opens up the way toward a systematic exploration of white-dwarf internal properties., Comment: 5 pages, 3 figures, 21st European Workshop on White Dwarfs, ed. Castanheira, Campos, & Montgomery

We present the results of the asteroseismic analysis of the hydrogen-deficient white dwarf PG 0112+104 from the $Kepler$-2 field. Our seismic procedure using the forward method based on physically sound, static models, includes the new core parameterization leading us to reproduce the periods of this star near the precision of the observations. This new fit outperforms current state-of-the-art standards by order of magnitudes. We precisely establish the internal structure and unravel the inner C/O stratification of its core. This opens up interesting perspectives on better constraining key processes in stellar physics such as nuclear burning, convection, and mixing, that shape this stratification over time., Comment: 4 pages, 3 figures, 21st European Workshop on White Dwarfs, ed. Castanheira, Vanderbosch, & Montgomery

Improving model generalization on held-out data is one of the core objectives in commonsense reasoning. Recent work has shown that models trained on the dataset with superficial cues tend to perform well on the easy test set with superficial cues but perform poorly on the hard test set without superficial cues. Previous approaches have resorted to manual methods of encouraging models not to overfit to superficial cues. While some of the methods have improved performance on hard instances, they also lead to degraded performance on easy instances. Here, we propose to explicitly learn a model that does well on both the easy test set with superficial cues and hard test set without superficial cues. Using a meta-learning objective, we learn such a model that improves performance on both the easy test set and the hard test set. By evaluating our models on Choice of Plausible Alternatives (COPA) and Commonsense Explanation, we show that our proposed method leads to improved performance on both the easy test set and the hard test set upon which we observe up to 16.5 percentage points improvement over the baseline.

We experimentally and theoretically study impacts into dense cornstarch and water suspensions. We vary impact speed as well as intruder size, shape, and mass, and we characterize the resulting dynamics using high-speed video and an onboard accelerometer. We numerically solve previously proposed models, most notably the added-mass model as well as a class of {viscous-like} models. In the {viscous-like models}, the intruder dynamics are dominated by {large, viscous-like forces} at the boundary of the jammed front {where large shear rates and accompanying large viscosities are present.} We find that our experimental data are consistent with this class of models and inconsistent with the added mass model. Our results strongly suggest that the added-mass model, which is the dominant model for understanding the dynamics of impact into shear-thickening dense suspensions, should be updated to include these viscous-like forces.

The objective of this work was to assess the clinical performance of an unsupervised machine learning model aimed at identifying unusual medication orders and pharmacological profiles. We conducted a prospective study between April 2020 and August 2020 where 25 clinical pharmacists dichotomously (typical or atypical) rated 12,471 medication orders and 1,356 pharmacological profiles. Based on AUPR, performance was poor for orders, but satisfactory for profiles. Pharmacists considered the model a useful screening tool., Comment: Machine Learning for Health (ML4H) at NeurIPS 2020 - Extended Abstract

It is shown that under reasonable assumptions a Drake-style equation can be obtained for the probability that our universe is the result of a deliberate simulation. Evaluating loose bounds for certain terms in the equation shows that the probability is unlikely to be as high as previously reported in the literature, especially in a scenario where the simulations are recursive. Furthermore, we investigate the possibility of eavesdropping from the outside of such a simulation and introduce a general attack that can circumvent attempts at quantum cryptography inside the simulation, even if the quantum properties of the simulation are genuine., Comment: 17 pages, 2 figures, 1 table

As they evolve, white dwarfs undergo major changes in surface composition, a phenomenon known as spectral evolution. In particular, some stars enter the cooling sequence with helium atmospheres (type DO) but eventually develop hydrogen atmospheres (type DA), most likely through the upward diffusion of residual hydrogen. Our empirical knowledge of this process remains scarce: the fractions of white dwarfs that are born helium-rich and that experience the DO-to-DA transformation are poorly constrained. We tackle this issue by performing a detailed model-atmosphere investigation of 1806 hot ($T_{\rm eff} \ge 30,000$ K) white dwarfs observed spectroscopically by the Sloan Digital Sky Survey. We first introduce our new generations of model atmospheres and theoretical cooling tracks, both appropriate for hot white dwarfs. We then present our spectroscopic analysis, from which we determine the atmospheric and stellar parameters of our sample objects. We find that $\sim$24% of white dwarfs begin their degenerate life as DO stars, among which $\sim$2/3 later become DA stars. We also infer that the DO-to-DA transition occurs at substantially different temperatures ($75,000 \ {\rm K} > T_{\rm eff} > 30,000$ K) for different objects, implying a broad range of hydrogen content within the DO population. Furthermore, we identify 127 hybrid white dwarfs, including 31 showing evidence of chemical stratification, and we discuss how these stars fit in our understanding of the spectral evolution. Finally, we uncover significant problems in the spectroscopic mass scale of very hot ($T_{\rm eff} > 60,000$ K) white dwarfs., Comment: 33 pages (plus a 49-page table), 20 figures, accepted for publication in The Astrophysical Journal

The continuous cooling of a white dwarf is punctuated by events that affect its cooling rate. Probably the most significant of those is the crystallization of its core, a phase transition that occurs once the C/O interior has cooled down below a critical temperature. This transition releases latent heat as well as gravitational energy due to the redistribution of the C and O ions during solidification, thereby slowing down the evolution of the white dwarf. The unambiguous observational signature of core crystallization - a pile-up of objects in the cooling sequence - was recently reported. However, existing evolution models struggle to quantitatively reproduce this signature, casting doubt on their accuracy when used to measure the ages of stellar populations. The timing and amount of the energy released during crystallization depend on the exact form of the C/O phase diagram. Using the advanced Gibbs-Duhem integration method and state-of-the-art Monte Carlo simulations of the solid and liquid phases, we have obtained a very accurate version of this phase diagram, allowing a precise modeling of the phase transition. Despite this improvement, the magnitude of the crystallization pile-up remains underestimated by current evolution models. We conclude that latent heat release and O sedimentation alone are not sufficient to explain the observations and that other unaccounted physical mechanisms, possibly $^{22}$Ne phase separation, play an important role., Comment: 4 pages, 2 figures. Accepted for publication in A&A Letters