Search

Large Language Models, despite their significant capabilities, are known to fail in surprising and unpredictable ways. Evaluating their true `understanding' of language is particularly challenging due to the extensive web-scale data they are trained on. Therefore, we construct an evaluation to systematically assess natural language understanding (NLU) in LLMs by leveraging Construction Grammar (CxG), which provides insights into the meaning captured by linguistic elements known as constructions (Cxns). CxG is well-suited for this purpose because provides a theoretical basis to construct targeted evaluation sets. These datasets are carefully constructed to include examples which are unlikely to appear in pre-training data, yet intuitive and easy for humans to understand, enabling a more targeted and reliable assessment. Our experiments focus on downstream natural language inference and reasoning tasks by comparing LLMs' understanding of the underlying meanings communicated through 8 unique Cxns with that of humans. The results show that while LLMs demonstrate some knowledge of constructional information, even the latest models including GPT-o1 struggle with abstract meanings conveyed by these Cxns, as demonstrated in cases where test sentences are dissimilar to their pre-training data. We argue that such cases provide a more accurate test of true language understanding, highlighting key limitations in LLMs' semantic capabilities. We make our novel dataset and associated experimental data including prompts and model responses publicly available.

Work on shallow discourse parsing in English has focused on the Wall Street Journal corpus, the only large-scale dataset for the language in the PDTB framework. However, the data is not openly available, is restricted to the news domain, and is by now 35 years old. In this paper, we present and evaluate a new open-access, multi-genre benchmark for PDTB-style shallow discourse parsing, based on the existing UD English GUM corpus, for which discourse relation annotations in other frameworks already exist. In a series of experiments on cross-domain relation classification, we show that while our dataset is compatible with PDTB, substantial out-of-domain degradation is observed, which can be alleviated by joint training on both datasets., Comment: Accepted to EMNLP 2024 (main, long); camera-ready version

The Universal Dependencies (UD) project has created an invaluable collection of treebanks with contributions in over 140 languages. However, the UD annotations do not tell the full story. Grammatical constructions that convey meaning through a particular combination of several morphosyntactic elements -- for example, interrogative sentences with special markers and/or word orders -- are not labeled holistically. We argue for (i) augmenting UD annotations with a 'UCxn' annotation layer for such meaning-bearing grammatical constructions, and (ii) approaching this in a typologically informed way so that morphosyntactic strategies can be compared across languages. As a case study, we consider five construction families in ten languages, identifying instances of each construction in UD treebanks through the use of morphosyntactic patterns. In addition to findings regarding these particular constructions, our study yields important insights on methodology for describing and identifying constructions in language-general and language-particular ways, and lays the foundation for future constructional enrichment of UD treebanks., Comment: LREC-COLING 2024

The Empirical Valence Bond (EVB) method offers a suitable framework to obtain reactive potentials through the coupling of non-reactive force fields. However, most of the implemented functional forms for the coupling terms depend on complex spatial coordinates, which precludes the computation of the stress tensor for condensed phase systems and prevents the possibility to carry out EVB molecular dynamics in the isothermal-isobaric (NPT) ensemble. In this work, we make use of coupling terms that depend on the energy gaps, defined as the energy differences between the participating non-reactive force fields, and derive an expression for the EVB stress tensor suitable for computations. Implementation of this new methodology is tested for a model of a single reactive malonaldehyde solvated in non-reactive water. Computed densities and classical probability distributions in the NPT ensemble reveals a negligible role of the reactive potential in the limit of low concentrated solutions, thus corroborating the validity of standard approximations customarily adopted for EVB simulations.

We present a simplified density functional theory (DFT) method to com- pute vertical electron and hole attachment energies to frontier orbitals of molecules absorbed on insulating films supported by a metal substrate. The adsorbate and the film is treated fully within DFT, whereas the metal is treated implicitly by a perfect conductor model. As illustrated for a pentacene molecule adsorbed on NaCl films sup- ported by a Cu substrate, we find that the computed energy gap between the highest and lowest occupied molecular orbitals - HOMO and LUMO -from the vertical attach- ment energies increases with the thickness of the insulating film, in agreement with experiments. This increase of the gap can be rationalized in a simple dielectric model with parameters determined from DFT calculations and is found to be dominated by the image interaction with the metal. However, this model overestimates the down- ward shift of the energy gap in the limit of an infinitely thick film. This work provides a new and efficient strategy to extend the use of density functional theory to the study of charging and discharging of large molecular absorbates on insulating films supported by a metal substrate.

Noncontact atomic force microscopy (NC-AFM) and Kelvin probe force microscopy (KPFM) have become important tools for nanotechnology; however, their contrast mechanisms on the atomic scale are not entirely understood. Here we used chlorine vacancies in NaCl bilayers on Cu(111) as a model system to investigate atomic contrast as a function of applied voltage, tip height, and tip functionalization. We demonstrate that the AFM contrast on the atomic scale decisively depends on both the tip termination and the sample voltage. On the contrary, the local contact potential difference (LCPD) acquired with KPFM showed the same qualitative contrast for all tip terminations investigated, which resembled the contrast of the electric field of the sample. We find that the AFM contrast stems mainly from electrostatic interactions but its tip dependence cannot be explained by the tip dipole alone. With the aid of a simple electrostatic model and by density functional theory we investigate the underlying contrast mechanisms., Comment: 14 pages, 12 figures

A simplified density functional theory (DFT) method for charged adsorbates on an ultrathin, insulating film supported by a metal substrate is developed and presented. This new method is based on a previous DFT development that uses a perfect conductor (PC) model to approximate the electrostatic response of the metal substrate, while the film and the adsorbate are both treated fully within DFT [I. Scivetti and M. Persson, Journal of Physics: Condensed Matter 25, 355006 (2013)]. The missing interactions between the metal substrate and the insulating film in the PC approximation are modelled by a simple force field (FF). The parameters of the PC model and the force field are obtained from DFT calculations of the film and the substrate, here shown explicitly for a NaCl bilayer supported by a Cu(100) surface. In order to obtain some of these parameters and the polarisability of the force field, we have to include an external, uniformly charged plane in the DFT calculations, which has required the development of a periodic DFT formalism to include such a charged plane in the presence of a metal substrate. This extension and implementation should be of more general interest and applicable to other challenging problems, for instance, in electrochemistry. As illustrated for the gold atom on the NaCl bilayer supported by a Cu(100) surface, our new DFT-PC-FF method allows us to handle different charge states of adsorbates in a controlled and accurate manner with a considerable reduction of the computational time. In addition, it is now possible to calculate vertical transition and reorganisation energies for charging and discharging of adsorbates that cannot be obtained by current DFT methodologies that include the metal substrate. We find that the computed vertical transition energy for charging of the gold adatom is in good agreement with experiments.

We study the structural effects produced by the quantization of vibrational degrees of freedom in periodic crystals at zero temperature. To this end we introduce a methodology based on mapping a suitable subspace of the vibrational manifold and solving the Schroedinger equation in it. A number of increasingly accurate approximations ranging from the quasi-harmonic approximation (QHA) to the vibrational self-consistent field (VSCF) method and the exact solution are described. A thorough analysis of the approximations is presented for model monoatomic and hydrogen-bonded chains, and results are presented for a linear HF chain where the potential energy surface is obtained via first-principles electronic structure calculations. We focus on quantum nuclear effects on the lattice constant, and show that the VSCF is an excellent approximation, meaning that correlation between modes is not extremely important. The QHA is excellent for covalently-bonded, mildly anharmonic systems, but it fails for hydrogen-bonded ones. In the latter, the zero-point energy exhibits a non-analytic behavior at the lattice constant where the H-atoms center, which leads to a spurious secondary minimum in the quantum-corrected energy curve. An inexpensive anharmonic appoximation of non-interacting modes appears to produce rather good results for hydrogen-bonded chains, for small system sizes. However, it converges to the incorrect QHA results for increasing size. Isotope effects are studied for the first-principles HF chain. We show how the lattice constant and the HF distance increase with decreasing mass, and how the QHA proves to be insufficient to reproduce this behavior., Comment: 13 pages, 12 figures. To appear in Phys. Rev. B