Search

Does inverse scaling only occur as a function of model parameter size, or can it also occur over the course of training? We carry out an exploratory study investigating whether, over the course of training on the language modeling task, the performance of language models at specific tasks can decrease while general performance remains high. We find that for two tasks from the Inverse Scaling Challenge - quote-repetition and redefine-math - this is indeed the case. Specifically, we find that for Pythia (Biderman et al., 2023) models with a higher number of parameters, performance decreases over the course of training at these two tasks, despite these models showing standard (positive) scaling overall. This highlights the importance of testing model performance at all relevant benchmarks any time they are trained on additional data, even if their overall performance improves.

Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about what large language models can and cannot do., 31 pages

Context changes expectations about upcoming words - following a story involving an anthropomorphic peanut, comprehenders expect the sentence the peanut was in love more than the peanut was salted, as indexed by N400 amplitude (Nieuwland & van Berkum, 2006). This updating of expectations has been explained using Situation Models - mental representations of a described event. However, recent work showing that N400 amplitude is predictable from distributional information alone raises the question whether situation models are necessary for these contextual effects. We model the results of Nieuwland and van Berkum (2006) using six computational language models and three sets of word vectors, none of which have explicit situation models or semantic grounding. We find that a subset of these can fully model the effect found by Nieuwland and van Berkum (2006). Thus, at least some processing effects normally explained through situation models may not in fact require explicit situation models., Accepted at CogSci 2023

Most large language models are trained on linguistic input alone, yet humans appear to ground their understanding of words in sensorimotor experience. A natural solution is to augment LM representations with human judgments of a word's sensorimotor associations (e.g., the Lancaster Sensorimotor Norms), but this raises another challenge: most words are ambiguous, and judgments of words in isolation fail to account for this multiplicity of meaning (e.g., "wooden table" vs. "data table"). We attempted to address this problem by building a new lexical resource of contextualized sensorimotor judgments for 112 English words, each rated in four different contexts (448 sentences total). We show that these ratings encode overlapping but distinct information from the Lancaster Sensorimotor Norms, and that they also predict other measures of interest (e.g., relatedness), above and beyond measures derived from BERT. Beyond shedding light on theoretical questions, we suggest that these ratings could be of use as a "challenge set" for researchers building grounded language models.

Humans can attribute beliefs to others. However, it is unknown to what extent this ability results from an innate biological endowment or from experience accrued through child development, particularly exposure to language describing others' mental states. We test the viability of the language exposure hypothesis by assessing whether models exposed to large quantities of human language display sensitivity to the implied knowledge states of characters in written passages. In pre-registered analyses, we present a linguistic version of the False Belief Task to both human participants and a Large Language Model, GPT-3. Both are sensitive to others' beliefs, but while the language model significantly exceeds chance behavior, it does not perform as well as the humans, nor does it explain the full extent of their behavior -- despite being exposed to more language than a human would in a lifetime. This suggests that while statistical learning from language exposure may in part explain how humans develop the ability to reason about the mental states of others, other mechanisms are also responsible.

We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages. The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies. Shifting representations by language means is sufficient to induce token predictions in different languages. However, we also identify stable language-neutral axes that encode information such as token positions and part-of-speech. We visualize representations projected onto language-sensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information. These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning., Comment: Accepted to EMNLP 2022

Supplemental material, sj-docx-1-las-10.1177_00238309221087715 for The Role of Prosody in Disambiguating English Indirect Requests by Sean Trott, Stefanie Reed, Dan Kaliblotzky, Victor Ferreira and Benjamin Bergen in Language and Speech

Can we know a word by the company it keeps? Aspects of meaning that concern physical interactions might be particularly difficult to learn from language alone. Glenberg & Robertson (2000) found that although human comprehenders were sensitive to the distinction between afforded and nonafforded actions, distributional semantic models were not. We tested whether technological advances have made distributional models more sensitive to affordances by replicating their experiment with modern Neural Language Models (NLMs). We found that only one NLM (GPT-3) was sensitive to the affordedness of actions. Moreover, GPT-3 accounted for only one third of the effect of affordedness on human sensibility judgments. These results imply that people use processes that go beyond distributional statistics to understand linguistic expressions, and that NLP systems may need to be augmented with such capabilities.

Previous work has found competing evidence for how contextual diversity influences early word learning. In support of contextual diversity facilitating learning, the corpus-derived diversity metric from Hills, Maouene, Riordan, and Smith (2010) was found to correlate with earlier ages of acquisition in children. We extend this work to five languages, accounting for a nonlinear relationship between the raw contextual diversity metric and word log-frequencies, and we account for additional covariates such as word length, concreteness, and lexical class. In contrast with the original result, we find that contextually diverse words are acquired later by children across languages. Our findings support the hypothesis that contextual diversity introduces an excess of possible meanings for contextually diverse words, adding noise to the word learning process. This hindering effect overshadows any benefits of syntactic or semantic bootstrapping during early word learning, when children are still in the early stages of vocabulary and conceptual development.

How well do language models deal with quantification? In this study, we focus on 'few'-type quantifiers, as in 'few children like toys', which might pose a particular challenge for language models because the sentence components with out the quantifier are likely to co-occur, and 'few'-type quantifiers are rare. We present 960 English sentence stimuli from two human neurolinguistic experiments to 22 autoregressive transformer models of differing sizes. Not only do all the models perform poorly on 'few'-type quantifiers, but overall the larger the model, the worse its performance. This inverse scaling is consistent with previous work suggesting that larger models increasingly reflect online rather than offline human processing, and we argue that the decreasing performance of larger models may challenge uses of language models as the basis for natural language systems., Comment: Accepted to Findings of ACL 2023

Supplemental material, sj-docx-1-las-10.1177_00238309221087715 for The Role of Prosody in Disambiguating English Indirect Requests by Sean Trott, Stefanie Reed, Dan Kaliblotzky, Victor Ferreira and Benjamin Bergen in Language and Speech

Words in human languages cluster together in phonological neighborhoods more closely than would be expected by chance. But why? One explanation is that large neighborhoods are directly selected for, possibly because they scaffold word learning and production. But it's also possible that they emerge as a byproduct of other constraints or selection pressures operating over real lexica. We advance one such selection pressure as a candidate explanation. A pressure to avoid overloading unique wordforms with homophones may lead to clusters of words that are not identical but similar. Using simulated baselines, we test the viability of this alternative account. We find that a pressure against loading too many meanings on unique wordforms––paired with the phonotactics of a target language––produces lexica with neighborhoods that are at least as large on average as those in real lexica. This does not rule out the possibility of a pro-neighborhood pressure, but it does demonstrate the viability of a parsimonious alternative account based on a pressure against homonymy for which there is independent evidence.

Are the predictions of humans and language models affected by similar things? Research suggests that while comprehending language, humans make predictions about upcoming words, with more predictable words being processed more easily. However, evidence also shows that humans display a similar processing advantage for highly anomalous words when these words are semantically related to the preceding context or to the most probable continuation. Using stimuli from 3 psycholinguistic experiments, we find that this is also almost always also the case for 8 contemporary transformer language models (BERT, ALBERT, RoBERTa, XLM-R, GPT-2, GPT-Neo, GPT-J, and XGLM). We then discuss the implications of this phenomenon for our understanding of both human language comprehension and the predictions made by language models., Comment: Accepted at CoNLL 2022

Some languages allow arguments to be omitted in certain contexts. Yet human language comprehenders reliably infer the intended referents of these zero pronouns, in part because they construct expectations about which referents are more likely. We ask whether Neural Language Models also extract the same expectations. We test whether 12 contemporary language models display expectations that reflect human behavior when exposed to sentences with zero pronouns from five behavioral experiments conducted in Italian by Carminati (2005). We find that three models - XGLM 2.9B, 4.5B, and 7.5B - capture the human behavior from all the experiments, with others successfully modeling some of the results. This result suggests that human expectations about coreference can be derived from exposure to language, and also indicates features of language models that allow them to better reflect human behavior., Comment: Accepted at COLING 2022

Most words are ambiguous--i.e., they convey distinct meanings in different contexts--and even the meanings of unambiguous words are context-dependent. Both phenomena present a challenge for NLP. Recently, the advent of contextualized word embeddings has led to success on tasks involving lexical ambiguity, such as Word Sense Disambiguation. However, there are few tasks that directly evaluate how well these contextualized embeddings accommodate the more continuous, dynamic nature of word meaning--particularly in a way that matches human intuitions. We introduce RAW-C, a dataset of graded, human relatedness judgments for 112 ambiguous words in context (with 672 sentence pairs total), as well as human estimates of sense dominance. The average inter-annotator agreement (assessed using a leave-one-annotator-out method) was 0.79. We then show that a measure of cosine distance, computed using contextualized embeddings from BERT and ELMo, correlates with human judgments, but that cosine distance also systematically underestimates how similar humans find uses of the same sense of a word to be, and systematically overestimates how similar humans find uses of different-sense homonyms. Finally, we propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics., ACL-IJCNLP 2021 camera-ready

When do people use knowledge about the world in order to comprehend language? We asked whether pronoun resolution decisions are influenced by knowledge about physical plausibility. Results showed that referents which are more physically plausible in described events were more likely to be selected as antecedents of ambiguous pronouns, implying that resolution decisions were driven by physical inference. An alternative explanation is that these decisions were driven instead by distributional word knowledge. We tested this by including predictions of a statistical language model (BERT) and found that physical plausibility explained variance on top of the statistical language model predictions. This indicates that at least part of people's pronoun resolution judgments comes from knowledge about the world and not the word. This result constrains psycholinguistic models of comprehension—world knowledge must influence propositional interpretation—and raises the broader question of how non-linguistic physical inference processes are incorporated during comprehension.

Languages around the world use a recurring strategy to discuss abstract concepts: describe them metaphorically, borrowing language from more concrete domains. We "plan ahead" to the future, "count up" to higher numbers, and "warm" to new friends. Past work has found that these ways of talking have implications for how we think, so that shared systems of linguistic metaphors can produce shared conceptualizations. On the other hand, these systematic linguistic metaphors might not just be the cause but also the effect of shared, non-linguistic ways of thinking. Here, we present a case study of a variety of American English in which a shared, non-linguistic conceptualization of time has become crystallized as a new system of linguistic metaphors. Speakers of various languages, including English, conceptualize time as a lateral timeline, with the past leftward and the future rightward. Until now, this conceptualization has not been documented in the speech of any language. In two studies, we document how members of the U.S. military, but not U.S. civilians, talk about time using conventionalized lateral metaphors (e.g., "move the meeting right" to mean "move the meeting later"). We argue that, under the right cultural circumstances, implicit mental representations become conventionalized metaphors in language.

What concepts and words do communicative gestures activate in the minds of people who view them? It's widely believed that many gestures grow from iconic origins-they look like what they mean-but also that at some point they may become emblematic-conventionalized as culturally agreed-upon symbols. How long do links between physical movements of the body and the things in the world they denote persist in the minds of gesture-users? A pair of experiments asks this question for the Middle-Finger, a cross-culturally recognized obscene gesture. The prevailing view is that the gesture originates in a phallic symbol. Yet it is now predominantly used as an emblematic gesture displaying contempt (among other things). It is currently unknown whether the iconic origins of gestures persist through the emblematic stage in the minds of gesture users. Two experiments tested the hypothesis that viewing the Middle-Finger primes thoughts about penises or the word penis. The results showed that the Middle-Finger induced no priming of penis compared with control, unlike another obscene penis-representing gesture (Finger-Bang), which did. This suggests that the Middle-Finger no longer activates thoughts of penises in the minds of contemporary American English speakers. Emblematic gestures with iconic origins may undergo historical change not just in the functions they serve but also in the effects they have on the minds of people who use them.

In the scope of the present thesis, next generation sequencing, assessment of bacterial bulk parameters and interdisciplinary approaches were used in laboratory experiments and field studies to add to current knowledge on the impact of important environmental factors on bacterial communities. The results highlight the enormous overall impact of environmental conditions on bacterial community composition and activity. They also provide the first tentative identifications of key taxa likely to be affected by changes in those conditions.

The overall aim of the MSM18/5 project was a case study of the average x-z-structure of the interplay between matter transfers and marine communities downstream an upwelling center in the Northern Benguela Current Region at 20��S. For this purpose, we repeated measurements on a transect perpendicular to the coast four times within three weeks. It covered a range from 10 to 500 km distance to shore and depths between 30 and >4000m at most. The program considered forcing functions, such as wind field, Ekman transport, curl driven upwelling, insolation, stratification, and proportions of inorganic nutrient inputs by upwelling and nitrogen fixation and their influence on ecosystems structure and function. We observed successive changes in plankton communities, including microbial communities, in processes like production, respiration and remineralization of nutrients by plankton, in changing importance of new and regenerated nutrient sources, in vertical flux of particulate matter by sedimentation. Finally, the role of meroplankton patterns and the reaction of benthic communities were considered. There was a positive relationship between the pseudo-age (determined by a combination of heat flux and changes in oxygen content) and the distance to shore in the upper 30m of the water column. This supports the idea of the practicability of the Eulerian approach. The scatter was highest between 100 and 200km. This is an indication for higher mesoscale variability by filaments, eddies etc., in this area., MARIA S. MERIAN-Berichte

We discuss a spectral method for the numerical solution of the Vlasov-Poisson system where the velocity space is decomposed by means of an Hermite basis. We describe a semi-implicit time discretization that extends the range of numerical stability relative to an explicit scheme. We also introduce and discuss the effects of an artificial collisional operator, which is necessary to take care of the velocity space filamentation problem, unavoidable in collisionless plasmas. The computational efficiency and the cost-effectiveness of this method are compared to a Particle-in-Cell (PIC) method in the case of a two-dimensional phase space. The following examples are discussed: Langmuir wave, Landau damping, ion-acoustic wave, two-stream instability, and plasma echo. The Hermite spectral method can achieve solutions that are several orders of magnitude more accurate (at a fraction of the cost) with respect to the PIC method., 29 pages; 13 figures; submitted to Journal of Computational Physics

Compositionality is a hallmark of human language - words and morphemes can be factorially combined to produce a seemingly limitless number of viable strings. This contrasts with nonhuman communication systems, which for the most part are holistic - encoding a whole message through a single, gestalt form. Why does every human language adopt a compositional strategy? In this paper, we show that compositional language can arise automatically through grounded communication among populations of communicators. The proposed mechanism is the following: if a holistic and a compositional approach are in competition and if both structured (compositional) and atomic meanings need to be communicated, the holistic strategy becomes less successful as it does not recruit already acquired bits of language. We demonstrate the viability of this explanation through computer simulations in which populations of arti cial agents perform a communicative task - describing scenes that they have observed. Successful language strategies (that is, those yielding successful transmission of information about a scene) are reinforced while unsuccessful ones are demoted. The simulations show that this reinforcement on the basis of communicative success indeed leads to the dominance of compositional language as long as the fraction of unstructured meaning to be communicated is suficiently high. Moreover, following Elman (1993), we then show that the same effect can be achieved by, instead of manipulating the world (the fraction of unstructured meaning presented to the agents), letting the agents themselves go through developmental stages. These simulations con rm that simple reinforcement mechanisms applied during communicative interactions can account for the emergence of linguistic compositionality.