Search

Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of characters) under a language model to its cognitive cost experienced by readers, as operationalized, for example, by gaze duration on the region. However, the application of modern language models to psycholinguistic studies is complicated by the practice of using tokenization as an intermediate step in training a model. Doing so results in a language model over token strings rather than one over character strings. Vexingly, regions of interest are generally misaligned with these token strings. The paper argues that token-level language models should be (approximately) marginalized into character-level language models before they are used in psycholinguistic studies to compute the surprisal of a region of interest; then, the marginalized character-level language model can be used to compute the surprisal of an arbitrary character substring, which we term a focal area, that the experimenter may wish to use as a predictor. Our proposal of marginalizing a token-level model into a character-level one solves this misalignment issue independently of the tokenization scheme. Empirically, we discover various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself., Comment: Main conference long paper at EMNLP 2024. New version: copy-editing and updated bib

Tokenization - the practice of converting strings of characters from an alphabet into sequences of tokens over a vocabulary - is a critical step in the NLP pipeline. The use of token representations is widely credited with increased model performance but is also the source of many undesirable behaviors, such as spurious ambiguity or inconsistency. Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood. In particular, the impact of tokenization on statistical estimation has been investigated mostly through empirical means. The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models. Based on the category of stochastic maps, this framework enables us to establish general conditions for a principled use of tokenizers, and most importantly, the necessary and sufficient conditions for a tokenizer model to preserve the consistency of statistical estimators. Additionally, we discuss statistical and computational concerns crucial for designing and implementing tokenizer models, such as inconsistency, ambiguity, tractability, and boundedness. The framework and results advanced in this paper contribute to building robust theoretical foundations for representations in neural language modeling that can inform future empirical research.

Recent studies have shown that the electromagnetic shower induced by a high-energy electron, positron or photon incident along the axis of an oriented crystal develops in a space more compact than the ordinary. On the other hand, the properties of the hadronic interactions are not affected by the lattice structure. This means that, inside an oriented crystal, the natural difference between the hadronic and the electromagnetic shower profile is strongly accentuated. Thus, a calorimeter composed of oriented crystals could be intrinsically capable of identifying more accurately the nature of the incident particles, with respect to a detector composed only of non-aligned crystals. Since no oriented calorimeter has ever been developed, this possibility remains largely unexplored and can be investigated only by mean of numerical simulations. In this work, we report the first quantitative evaluation of the particle identification capability of such a calorimeter, focusing on the case of neutron-gamma discrimination. We demonstrate through Geant4 simulations that the use of oriented crystals significantly improves the performance of a Random Forest classifier trained on the detector data. This work is a proof that oriented calorimeters could be a viable option for all the environments where particle identification must be performed with a very high accuracy, such as future high-intensity particle physics experiments and satellite-based gamma-ray telescopes., Comment: 13 pages, 9 figures

We observed a substantial increase of the scintillation light output of lead tungstate (PbWO$_4$) at a small incidence angle with respect to two main lattice axes. This reflects the acceleration of electromagnetic shower development that occurs in the crystalline Strong Field. We measured the scintillation light generated by $120$-$\mathrm{GeV}$ electrons and $10$-$100$-$\mathrm{GeV}$ $\gamma$ rays on thick samples. This result deepens the knowledge of the shower development mechanisms in crystal scintillators and could pave the way to the development of innovative accelerator- and space-borne calorimeters.

Using the newly developed 530 MeV positron beam from the Mainz Microtron MAMI and employing a bent silicon crystal, we demonstrate the first successful manipulation with high efficiencies of the trajectories of positrons through planar channeling and volume reflection. This uncovered the presence of fine structure within the angular distribution of charged particles when they are channeled between the planes of bent crystals. The alignment of our experimental findings with simulation results not only demonstrates a deeper understanding of the interactions between charged particle beams and bent crystals but also signals a new phase in the development of innovative methodologies for slow extraction in circular accelerators operating in the GeV range, with implications for worldwide accelerators. Our results also mark a considerable progression in the generation of advanced x-ray sources through the channeling process in periodically bent crystals, rooted in a comprehensive understanding of the interactions between positron beams and such crystals.

For nearly three decades, language models derived from the $n$-gram assumption held the state of the art on the task. The key to their success lay in the application of various smoothing techniques that served to combat overfitting. However, when neural language models toppled $n$-gram models as the best performers, $n$-gram smoothing techniques became less relevant. Indeed, it would hardly be an understatement to suggest that the line of inquiry into $n$-gram smoothing techniques became dormant. This paper re-opens the role classical $n$-gram smoothing techniques may play in the age of neural language models. First, we draw a formal equivalence between label smoothing, a popular regularization technique for neural language models, and add-$\lambda$ smoothing. Second, we derive a generalized framework for converting any $n$-gram smoothing technique into a regularizer compatible with neural language models. Our empirical results find that our novel regularizers are comparable to and, indeed, sometimes outperform label smoothing on language modeling and machine translation., Comment: NAACL 2024

Sampling is a common strategy for generating text from probabilistic models, yet standard ancestral sampling often results in text that is incoherent or ungrammatical. To alleviate this issue, various modifications to a model's sampling distribution, such as nucleus or top-k sampling, have been introduced and are now ubiquitously used in language generation systems. We propose a unified framework for understanding these techniques, which we term sampling adapters. Sampling adapters often lead to qualitatively better text, which raises the question: From a formal perspective, how are they changing the (sub)word-level distributions of language generation models? And why do these local changes lead to higher-quality text? We argue that the shift they enforce can be viewed as a trade-off between precision and recall: while the model loses its ability to produce certain strings, its precision rate on desirable text increases. While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution. Further, these measures correlate with higher sequence-level quality scores, specifically, Mauve., Comment: ACL 2023 Main Conference Proceedings

In this paper we give a meromorphic continuation of the spectral zeta function for semiregular Non-Commutative Harmonic Oscillators (NCHO). By ``semiregular system'' we mean systems with terms with degree of homogeneity scaling by $1$ in their asymptotic expansion. As an application of our results, we first compute the meromorphic continuation of the Jaynes-Cummings (JC) model spectral zeta function. Then we compute the spectral zeta function of the JC generalization to a 3-level atom in a cavity. For both of them we show that it has only one pole in 1., Comment: 22 pages

Starting from the results of Charles Fefferman and Janos Koll\'ar in \texit{Continuous Solutions of Linear Equations} [1], we adopt a new approach based on Fefferman's techniques of Glaeser refinement to show a more general result than the one proved by Koll\'ar by using techniques from algebraic geometry. Considering a system of linear equations with semialgebraic (not only polynomial as in [1]) coefficients on $\mathbb{R}^{n}$, we get a necessary and sufficient condition for the existence of a continuous and semialgebraic solution on $\mathbb{R}^{n}$. This is different from what Fefferman and Luli obtained in \textit{Semialgebraic Sections Over the Plane} since they stated their result for solutions of regularity $C^m$ on the plane $\mathbb{R}^2$. More in depth, we prove that a continuous and semialgebraic solution on $\mathbb{R}^{n}$ exists if and only if there is a continuous solution i.e., if the Glaeser-stable bundle associated to the system has no empty fiber., Comment: 12 pages. arXiv admin note: text overlap with arXiv:2005.00067

With the advent of deep learning algorithms, fully automated radiological image analysis is within reach. In spine imaging, several atlas- and shape-based as well as deep learning segmentation algorithms have been proposed, allowing for subsequent automated analysis of morphology and pathology. The first Large Scale Vertebrae Segmentation Challenge (VerSe 2019) showed that these perform well on normal anatomy, but fail in variants not frequently present in the training dataset. Building on that experience, we report on the largely increased VerSe 2020 dataset and results from the second iteration of the VerSe challenge (MICCAI 2020, Lima, Peru). VerSe 2020 comprises annotated spine computed tomography (CT) images from 300 subjects with 4142 fully visualized and annotated vertebrae, collected across multiple centres from four different scanner manufacturers, enriched with cases that exhibit anatomical variants such as enumeration abnormalities (n=77) and transitional vertebrae (n=161). Metadata includes vertebral labelling information, voxel-level segmentation masks obtained with a human-machine hybrid algorithm and anatomical ratings, to enable the development and benchmarking of robust and accurate segmentation algorithms., Comment: 18 pages, 2 figures, 2 tables; Hans Liebl, David Schinz equally contributed to this manuscript

Starting from the results of Charles Fefferman and Janos Koll\`ar in Continuous Solutions of Linear Equations [1], we adopt a new approach based on Fefferman's techniques of Glaeser refinement to show a more general result than the one proved by Koll\`ar by using techniques from algebraic geometry. Considering a system of linear equations with semialgebraic (not only polynomial as in [1]) coefficients on R^n, we get a necessary and sufficient condition for the existence of a continuous and semialgebraic solution on R^n. This is different from what Fefferman and Luli obtained in Semialgebraic Sections Over the Plane [3] since they stated their result for solutions of regularity C^m on the plane R^2. More in depth, we prove that a continuous and semialgebraic solution on Rn exists if and only if there is a continuous solution i.e., if the Glaeser-stable bundle associated to the system has no empty fiber., Comment: 12 pages