Search

Question Answering (QA) datasets have been instrumental in developing and evaluating Large Language Model (LLM) capabilities. However, such datasets are scarce for languages other than English due to the cost and difficulties of collection and manual annotation. This means that producing novel models and measuring the performance of multilingual LLMs in low-resource languages is challenging. To mitigate this, we propose $\textbf{S}$yn$\textbf{DAR}$in, a method for generating and validating QA datasets for low-resource languages. We utilize parallel content mining to obtain $\textit{human-curated}$ paragraphs between English and the target language. We use the English data as context to $\textit{generate}$ synthetic multiple-choice (MC) question-answer pairs, which are automatically translated and further validated for quality. Combining these with their designated non-English $\textit{human-curated}$ paragraphs form the final QA dataset. The method allows to maintain the content quality, reduces the likelihood of factual errors, and circumvents the need for costly annotation. To test the method, we created a QA dataset with $1.2$K samples for the Armenian language. The human evaluation shows that $98\%$ of the generated English data maintains quality and diversity in the question types and topics, while the translation validation pipeline can filter out $\sim70\%$ of data with poor quality. We use the dataset to benchmark state-of-the-art LLMs, showing their inability to achieve human accuracy with some model performances closer to random chance. This shows that the generated dataset is non-trivial and can be used to evaluate reasoning capabilities in low-resource language.

Recent studies of the emergent capabilities of transformer-based Natural Language Understanding (NLU) models have indicated that they have an understanding of lexical and compositional semantics. We provide evidence that suggests these claims should be taken with a grain of salt: we find that state-of-the-art Natural Language Inference (NLI) models are sensitive towards minor semantics preserving surface-form variations, which lead to sizable inconsistent model decisions during inference. Notably, this behaviour differs from valid and in-depth comprehension of compositional semantics, however does neither emerge when evaluating model accuracy on standard benchmarks nor when probing for syntactic, monotonic, and logically robust reasoning. We propose a novel framework to measure the extent of semantic sensitivity. To this end, we evaluate NLI models on adversarially generated examples containing minor semantics-preserving surface-form input noise. This is achieved using conditional text generation, with the explicit condition that the NLI model predicts the relationship between the original and adversarial inputs as a symmetric equivalence entailment. We systematically study the effects of the phenomenon across NLI models for $\textbf{in-}$ and $\textbf{out-of-}$ domain settings. Our experiments show that semantic sensitivity causes performance degradations of $12.92\%$ and $23.71\%$ average over $\textbf{in-}$ and $\textbf{out-of-}$ domain settings, respectively. We further perform ablation studies, analysing this phenomenon across models, datasets, and variations in inference and show that semantic sensitivity can lead to major inconsistency within model predictions., Comment: EACL 2024

Knowledge graphs (KGs) are inherently incomplete because of incomplete world knowledge and bias in what is the input to the KG. Additionally, world knowledge constantly expands and evolves, making existing facts deprecated or introducing new ones. However, we would still want to be able to answer queries as if the graph were complete. In this chapter, we will give an overview of several methods which have been proposed to answer queries in such a setting. We will first provide an overview of the different query types which can be supported by these methods and datasets typically used for evaluation, as well as an insight into their limitations. Then, we give an overview of the different approaches and describe them in terms of expressiveness, supported graph types, and inference capabilities., Comment: Preprint of Ch. 17 "Approximate Answering of Graph Queries" in "Compendium of Neurosymbolic Artificial Intelligence", https://ebooks.iospress.nl/ISBN/978-1-64368-406-2

We systematically test a quasi-classical spin model of a large spin-lattice in energy space, with a tunable, reversible Hamiltonian and effective long-range interactions. The system is simulated by a weakly interacting Fermi gas undergoing perturbed quantum rewinding using radio-frequency(RF) pulses. The model reported here is found to be in a quantitative agreement with measurements of the ensemble-averaged energy-resolved spin density. This work elucidates the effects of RF detunings on the system and measurements, pointing the way to new correlation measurement methods., Comment: 13 pages 8 figures

Stance Detection is concerned with identifying the attitudes expressed by an author towards a target of interest. This task spans a variety of domains ranging from social media opinion identification to detecting the stance for a legal claim. However, the framing of the task varies within these domains, in terms of the data collection protocol, the label dictionary and the number of available annotations. Furthermore, these stance annotations are significantly imbalanced on a per-topic and inter-topic basis. These make multi-domain stance detection a challenging task, requiring standardization and domain adaptation. To overcome this challenge, we propose $\textbf{T}$opic $\textbf{E}$fficient $\textbf{St}$anc$\textbf{E}$ $\textbf{D}$etection (TESTED), consisting of a topic-guided diversity sampling technique and a contrastive objective that is used for fine-tuning a stance classifier. We evaluate the method on an existing benchmark of $16$ datasets with in-domain, i.e. all topics seen and out-of-domain, i.e. unseen topics, experiments. The results show that our method outperforms the state-of-the-art with an average of $3.5$ F1 points increase in-domain, and is more generalizable with an averaged increase of $10.2$ F1 on out-of-domain evaluation while using $\leq10\%$ of the training data. We show that our sampling technique mitigates both inter- and per-topic class imbalances. Finally, our analysis demonstrates that the contrastive learning objective allows the model a more pronounced segmentation of samples with varying labels., Comment: ACL 2023 (Oral)

We systematically study how three large language models with code capabilities - CodeT5, Codex, and ChatGPT - generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We split data into domains following its natural boundaries - by an organization, by a project, and by a module within the software project. We establish that samples from each new domain present all the models with a significant challenge of distribution shift. We study how established methods adapt models to better generalize to new domains. Our experiments show that while multitask learning alone is a reasonable baseline, combining it with few-shot finetuning on examples retrieved from training data can achieve very strong performance. Moreover, this solution can outperform direct finetuning for very low-data scenarios. Finally, we consider variations of this approach to create a more broadly applicable method to adapt to multiple domains at once. We find that for code generation, a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain

Answering complex queries on incomplete knowledge graphs is a challenging task where a model needs to answer complex logical queries in the presence of missing knowledge. Prior work in the literature has proposed to address this problem by designing architectures trained end-to-end for the complex query answering task with a reasoning process that is hard to interpret while requiring data and resource-intensive training. Other lines of research have proposed re-using simple neural link predictors to answer complex queries, reducing the amount of training data by orders of magnitude while providing interpretable answers. The neural link predictor used in such approaches is not explicitly optimised for the complex query answering task, implying that its scores are not calibrated to interact together. We propose to address these problems via CQD$^{\mathcal{A}}$, a parameter-efficient score \emph{adaptation} model optimised to re-calibrate neural link prediction scores for the complex query answering task. While the neural link predictor is frozen, the adaptation component -- which only increases the number of model parameters by $0.03\%$ -- is trained on the downstream complex query answering task. Furthermore, the calibration component enables us to support reasoning over queries that include atomic negations, which was previously impossible with link predictors. In our experiments, CQD$^{\mathcal{A}}$ produces significantly more accurate results than current state-of-the-art methods, improving from $34.4$ to $35.1$ Mean Reciprocal Rank values averaged across all datasets and query types while using $\leq 30\%$ of the available training query types. We further show that CQD$^{\mathcal{A}}$ is data-efficient, achieving competitive results with only $1\%$ of the training complex queries, and robust in out-of-domain evaluations.

We have studied the system of globular clusters (GCs) that formed in other galaxies and eventually accreted onto the Milky Way. Thus, the samples of GCs belonging to different tidal streams, obtained on the basis of the latest data from the Gaia observatory, were taken from the literature. We measured the anisotropy of the distribution of these GCs using the gyration tensor and found that the distribution of GCs in the streams is isotropic. Nevertheless, it can be seen that some of the accreted GCs included into existing samples actually belong to the disk of the Galaxy. To clarify the origin of GCs, we investigated the ``age--metallicity'' relation. This dependence demonstrates bimodality and its two different branches clearly show the difference between the clusters formed in the streams and in the disk of the Galaxy. Furthermore, we have studied the influence of the large--scale environment of the Galaxy (i.e., the Local Supercluster) on the distribution of satellite galaxies and Galactic GCs. The satellite galaxies of the Milky Way are known to form an anisotropic planar structure, so we included them in our analysis too. An inspection has shown that the plane of the satellite galaxies is perpendicular both to the disk of the Galaxy and the supergalactic plane. For GCs more distant than 100~Kpc, a similar picture is observed., Comment: 15 pages, 3 figures

In this work we address graph based semi-supervised learning using the theory of the spatial segregation of competitive systems. First, we define a discrete counterpart over connected graphs by using direct analogue of the corresponding competitive system. This model turns out doesn't have a unique solution as we expected. Nevertheless, we suggest gradient projected and regularization methods to reach some of the solutions. Then we focus on a slightly different model motivated from the recent numerical results on the spatial segregation of reaction-diffusion systems. In this case we show that the model has a unique solution and propose a novel classification algorithm based on it. Finally, we present numerical experiments showing the method is efficient and comparable to other semi-supervised learning algorithms at high and low label rates., Comment: 27 pages, 45 figures, 2 tables; Key words and phrases. Free boundary, Semi-supervised learning, Laplace learning

We consider the experimental data on the production of strange Lambdas, and multistrange baryons (Xi, Omega), and antibaryons, on nuclear targets, at the energy region from SPS up to LHC, in the framework of the Quark-Gluon String Model. One remarkable result of this analysis is the significant dependence on the centrality of the collision of the experimental ratios bar(Xi)+/bar(Lambda) and bar(Omega)+/bar(Lambda) ratios in heavy-ion collisions, at SPS energies., Comment: 4 pages, 2 figures, 1 table Contribution to the 51th International Symposium on Multiparticle Dynamics (ISMD2022), Pitlochry, Scotland, UK, July 31st-August 5th (2022)

In this work we prove uniqueness result for an implicit discrete system defined on connected graphs. Our discrete system is motivated from a certain class of spatial segregation of reaction-diffusion equations., Comment: 14 pages, 1 figure

Semantic code search is the task of retrieving a code snippet given a textual description of its functionality. Recent work has been focused on using similarity metrics between neural embeddings of text and code. However, current language models are known to struggle with longer, compositional text, and multi-step reasoning. To overcome this limitation, we propose supplementing the query sentence with a layout of its semantic structure. The semantic layout is used to break down the final reasoning decision into a series of lower-level decisions. We use a Neural Module Network architecture to implement this idea. We compare our model - NS3 (Neuro-Symbolic Semantic Search) - to a number of baselines, including state-of-the-art semantic code retrieval methods, and evaluate on two datasets - CodeSearchNet and Code Search and Question Answering. We demonstrate that our approach results in more precise code retrieval, and we study the effectiveness of our modular design when handling compositional queries.

We consider the experimental data on yields of protons, strange $\Lambda^{\prime}$s, and multistrange baryons ($\Xi$, $\Omega$), and antibaryons production on nuclear targets, and the experimental ratios of multistrange to strange antibaryon production, at the energy region from SPS up to LHC, and compare them to the results of the Quark-Gluon String Model calculations. In the case of heavy nucleus collisions, the experimental dependence of the $\overline{\Xi}^+/\overline{\Lambda}$, and, in particular, of the $\overline{\Omega}^+/\overline{\Lambda}$ ratios, on the centrality of the collision, shows a manifest violation of quark combinatorial rules., Comment: 18 pages, 7 figures, 4 tables. Accepted for publication in Physical Review D

We measure the free decay of a spatially periodic density profile in a normal fluid strongly interacting Fermi gas, which is confined in a box potential. This spatial profile is initially created in thermal equilibrium by a perturbing potential. After the perturbation is abruptly extinguished, the dominant spatial Fourier component exhibits an exponentially decaying (thermally diffusive) mode and a decaying oscillatory (first sound) mode, enabling independent measurement of the thermal conductivity and the shear viscosity directly from the time-dependent evolution., Comment: 16 pages, 12 Figures

In this work a search was carried out for globular clusters belonging to the Sagittarius (Sgr) tidal stream using the analysis of spatial positions, radial velocities relative to the Galactic Standard of Rest (V_{GSR}),proper motions and ratio of "age -- metallicity" ([Fe/H]) for globular clusters and for stars in the tidal stream. As a result, three categories of globular clusters were obtained: A -- most certainly in the stream: Terzan 8, Whiting 1, Arp 2, NGC 6715, Terzan 7, Pal 12; B -- kinematic outliers: Pal 5, NGC 5904, NGC 5024, NGC 5053, NGC 5272, NGC 288; C -- lowest rank candidates: NGC 6864, NGC 5466, NGC 5897, NGC 7492, NGC 4147.