Search

Instruction tuning enhances large language models (LLMs) by aligning them with human preferences across diverse tasks. Traditional approaches to create instruction tuning datasets face serious challenges for low-resource languages due to their dependence on data annotation. This work introduces a novel method, Multilingual Reverse Instructions (MURI), which generates high-quality instruction tuning datasets for low-resource languages without requiring human annotators or pre-existing multilingual models. Utilizing reverse instructions and a translation pipeline, MURI produces instruction-output pairs from existing human-written texts in low-resource languages. This method ensures cultural relevance and diversity by sourcing texts from different native domains and applying filters to eliminate inappropriate content. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages. Evaluation by native speakers and fine-tuning experiments with mT5 models demonstrate the approach's effectiveness for both NLU and open-ended generation. We publicly release datasets and models at https://github.com/akoksal/muri.

Surgical scenes convey crucial information about the quality of surgery. Pixel-wise localization of tools and anatomical structures is the first task towards deeper surgical analysis for microscopic or endoscopic surgical views. This is typically done via fully-supervised methods which are annotation greedy and in several cases, demanding medical expertise. Considering the profusion of surgical videos obtained through standardized surgical workflows, we propose an annotation-efficient framework for the semantic segmentation of surgical scenes. We employ image-based self-supervised object discovery to identify the most salient tools and anatomical structures in surgical videos. These proposals are further refined within a minimally supervised fine-tuning step. Our unsupervised setup reinforced with only 36 annotation labels indicates comparable localization performance with fully-supervised segmentation models. Further, leveraging surgical phase labels as weak labels can better guide model attention towards surgical tools, leading to $\sim 2\%$ improvement in tool localization. Extensive ablation studies on the CaDIS dataset validate the effectiveness of our proposed solution in discovering relevant surgical objects with minimal or no supervision., Comment: 9 pages, 4 figures, 2 tables

Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed. Given the few-shot examples, we use large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents. Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning. We demonstrate that CRAFT can efficiently generate large-scale task-specific training datasets for four diverse tasks: biology question-answering (QA), medicine QA and commonsense QA as well as summarization. Our experiments show that CRAFT-based models outperform or achieve comparable performance to general LLMs for QA tasks, while CRAFT-based summarization models outperform models trained on human-curated data by 46 preference points.

Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability to offer comprehensive, interpretable, and dynamic assessments of NLP models. Recently, works like DynaBench (Kiela et al., 2021) and CheckList (Ribeiro et al., 2020) have addressed these limitations through behavioral testing of NLP models with test types generated by a multistep human-annotated pipeline. Unfortunately, manually creating a variety of test types requires much human labor, often at prohibitive cost. In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models. SYNTHEVAL first generates sentences via LLMs using controlled generation, and then identifies challenging examples by comparing the predictions made by LLMs with task-specific NLP models. In the last stage, human experts investigate the challenging examples, manually design templates, and identify the types of failures the taskspecific models consistently exhibit. We apply SYNTHEVAL to two classification tasks, sentiment analysis and toxic language detection, and show that our framework is effective in identifying weaknesses of strong models on these tasks. We share our code in https://github.com/Loreley99/SynthEval_CheckList.

The anomalous magnetic ($\tilde{a}_\tau$) and electric dipole ($\tilde{d}_\tau$) moment of tau lepton described in $\tau\bar{\tau}\gamma$ vertices are studied via $\mu^{+}\mu^{-} \rightarrow \tau^+\bar{\tau}^-$ process at the Muon colliders designed with the proposed center-of-mass-energy / integrated luminosity configurations of 3 TeV/ 1 ab$^{-1}$ and 10 TeV/ 10 ab$^{-1}$.We obtained the $95\%$ confidence level limits on the $\tilde{a}_\tau$ and $\tilde{d}_\tau$ parameters without and with systematic uncertainty of $10\%$ and compared with the experimental results.The most stringent limits on the anomalous couplings without systematic uncertainty are $-2.63\times10^{-4}< \tilde{a}_\tau <2. 65\times10^{-4}$ and $\mid \tilde{d}_\tau \mid \leq 1.47\times10^{-18} e cm$ at $\sqrt s =10$ TeV and $L_{int}$=10 ab$^{-1}$ option of the Muon collider. Our results show that the Muon colliders lead to a remarkable improvement on the current experimental limits of the anomalous magnetic and electric dipole moments of the tau lepton., Comment: 12 pages, 3 figures

Graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition, Comment: 9 pages, 3 figures, 3 tables, MICCAI GRAIL Workshop paper

Multiple choice question answering tasks evaluate the reasoning, comprehension, and mathematical abilities of Large Language Models (LLMs). While existing benchmarks employ automatic translation for multilingual evaluation, this approach is error-prone and potentially introduces culturally biased questions, especially in social sciences. We introduce the first multitask, multiple-choice Turkish QA benchmark, TurkishMMLU, to evaluate LLMs' understanding of the Turkish language. TurkishMMLU includes over 10,000 questions, covering 9 different subjects from Turkish high-school education curricula. These questions are written by curriculum experts, suitable for the high-school curricula in Turkey, covering subjects ranging from natural sciences and math questions to more culturally representative topics such as Turkish Literature and the history of the Turkish Republic. We evaluate over 20 LLMs, including multilingual open-source (e.g., Gemma, Llama, MT5), closed-source (GPT 4o, Claude, Gemini), and Turkish-adapted (e.g., Trendyol) models. We provide an extensive evaluation, including zero-shot and few-shot evaluation of LLMs, chain-of-thought reasoning, and question difficulty analysis along with model performance. We provide an in-depth analysis of the Turkish capabilities and limitations of current LLMs to provide insights for future LLMs for the Turkish language. We publicly release our code for the dataset and evaluation: https://github.com/ArdaYueksel/TurkishMMLU., Comment: EMNLP 2024 - Findings

Many datasets have been developed to train and evaluate document-level relation extraction (RE) models. Most of these are constructed using real-world data. It has been shown that RE models trained on real-world data suffer from factual biases. To evaluate and address this issue, we present CovEReD, a counterfactual data generation approach for document-level relation extraction datasets using entity replacement. We first demonstrate that models trained on factual data exhibit inconsistent behavior: while they accurately extract triples from factual data, they fail to extract the same triples after counterfactual modification. This inconsistency suggests that models trained on factual data rely on spurious signals such as specific entities and external knowledge $\unicode{x2013}$ rather than on the input context $\unicode{x2013}$ to extract triples. We show that by generating document-level counterfactual data with CovEReD and training models on them, consistency is maintained with minimal impact on RE performance. We release our CovEReD pipeline as well as Re-DocRED-CF, a dataset of counterfactual RE documents, to assist in evaluating and addressing inconsistency in document-level RE.

While current large language models (LLMs) demonstrate some capabilities in knowledge-intensive tasks, they are limited by relying on their parameters as an implicit storage mechanism. As a result, they struggle with infrequent knowledge and temporal degradation. In addition, the uninterpretable nature of parametric memorization makes it challenging to understand and prevent hallucination. Parametric memory pools and model editing are only partial solutions. Retrieval Augmented Generation (RAG) $\unicode{x2013}$ though non-parametric $\unicode{x2013}$ has its own limitations: it lacks structure, complicates interpretability and makes it hard to effectively manage stored knowledge. In this paper, we introduce MemLLM, a novel method of enhancing LLMs by integrating a structured and explicit read-and-write memory module. MemLLM tackles the aforementioned challenges by enabling dynamic interaction with the memory and improving the LLM's capabilities in using stored knowledge. Our experiments indicate that MemLLM enhances the LLM's performance and interpretability, in language modeling in general and knowledge-intensive tasks in particular. We see MemLLM as an important step towards making LLMs more grounded and factual through memory augmentation.

We introduce, XoFTR, a cross-modal cross-view method for local feature matching between thermal infrared (TIR) and visible images. Unlike visible images, TIR images are less susceptible to adverse lighting and weather conditions but present difficulties in matching due to significant texture and intensity differences. Current hand-crafted and learning-based methods for visible-TIR matching fall short in handling viewpoint, scale, and texture diversities. To address this, XoFTR incorporates masked image modeling pre-training and fine-tuning with pseudo-thermal image augmentation to handle the modality differences. Additionally, we introduce a refined matching pipeline that adjusts for scale discrepancies and enhances match reliability through sub-pixel level refinement. To validate our approach, we collect a comprehensive visible-thermal dataset, and show that our method outperforms existing methods on many benchmarks., Comment: CVPR Image Matching Workshop, 2024. 12 pages, 7 figures, 5 tables. Codes and dataset are available at https://github.com/OnderT/XoFTR

Argument Structure Constructions (ASCs) are one of the most well-studied construction groups, providing a unique opportunity to demonstrate the usefulness of Construction Grammar (CxG). For example, the caused-motion construction (CMC, ``She sneezed the foam off her cappuccino'') demonstrates that constructions must carry meaning, otherwise the fact that ``sneeze'' in this context causes movement cannot be explained. We form the hypothesis that this remains challenging even for state-of-the-art Large Language Models (LLMs), for which we devise a test based on substituting the verb with a prototypical motion verb. To be able to perform this test at statistically significant scale, in the absence of adequate CxG corpora, we develop a novel pipeline of NLP-assisted collection of linguistically annotated text. We show how dependency parsing and GPT-3.5 can be used to significantly reduce annotation cost and thus enable the annotation of rare phenomena at scale. We then evaluate GPT, Gemini, Llama2 and Mistral models for their understanding of the CMC using the newly collected corpus. We find that all models struggle with understanding the motion component that the CMC adds to a sentence.

In the post-LHC era, the muon collider represents a frontier project capable of providing high-energy and high-luminosity leptonic collisions among future lepton-lepton particle accelerators. In addition, it provides significantly cleaner final states than those produced in hadron collisions. With this expectation in mind, in this article, we research the sensitivity of the anomalous neutral gauge boson couplings $Z\gamma\gamma\gamma$, $ZZ\gamma \gamma$, and $ZZZ\gamma$ defined by dimension-8 operators, through the $\mu^+\mu^- \to \mu^+\mu^-Z\gamma$ signal, with the $Z$-boson decaying to neutrino pair. The projections of new physics at the future muon collider with the center-of-mass energy of $\sqrt{s}=10$ TeV, integrated luminosity of ${\cal L}=10$ $\rm ab^{-1}$, and systematic uncertainties of $\delta_{sys}=0\%, 3\%, 5\%$, for extraction of expected sensitivity on the anomalous $f_ {T,j}/\Lambda^4$ couplings at $95\%$ confidence level, are of the order of ${\cal O}(10^{-4}- 10^{-3})$. Compared with the research of the ATLAS and CMS Collaborations on the anomalous quartic gauge couplings, we find that the high-luminosity future muon collider could have better sensitivity.

Attribution is a key concept in large language models (LLMs) as it enables control over information sources and enhances the factuality of LLMs. While existing approaches utilize open book question answering to improve attribution, factual datasets may reward language models to recall facts that they already know from their pretraining data, not attribution. In contrast, counterfactual open book QA datasets would further improve attribution because the answer could only be grounded in the given text. We propose Hallucination Augmented Recitations (HAR) for creating counterfactual datasets by utilizing hallucination in LLMs to improve attribution. For open book QA as a case study, we demonstrate that models finetuned with our counterfactual datasets improve text grounding, leading to better open book QA performance, with up to an 8.0% increase in F1 score. Our counterfactual dataset leads to significantly better performance than using humanannotated factual datasets, even with 4x smaller datasets and 4x smaller models. We observe that improvements are consistent across various model sizes and datasets, including multi-hop, biomedical, and adversarial QA datasets.

In this study, we explore the non-standard $\nu\bar{\nu}\gamma \gamma$ couplings parametrized by dimension-seven operators via $e^{+}e^{-} \to \nu\bar{\nu}\gamma$ process at the FCC-ee/CEPC and $\mu^{+}\mu^{-}\to\nu\bar{\nu}\gamma$ process at the Muon Colliders. For the detailed Monte Carlo simulation, all signal and relevant background events are produced within the framework of Madgraph where non-standard $\nu\bar{\nu}\gamma \gamma$ couplings are implemented. After passing through Pythia for parton showering and hadronization, detector effects are included via tuned corresponding detector cards for each collider in Delphes. Projected sensitivities on $\nu\bar{\nu}\gamma \gamma$ couplings are obtained at a 5$\sigma$ confidence level without and with $5\%$ systematic uncertainties for the FCC-ee/CEPC and the Muon Colliders, showcasing the complementarity between lepton colliders. Our best limit on the anomalous $\nu\bar{\nu}\gamma \gamma$ couplings even with 5\% systematic uncertainties for muon collider with $\sqrt{s}=10$ TeV and $L_{int}=3$ ab$^{-1}$ are found to be thirteen orders of magnitude stronger than the upper bound obtained from rare decay $Z\to\gamma\gamma\nu\bar{\nu}$ analysis using LEP data., Comment: 17 pages, 9 figures

The presence of multi-boson self-interactions is implied by the non-Abelian gauge structure of the Standard Model (SM). Precise measurements of these interactions allow not only testing the nature of the SM but also new physics contribution arising from the beyond SM. The investigation of these interactions can be approached in a model-independent manner using an effective theory approach, which forms the main motivation of this study. In this paper, we examine the anomalous neutral quartic gauge couplings through the process $\gamma \gamma \rightarrow Z Z$ at the Compact Linear Collider (CLIC) with the center-of-mass energy of $\sqrt{s}=3$ TeV, integrated luminosities of ${\cal L}=5$ $\rm ab^{-1}$. The anomalous neutral quartic gauge couplings is implemented into FeynRules to generate a UFO module inserted into Madgraph to generate both background and signal events. These events are then passed through Pythia 8 for parton showering and Delphes to include realistic detector effects. We obtain that the sensitivities on the anomalous quartic neutral gauge couplings with $95\%$ Confidence Level are given as: $f_{T0}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$, $f_{T1}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$,$f_{T2}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$,$f_{T0}/\Lambda^{4}=[-1.06; 1.08]\times 10^{-3}$ ${\rm TeV^{-4}}$, $f_{T5}/\Lambda^{4}=[-4.08; 4.08]\times 10^{-4}$ ${\rm TeV^{-4}}$ and $f_{T8}/\Lambda^{4}=[-1.10; 1.10]\times10^{-4}$ ${\rm TeV^{-4}}$. Our results on the anomalous quartic neutral gauge couplings are set more stringent sensitivity with respect to the recent experimental limits., Comment: 21 pages, 6 figures

Spontaneous breaking of the $SU(2)_{L}\times U(1)_{Y}$ electroweak symmetry of the Standard Model (SM) sets the constraints on triple gauge couplings and quartic gauge couplings. Therefore, the measurement of multiboson production in $e^{-} e^{+}$ collisions allows us to directly examine the SM predictions and perform indirect investigations of new physics beyond the SM. In this paper, we concentrate the process $e^{-} e^{+} \to e^{-} Z\gamma e^{+}$ with $Z$ boson decaying to neutrinos to investigate the anomalous quartic gauge couplings using the effective Lagrangian approach at the Compact Linear Collider (CLIC). We obtain the sensitivities on the anomalous $ f_ {Ti}/\Lambda^4$ ($i=0,2,5,6,7,8,9$) couplings taking into account the systematic uncertainties of $3, 5 \%$ at $95\%$ Confidence Level for the CLIC with $\sqrt{s}=3$ TeV. Our results show that the sensitivities on some anomalous couplings without systematic errors are up to two orders of magnitude better than the current experimental limits. Considering a realistic systematic uncertainty such as $5 \%$ from possible experimental sources, the sensitivity of all anomalous quartic couplings gets worse by about $10\%$ compared to those without systematic uncertainty for the CLIC., Comment: 23 Pages, 10 Figures, 2 Tables

Pretrained language models (PLMs) are key components in NLP, but they contain strong social biases. Quantifying these biases is challenging because current methods focusing on fill-the-mask objectives are sensitive to slight changes in input. To address this, we propose a bias probing technique called LABDet, for evaluating social bias in PLMs with a robust and language-agnostic method. For nationality as a case study, we show that LABDet `surfaces' nationality bias by training a classifier on top of a frozen PLM on non-nationality sentiment detection. We find consistent patterns of nationality bias across monolingual PLMs in six languages that align with historical and political context. We also show for English BERT that bias surfaced by LABDet correlates well with bias in the pretraining data; thus, our work is one of the few studies that directly links pretraining data to PLM behavior. Finally, we verify LABDet's reliability and applicability to different templates and languages through an extensive set of robustness checks. We publicly share our code and dataset in https://github.com/akoksal/LABDet., Comment: EMNLP 2023 Findings

The dice lattice, containing a stack of three triangular lattices, has been proposed to exhibit nontrivial flat bands with nonzero Chern numbers, but unlike the honeycomb lattice it is much less studied. By employing density-functional theory (DFT) calculations with an on-site Coulomb repulsion term, we explore systematically the electronic and topological properties of (La$X$O$_3$)$_3$/(LaAlO$_3$)$_3$(111) superlattices with $X=$ Ti, Mn and Co, where a LaAlO$_3$ trilayer spacer confines the La$X$O$_3$ (L$X$O) dice lattice. In the absence of spin-orbit coupling (SOC) with symmetry constrained to P3, the ferromagnetic (FM) phase of the L$X$O(111) trilayers exhibits a distinct spin-polarized half-metallic state with multiple Dirac crossings and coupled electron-hole pockets around the Fermi energy. Symmetry lowering induces a significant rearrangement of bands and triggers a metal-to-insulator transition. Inclusion of SOC leads to a substantial anomalous Hall conductivity (AHC) around the Fermi energy reaching values up to $\sim-3e^2/h$ for $X=$ Mn and Co in P3 symmetry and both in- and out-of-plane magnetization directions in the first case and along [001] in the latter. The dice lattice emerges as a promising playground to realise nontrivial topological phases with high Chern numbers., Comment: 10 pages, 6 figures

Instruction tuning enables language models to more effectively generalize and better follow user intent. However, obtaining instruction data is costly and challenging. Prior work employs methods such as expensive human annotation, crowd-sourced datasets with alignment issues, and generating noisy examples via LLMs. We introduce the LongForm-C dataset, which is created by reverse instructions. We generate instructions via LLMs for human-written corpus examples using reverse instructions. First we select a diverse set of human-written documents from corpora such as C4 and Wikipedia; then we generate instructions for these documents via LLMs. This approach provides a cheaper and cleaner instruction-tuning dataset with natural output and one suitable for long text generation. Our models outperform 10x larger language models without instruction tuning on tasks such as story/recipe generation and long-form question answering. Moreover, LongForm models outperform prior instruction-tuned models such as FLAN-T5 and Alpaca by a large margin, and improve language understanding capabilities further. We publicly release our data and models: https://github.com/akoksal/LongForm., Comment: EMNLP 2024 Findings. This version extends the training with recent LLMs, evaluation with new metrics, and NLU tasks

We introduce HATELEXICON, a lexicon of slurs and targets of hate speech for the countries of Brazil, Germany, India and Kenya, to aid training and interpretability of models. We demonstrate how our lexicon can be used to interpret model predictions, showing that models developed to classify extreme speech rely heavily on target words when making predictions. Further, we propose a method to aid shot selection for training in low-resource settings via HATELEXICON. In few-shot learning, the selection of shots is of paramount importance to model performance. In our work, we simulate a few-shot setting for German and Hindi, using HASOC data for training and the Multilingual HateCheck (MHC) as a benchmark. We show that selecting shots based on our lexicon leads to models performing better on MHC than models trained on shots sampled randomly. Thus, when given only a few training examples, using our lexicon to select shots containing more sociocultural information leads to better few-shot performance.

The $\nu\bar{\nu}\gamma \gamma$ couplings parametrized with the non-standard dimension-seven operators defined by the Effective Field Theory framework are investigated through the process $pp\to \nu\bar{\nu}\gamma$ at the High Luminosity-LHC and the Future Circular proton-proton Collider. The effective Lagrangian of $\nu\bar{\nu}\gamma \gamma$ couplings is implemented into FeynRules to generate a UFO module inserted into Madgraph to generate both background and signal events. These events are then passed through Pythia 8 for parton showering and Delphes to include realistic detector effects. The sensitivities on $\nu\bar{\nu}\gamma \gamma$ couplings are obtained at $95\%$ confidence level. We show that the analysis of the signal emerging from the process $pp\to \nu\bar{\nu}\gamma$ allows to improve constraints on $\nu\bar{\nu}\gamma \gamma$ couplings given by the LEP collaboration., Comment: 14 pages, 6 figures, 2 tables, version accepted for publication in Physics Letters B

This work offers the constraints on the anomalous neutral triple gauge couplings for the process $e^+e^- \to ZZ $ at the CLIC with $\sqrt{s}=3$ TeV. The realistic CLIC detector environments and their effects are considered in our analysis. The study is planned for the decays of produced $Z$ bosons to a pair of charged leptons (electrons or muons) and neutrino pairs. The bounds on the anomalous neutral triple gauge couplings defining $CP$-violating $C_{\widetilde{B}W}/{\Lambda^4}$ coupling and three $CP$-conserving $C_{WW}/{\Lambda^4}$, $C_{BW}/{\Lambda^4}$, and $C_{BB}/{\Lambda^4}$ couplings are obtained. Also, the effects and advantages of polarization for incoming electron beams in these calculations are investigated., Comment: 22 pages, 13 figures