Search

The rise of powerful multimodal LLMs has enhanced the viability of building web agents which can, with increasing levels of autonomy, assist users to retrieve information and complete tasks on various human-computer interfaces. It is hence necessary to build challenging benchmarks that span a wide-variety of use cases reflecting real-world usage. In this work, we present WebQuest, a multi-page question-answering dataset that requires reasoning across multiple related web pages. In contrast to existing UI benchmarks that focus on multi-step web navigation and task completion, our dataset evaluates information extraction, multimodal retrieval and composition of information from many web pages. WebQuest includes three question categories: single-screen QA, multi-screen QA, and QA based on navigation traces. We evaluate leading proprietary multimodal models like GPT-4V, Gemini Flash, Claude 3, and open source models like InstructBLIP, PaliGemma on our dataset, revealing a significant gap between single-screen and multi-screen reasoning. Finally, we investigate inference time techniques like Chain-of-Thought prompting to improve model capabilities on multi-screen reasoning.

We report results of magnetization and $^{19}$F NMR measurements in the normal state of as-grown LaO$_{0.5}$F$_{0.5}$BiS$_2$. The magnetization is dominated by a temperature-independent diamagnetic component and a field- and temperature-dependent paramagnetic contribution $M_\mu(H,T)$ from a $\sim$1000~ppm concentration of local moments, an order of magnitude higher than can be accounted for by measured rare-earth impurity concentrations. $M_\mu(H,T)$ can be fit by the Brillouin function $B_J(x)$ or, perhaps more realistically, a two-level $\tanh(x)$ model for magnetic Bi $6p$ ions in defect crystal fields. Both fits require a phenomenological Curie-Weiss argument $x = \mu_\mathrm{eff}H/(T + T_W)$, $T_W \approx 1.7$ K. There is no evidence for magnetic order down to 2 K, and the origin of $T_W$ is not clear. $^{19}$F frequency shifts, linewidths, and spin-lattice relaxation rates are consistent with purely dipolar $^{19}$F/defect-spin interactions. The defect-spin correlation time $\tau_c(T)$ obtained from $^{19}$F spin-lattice relaxation rates obeys the Korringa relation $\tau_cT = \text{const.}$, indicating the relaxation is dominated by conduction-band fluctuations., Comment: Corrected citations, 15 pages, 15 figures, 6 tables, 69 references, corresponding author O. O. Bernal: obernal@calstatela.edu

Complex multi-step reasoning tasks, such as solving mathematical problems or generating code, remain a significant hurdle for even the most advanced large language models (LLMs). Verifying LLM outputs with an Outcome Reward Model (ORM) is a standard inference-time technique aimed at enhancing the reasoning performance of LLMs. However, this still proves insufficient for reasoning tasks with a lengthy or multi-hop reasoning chain, where the intermediate outcomes are neither properly rewarded nor penalized. Process supervision addresses this limitation by assigning intermediate rewards during the reasoning process. To date, the methods used to collect process supervision data have relied on either human annotation or per-step Monte Carlo estimation, both prohibitively expensive to scale, thus hindering the broad application of this technique. In response to this challenge, we propose a novel divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named \textit{OmegaPRM} for the efficient collection of high-quality process supervision data. This algorithm swiftly identifies the first error in the Chain of Thought (CoT) with binary search and balances the positive and negative examples, thereby ensuring both efficiency and quality. As a result, we are able to collect over 1.5 million process supervision annotations to train Process Reward Models (PRMs). This fully automated process supervision alongside the weighted self-consistency algorithm is able to enhance LLMs' math reasoning performances. We improved the success rates of the instruction-tuned Gemini Pro model from 51\% to 69.4\% on MATH500 and from 86.4\% to 93.6\% on GSM8K. Similarly, we boosted the success rates of Gemma2 27B from 42.3\% to 58.2\% on MATH500 and from 74.0\% to 92.2\% on GSM8K. The entire process operates without any human intervention or supervision, making our method both financially and ..., Comment: 17 pages, 5 figures, 2 table

Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing a dramatic increase in latency. In this paper, we propose a novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Specifically, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. It is notable that Sparse RAG combines the assessment of each individual document and the generation of the response into a single process. The designed sparse mechanism in a RAG system can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, filtering out undesirable contexts enhances the model's focus on relevant context, inherently improving its generation quality. Evaluation results of two datasets show that Sparse RAG can strike an optimal balance between generation quality and computational efficiency, demonstrating its generalizability across both short- and long-form generation tasks.

We present a combined growth and transport study of superconducting single-crystalline NdO$_{0.5}$F$_{0.5}$BiS$_2$. Evidence of two-dimensional superconductivity with significant phase fluctuations of preformed Cooper pairs preceding the superconducting transition is reported. This result is based on three key observations. (1) The resistive superconducting transition temperature $T_c$ (defined by resistivity $\rho \rightarrow 0$) increases with increasing disorder. (2) As $T\rightarrow T_c$, the conductivity diverges significantly faster than what is expected from Gaussian fluctuations in two and three dimensions. (3) Non-Ohmic resistance behavior is observed in the superconducting state. Altogether, our observations are consistent with a temperature regime of phase-fluctuating superconductivity. The crystal structure with magnetic ordering tendencies in the NdO$_{0.5}$F$_{0.5}$ layers and (super)conductivity in the BiS$_2$ layers is likely responsible for the two-dimensional phase fluctuations. As such, NdO$_{0.5}$F$_{0.5}$BiS$_2$ falls into the class of unconventional ``laminar" bulk superconductors that include cuprate materials and 4Hb-TaS$_2$.

Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of these reward signals - typically, there is only a single reward for an entire output. This sparsity of rewards can lead to inefficient and unstable learning. To address this challenge, our paper introduces an novel framework that utilizes the critique capability of Large Language Models (LLMs) to produce intermediate-step rewards during RL training. Our method involves coupling a policy model with a critic language model, which is responsible for providing comprehensive feedback of each part of the output. This feedback is then translated into token or span-level rewards that can be used to guide the RL training process. We investigate this approach under two different settings: one where the policy model is smaller and is paired with a more powerful critic model, and another where a single language model fulfills both roles. We assess our approach on three text generation tasks: sentiment control, language model detoxification, and summarization. Experimental results show that incorporating artificial intrinsic rewards significantly improve both sample efficiency and the overall performance of the policy model, supported by both automatic and human evaluation.

The recently discovered (Ba,Sr)Ni$_2$As$_2$ family provides an ideal platform for investigating the interaction between electronic nematicity and superconductivity. Here we report the muon spin relaxation ($\mu$SR) measurements on BaNi$_2$As$_2$. Transverse-field $\mu$SR experiments indicate that the temperature dependence of superfluid density is best fitted with a single-band $s$-wave model. On the other hand, the magnetic penetration depth $\lambda$ shows magnetic field dependence, which contradicts with the single-band fully-gapped scenario. Zero-field $\mu$SR experiments indicate the absence of spontaneous magnetic field in the superconducting state, showing the preservation of time-reversal symmetry in the superconducting state. Our $\mu$SR experiments suggest that BaNi$_2$As$_2$ is a fully-gapped multiband superconductor. The superconducting gap amplitudes of each band are nearly the same while different bands exhibit different coherence lengths. The present work helps to elucidate the controversial superconducting property of this parent compound, paving the way for further research on doping the system with Sr to enhance superconductivity., Comment: Accepted by Phys. Rev. B

The recently discovered superconductivity with critical temperature $T_c$ up to 80 K in the double-layer Nickelate La$_3$Ni$_2$O$_{7-\delta}$ under pressure has drawn great attention. Here we report the positive muon spin relaxation ($\mu^+$SR) study of polycrystalline La$_3$Ni$_2$O$_{6.92}$ under ambient pressure. Zero-field $\mu^+$SR experiments reveal the existence of magnetic order in La$_3$Ni$_2$O$_{6.92}$ with $T_{N}=154\ \rm{K}$. The weak transverse field $\mu^+$SR measurements confirms the bulk nature of magnetism. In addition, a small quantity of oxygen deficiencies can greatly broaden the internal magnetic field distribution sensed by muons.

Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach that leverages Large Language Models (LLMs) to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses. Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval's significant potential in the realm of natural language system evaluation.

Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters, where all parameters are used to adapt certain task. We found this less effective empirically using the example of LoRA that introducing more trainable parameters does not help. Motivated by this we investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank adaption. SiRA leverages the Sparse Mixture of Expert(SMoE) to boost the performance of LoRA. Specifically it enforces the top $k$ experts routing with a capacity limit restricting the maximum number of tokens each expert can process. We propose a novel and simple expert dropout on top of gating network to reduce the over-fitting issue. Through extensive experiments, we verify SiRA performs better than LoRA and other mixture of expert approaches across different single tasks and multitask settings.

Critical thinking is essential for rational decision-making and problem-solving. This skill hinges on the ability to provide precise and reasoned critiques and is a hallmark of human intelligence. In the era of large language models (LLMs), this study explores the ability of LLMs to deliver accurate critiques across various tasks. We are interested in this topic as a capable critic model could not only serve as a reliable evaluator, but also as a source of supervised signals for model tuning. Particularly, if a model can self-critique, it has the potential for autonomous self-improvement. To examine this, we introduce a unified evaluation framework for assessing the critique abilities of LLMs. We develop a benchmark called CriticBench, which comprises 3K high-quality natural language queries and corresponding model responses; and annotate the correctness of these responses. The benchmark cover tasks such as math problem-solving, code completion, and question answering. We evaluate multiple LLMs on the collected dataset and our analysis reveals several noteworthy insights: (1) Critique is generally challenging for most LLMs, and this capability often emerges only when models are sufficiently large. (2) In particular, self-critique is especially difficult. Even top-performing LLMs struggle to achieve satisfactory performance. (3) Models tend to have lower critique accuracy on problems where they are most uncertain. To this end, we introduce a simple yet effective baseline named self-check, which leverages self-critique to improve task performance for various models. We hope this study serves as an initial exploration into understanding the critique abilities of LLMs, and aims to inform future research, including the development of more proficient critic models and the application of critiques across diverse tasks.

We report the results of muon spin rotation and relaxation ($\rm \mu$SR) measurements on the recently discovered layered Cu-based superconducting material La$_{2}($Cu$_{1-x}$Ni$_{x}$)$_{5}$As$_{3}$O$_{2}$ ($x =$ 0.40, 0.45). Transverse-field $\rm \mu$SR experiments on both samples show that the temperature dependence of superfluid density is best described by a two-band model. The absolute values of zero-temperature magnetic penetration depth $\lambda_{\rm ab}(0)$ were found to be 427(1.7) nm and 422(1.5) nm for $x =$ 0.40 and 0.45, respectively. Both compounds are located between the unconventional and the standard BCS superconductors in the Uemura plot. No evidence of time-reversal symmetry (TRS) breaking in the superconducting state is suggested by zero-field $\rm \mu$SR measurements.

The structure and magnetic properties of rare-earth ions Tm$^{3+}$ kagom\'{e} lattice Tm$_3$Sb$_3$Mg$_2$O$_{14}$ are studied by X-ray diffraction, magnetic susceptibility and muon spin relaxation ($\mu$SR) experiments. The existence of a small amount of Tm/Mg site-mixing disorder is revealed. DC magnetic susceptibility measurement shows that Tm$^{3+}$ magnetic moments are antiferromagnetically correlated with a negative Curie-Weiss temperature of -26.3 K. Neither long-range magnetic order nor spin-glass transition is observed by DC and AC magnetic susceptibility, and confirmed by $\mu$SR experiment down to 0.1 K. However, the emergence of short-range magnetic order is indicated by the zero-field $\mu$SR experiments, and the absence of spin dynamics at low temperatures is evidenced by the longitudinal-field $\mu$SR technique. Compared with the results of Tm$_3$Sb$_3$Zn$_2$O$_{14}$, another Tm-based kagom\'{e} lattice with much more site-mixing disorder, the gapless spin liquid like behaviors in Tm$_3$Sb$_3$Zn$_2$O$_{14}$ can be induced by disorder effect. Samples with perfect geometrical frustration are in urgent demand to establish whether QSL exits in this kind of materials with rare-earth kagom\'{e} lattice.

Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting. Nonetheless, the large sizes of these models make them impractical for on-device inference, which would otherwise allow for enhanced privacy and economical inference. Creating a smaller yet potent language model for text rewriting presents a formidable challenge because it requires balancing the need for a small size with the need to retain the emergent capabilities of the LLM, that requires costly data collection. To address the above challenge, we introduce a new instruction tuning approach for building a mobile-centric text rewriting model. Our strategies enable the generation of high quality training data without any human labeling. In addition, we propose a heuristic reinforcement learning framework which substantially enhances performance without requiring preference data. To further bridge the performance gap with the larger server-side model, we propose an effective approach that combines the mobile rewrite agent with the server model using a cascade. To tailor the text rewriting tasks to mobile scenarios, we introduce MessageRewriteEval, a benchmark that focuses on text rewriting for messages through natural language instructions. Through empirical experiments, we demonstrate that our on-device model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size. Notably, we show that our proposed cascading approach improves model performance.

In this paper we revisit $D_s \to f_0$ form factors from the light-cone sum rules with the light meson light-cone distribution amplitudes. The main motivation of this study is the differential decay width of $D_s \to \left[\pi\pi \right]_{\rm S} e \nu_e$ measured recently by BESIII collaboration and the $D_s \to f_0$ form factor extracted under the intermediate resonant model. Our result of the differential width of $D_s^+ \to f_0 (\to \left[ \pi\pi \right]_{\rm S}) e^+ \nu_e$ decay obtained under the narrow width approximation is a litter bit lower than the data, the result obtained under the resonant Flatt\'e model is in consistent with the data while shows a litter bit larger, indicating a sizable mixing $\sim 20\degree$ between ${\bar s}s$ and ${\bar u}u+{\bar d}d$ of $f_0$. In order to obtain a model independent prediction, we suggest to calculate $D_s \to \left[ \pi\pi \right]_{\rm S}$ form factors with the isoscalar scalar dipion light-cone distribution amplitudes. Our calculation of $D_s \to \left[ \pi\pi \right]_{\rm S}$ form factors is carried out at the leading twist level due to the finite knowledge of dipion system, the result of differential width shows a moderate evolution in contrast to that obtained from the narrow width approximation and the Flatt\'e model, revealing a bright prospect to study the four-body leptonic decays of heavy mesons with the dimeson light-cone distribution amplitudes., Comment: 23 pages, 14 figures, 7 tables

Large Language Models (LLMs) have demonstrated impressive capabilities in creative tasks such as storytelling and E-mail generation. However, as LLMs are primarily trained on final text results rather than intermediate revisions, it might be challenging for them to perform text rewriting tasks. Most studies in the rewriting tasks focus on a particular transformation type within the boundaries of single sentences. In this work, we develop new strategies for instruction tuning and reinforcement learning to better align LLMs for cross-sentence rewriting tasks using diverse wording and structures expressed through natural languages including 1) generating rewriting instruction data from Wiki edits and public corpus through instruction generation and chain-of-thought prompting; 2) collecting comparison data for reward model training through a new ranking function. To facilitate this research, we introduce OpenRewriteEval, a novel benchmark covers a wide variety of rewriting types expressed through natural language instructions. Our results show significant improvements over a variety of baselines. The public repository is available on GitHub under Google Research (https://github.com/google-research/google-research/tree/master/rewritelm).

Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach., Comment: EMNLP 2022

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.