Search

Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition., Comment: 20 pages, 7 figures

We describe the planned near-term and potential longer-term upgrades of the Belle II detector at the SuperKEKB electron-positron collider operating at the KEK laboratory in Tsukuba, Japan. These upgrades will allow increasingly sensitive searches for possible new physics beyond the Standard Model in flavor, tau, electroweak and dark sector physics that are both complementary to and competitive with the LHC and other experiments., Comment: Editor: F. Forti 170 pages

Let $X$ be an arbitrary set. Then a topology $t$ on $X$ is said to be completely useful if every upper semicontinuous linear (total) preorder $\precsim$ on $X$ can be represented by an upper semicontinuous real-valued order preserving function. In this paper, appealing, simple and new characterizations of completely useful topologies will be proved, therefore clarifying the structure of such topologies., Comment: Title changed

Theory of Mind (ToM) is the cognitive capability to perceive and ascribe mental states to oneself and others. Recent research has sparked a debate over whether large language models (LLMs) exhibit a form of ToM. However, existing ToM evaluations are hindered by challenges such as constrained scope, subjective judgment, and unintended contamination, yielding inadequate assessments. To address this gap, we introduce ToMBench with three key characteristics: a systematic evaluation framework encompassing 8 tasks and 31 abilities in social cognition, a multiple-choice question format to support automated and unbiased evaluation, and a build-from-scratch bilingual inventory to strictly avoid data leakage. Based on ToMBench, we conduct extensive experiments to evaluate the ToM performance of 10 popular LLMs across tasks and abilities. We find that even the most advanced LLMs like GPT-4 lag behind human performance by over 10% points, indicating that LLMs have not achieved a human-level theory of mind yet. Our aim with ToMBench is to enable an efficient and effective evaluation of LLMs' ToM capabilities, thereby facilitating the development of LLMs with inherent social intelligence., Comment: Under review

Let $\precsim$ be a reflexive binary relation on a topological space $(X, \tau )$. A pair $(u,v)$ of continuous real-valued functions on $(X, \tau )$ is said to be a {\em continuous representation} of $\precsim$ if, for all $x,y \in X$, [$(x \precsim y \Leftrightarrow u(x) \leq v(y))$]. In this paper we provide a characterization of the existence of a continuous representation of this kind in the general case when neither the functions $u$ and $v$ nor the topological space $(X,\tau )$ are required to satisfy any particular assumptions. Such characterization is based on a suitable continuity assumption of the binary relation $\precsim$, called {\em weak continuity}. In this way, we generalize all the previous results on the continuous representability of interval orders, and also of total preorders, as particular cases.

In the present paper we study necessary and sufficient conditions for the existence of a semicontinuous and finite Richter-Peleg multi-utility for a preorder. It is well know that, given a preorder on a topological space, if there is a lower (upper) semicontinuous Richter-Peleg multi-utility, then the topology of the space must be finer than the Upper (resp. Lower) topology. However, this condition does not guarantee the existence of a semicontinuous representation. We search for finer topologies which are necessary for semicontinuity, as well as that they could guarantee the existence of a semicontinuous representation. As a result, we prove that Scott topology (that refines the Upper one) must be contained in the topology of the space in case there exists a finite lower semicontinuous Richter-Peleg multi-utility. However, as it is shown, the existence of this representation cannot be guaranteed.

In the present paper a new concept of representability is introduced, which can be applied to not total and also to intransitive relations (semiorders in particular). This idea tries to represent the orderings in the simplest manner, avoiding any unnecessary information. For this purpose, the new concept of representability is developed by means of partial functions, so that other common definitions of representability (i.e. (Richter-Peleg) multi-utility, Scott-Suppes representability,...) are now particular cases in which the partial functions are actually functions. The paper also presents a collection of examples and propositions showing the advantages of this kind of representations, particularly in the case of partial orders and semiorders, as well as some results showing the connections between distinct kinds of representations.

Since the natural language processing (NLP) community started to make large language models (LLMs) act as a critic to evaluate the quality of generated texts, most of the existing works train a critique generation model on the evaluation data labeled by GPT-4's direct prompting. We observe that these models lack the ability to generate informative critiques in both pointwise grading and pairwise comparison especially without references. As a result, their generated critiques cannot provide fine-grained distinguishability on generated texts, causing unsatisfactory evaluation performance. In this paper, we propose a simple yet effective method called Eval-Instruct, which can first acquire pointwise grading critiques with pseudo references and then revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings, including pointwise grading and pairwise comparison with / without references. After fine-tuning on these data, the resulting model CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines and even achieve comparable evaluation performance to GPT-4 in system-level correlations of pointwise grading. We also demonstrate that our generated critiques can act as scalable feedback to further improve the generation quality of strong LLMs like ChatGPT., Comment: Accepted by ACL 2024 (Main Conference)

Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{https://github.com/THUDM/AlignBench}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab., Comment: Accepted to ACL 2024

The Belle II experiment at KEK in Japan considers upgrading its vertex detector system to address the challenges posed by high background levels caused by the increased luminosity of the SuperKEKB collider. One proposal for upgrading the vertex detector aims to install a 5-layer all monolithic pixel vertex detector based on fully depleted CMOS sensors in 2027. The new system will use the OBELIX MAPS chips to improve background robustness and reduce occupancy levels through small and fast pixels. This causes better track finding, especially for low transverse momenta tracks. This text will focus on the predecessor of the OBELIX sensor, the TJ-Monopix2, presenting laboratory and test beam results on pixel response, efficiency, and spatial resolution., Comment: 8 pages, 8 figures, Proceedings for 24th international Workshop on Radiation Imaging Detectors,25-29 JUNE 20233 Oslo

The Indy Autonomous Challenge (IAC) brought together for the first time in history nine autonomous racing teams competing at unprecedented speed and in head-to-head scenario, using independently developed software on open-wheel racecars. This paper presents the complete software architecture used by team TII EuroRacing (TII-ER), covering all the modules needed to avoid static obstacles, perform active overtakes and reach speeds above 75 m/s (270 km/h). In addition to the most common modules related to perception, planning, and control, we discuss the approaches used for vehicle dynamics modelling, simulation, telemetry, and safety. Overall results and the performance of each module are described, as well as the lessons learned during the first two events of the competition on oval tracks, where the team placed respectively second and third., Comment: Preprint: Accepted to Field Robotics "Opportunities and Challenges with Autonomous Racing" Special Issue

Traditional quantum system control methods often face different constraints, and are easy to cause both leakage and stochastic control errors under the condition of limited resources. Reinforcement learning has been proved as an efficient way to complete the quantum system control task. To learn a satisfactory control strategy under the condition of limited resources, a quantum system control method based on enhanced reinforcement learning (QSC-ERL) is proposed. The states and actions in reinforcement learning are mapped to quantum states and control operations in quantum systems. By using new enhanced neural networks, reinforcement learning can quickly achieve the maximization of long-term cumulative rewards, and a quantum state can be evolved accurately from an initial state to a target state. According to the number of candidate unitary operations, the three-switch control is used for simulation experiments. Compared with other methods, the QSC-ERL achieves close to 1 fidelity learning control of quantum systems, and takes fewer episodes to quantum state evolution under the condition of limited resources., Comment: 10 pages, 3 figures

Due to the large state space of the two-qubit system, and the adoption of ladder reward function in the existing quantum state preparation methods, the convergence speed is slow and it is difficult to prepare the desired target quantum state with high fidelity under limited conditions. To solve the above problems, a difference-driven reinforcement learning (RL) algorithm for quantum state preparation of two-qubit system is proposed by improving the reward function and action selection strategy. Firstly, a model is constructed for the problem of preparing quantum states of a two-qubit system, with restrictions on the type of quantum gates and the time for quantum state evolution. In the preparation process, a weighted differential dynamic reward function is designed to assist the algorithm quickly obtain the maximum expected cumulative reward. Then, an adaptive e-greedy action selection strategy is adopted to achieve a balance between exploration and utilization to a certain extent, thereby improving the fidelity of the final quantum state. The simulation results show that the proposed algorithm can prepare quantum state with high fidelity under limited conditions. Compared with other algorithms, it has different degrees of improvement in convergence speed and fidelity of the final quantum state., Comment: 12 pages, 9 figures

The rapid development of gravitational wave astronomy provides the unique opportunity of exploring the dynamics of the Universe using clustering properties of coalescing binary black hole mergers. Gravitational wave data, along with information coming from future galaxy surveys, have the potential of shedding light about many open questions in Cosmology, including those regarding the nature of dark matter and dark energy. In this work we explore which combination of gravitational wave and galaxy survey datasets are able to provide the best constraints both on modified gravity theories and on the nature of the very same binary black hole events. In particular, by using the public Boltzmann code \texttt{Multi\_CLASS}, we compare cosmological constraints on popular $\Lambda$CDM extensions coming from gravitational waves alone and in conjunction with either deep and localized or wide and shallow galaxy surveys. We show that constraints on extensions of General Relativity will be at the same level of existing limits from gravitational waves alone or one order of magnitude better when galaxy surveys are included. Furthermore, cross-correlating both kind of galaxy survey with gravitational waves datasets will allow to confidently rule in or out primordial black holes as dark matter candidate in the majority of the allowed parameter space., Comment: 34 pages, 7 figures. Matches the published version

Strain engineering is an efficient tool to tune and tailor the electrical and optical properties of 2D materials. The built-in strain can be tuned during the synthesis process of a two dimensional semiconductor, as molybdenum disulfide, by employing different growth substrate with peculiar thermal properties. In this work we demonstrate that the built-in strain of MoS2 monolayers, grown on SiO2/Si substrate using liquid precursors chemical vapor deposition, is mainly dominated by the size of the monolayer. In fact, we identify a critical size equal to 20 um, from which the built-in strain increases drastically. The built-in strain is maximized for 60 um sized monolayer, leading to 1.2% tensile strain with a partial release of strain close to the monolayer triangular vertexes due to formation of nanocracks. These findings also imply that the standard method for evaluation of the number of layers based on the Raman modes separation becomes unreliable for monolayer with a lateral size above 20 um.