Search

Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. However, such a paradigm fails to comprehensively differentiate the fine-grained language and cognitive skills, rendering the lack of sufficient interpretation to LLMs' capabilities. In this paper, we present FAC$^2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation. Specifically, we formulate LLMs' evaluation in a multi-dimensional and explainable manner by dissociating the language-related capabilities and the cognition-related ones. Besides, through extracting the intermediate reasoning from LLMs, we further break down the process of applying a specific capability into three sub-steps: recalling relevant knowledge, utilizing knowledge, and solving problems. Finally, FAC$^2$E evaluates each sub-step of each fine-grained capability, providing a two-faceted diagnosis for LLMs. Utilizing FAC$^2$E, we identify a common shortfall in knowledge utilization among models and propose a straightforward, knowledge-enhanced method to mitigate this issue. Our results not only showcase promising performance enhancements but also highlight a direction for future LLM advancements., Comment: Accepted at EMNLP 2024 main conference

This paper addresses the challenge of train-short-test-long (TSTL) scenarios in Large Language Models (LLMs) equipped with Rotary Position Embedding (RoPE), where models pre-trained on shorter sequences face difficulty with out-of-distribution (OOD) token positions in longer sequences. We introduce Resonance RoPE, a novel approach designed to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions, significantly improving the model performance without additional online computational costs. Furthermore, we present PosGen, a new synthetic benchmark specifically designed for fine-grained behavior analysis in TSTL scenarios, aiming to isolate the constantly increasing difficulty of token generation on long contexts from the challenges of recognizing new token positions. Our experiments on synthetic tasks show that after applying Resonance RoPE, Transformers recognize OOD position better and more robustly. Our extensive LLM experiments also show superior performance after applying Resonance RoPE to the current state-of-the-art RoPE scaling method, YaRN, on both upstream language modeling tasks and a variety of downstream long-text applications., Comment: 13 pages, 4 figures, accepted at ACL 2024 Findings

Large Language Model (LLM)-based agents have shown effectiveness across many applications. However, their use in data science scenarios requiring solving long-term interconnected tasks, dynamic data adjustments and domain expertise remains challenging. Previous approaches primarily focus on individual tasks, making it difficult to assess the complete data science workflow. Moreover, they struggle to handle real-time changes in intermediate data and fail to adapt dynamically to evolving task dependencies inherent to data science problems. In this paper, we present Data Interpreter, an LLM-based agent designed to automatically solve various data science problems end-to-end. Our Data Interpreter incorporates two key modules: 1) Hierarchical Graph Modeling, which breaks down complex problems into manageable subproblems, enabling dynamic node generation and graph optimization; and 2) Programmable Node Generation, a technique that refines and verifies each subproblem to iteratively improve code generation results and robustness. Extensive experiments consistently demonstrate the superiority of Data Interpreter. On InfiAgent-DABench, it achieves a 25% performance boost, raising accuracy from 75.9% to 94.9%. For machine learning and open-ended tasks, it improves performance from 88% to 95%, and from 60% to 97%, respectively. Moreover, on the MATH dataset, Data Interpreter achieves remarkable performance with a 26% improvement compared to state-of-the-art baselines. The code is available at https://github.com/geekan/MetaGPT.

In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

Layered transition metal chalcogenides have stimulated a wide research interest due to their many exotic physical properties. In this paper, we studied the magnetotransport properties of the exfoliated TaNiTe5, a recently discovered Dirac nodal-line semimetal. A giant positive magnetoresistance (MR) is observed when the current is parallel to the crystallographic c axis, while it is strongly diminished when the current flows along the a axis. The observed giant MR is gradually suppressed either on reducing the thickness of nanoflake or on increasing temperature. By performing MR measurement in tilted magnetic fields, the interlayer coupling is found to be weakened both by reducing the thickness and by increasing temperature. We propose a mechanism of electron-electron interaction-assisted interlayer transport as a origin of the giant MR. The mechanism is likely to provide a explanation for the giant MR in other layered materials., Comment: 21 pages, 7 figures, 1 table

[Abridged] The extension of conceptual DFT to include external fields in chemical systems is utilised to investigate the effects of strong magnetic fields on the electronic charge distribution and its consequences on the reactivity of $\pi$-systems. Formaldehyde, H$_2$CO, is considered as a prototypical example and current-DFT calculations are used to evaluate the electric dipole moment together with the electron density and the Fukui functions, which provide insight into how H$_2$CO behaves chemically in a magnetic field. In particular, the symmetries of these quantities are analysed based on group, representation, and corepresentation theories using QSym$^2$. This allows us to leverage the simple symmetry constraints on the macroscopic electric dipole moments to make profound predictions on the more nuanced symmetry transformation properties of the microscopic frontier MOs, electron densities, and Fukui functions. This is especially useful for complex-valued MOs in magnetic fields whose detailed symmetry analyses lead us to define the new concepts of modular and phasal symmetry breaking. Through these, the connection between the vanishing constraints on the electric dipole moments and the symmetry of electron densities and Fukui functions can be formalised, and the inability of the magnetic field in all three orientations considered to induce asymmetry with respect to the molecular plane can be understood from a molecular perspective. Furthermore, the detailed forms of the Fukui functions reveal remarkable reversals in the direction of the C=O dipole moment in the presence of a parallel or perpendicular magnetic field, which can be attributed to the mixing between frontier MOs due to their subduced symmetries in magnetic fields. The findings in this work are also discussed in the wider context of a long-standing debate on the possibility to create enantioselectivity by external fields., Comment: Main text: 26 pages (two-column), 7 figures, 9 tables; Supplementary information: 7 pages (one-column), 4 figures, 2 tables

The Center for Underground Physics of the Institute for Basic Science (IBS) in Korea has been planning the construction of a deep underground laboratory since 2013 to search for extremely rare interactions such as dark matter and neutrinos. In September 2022, a new underground laboratory, Yemilab, was finally completed in Jeongseon, Gangwon Province, with a depth of 1,000 m and an exclusive experimental area spanning 3,000 m$^3$. The tunnel is encased in limestone and accommodates 17 independent experimental spaces. Over two years, from 2023 to 2024, the Yangyang Underground Laboratory facilities will be relocated to Yemilab. Preparations are underway for the AMoRE-II, a neutrinoless double beta decay experiment, scheduled to begin in Q2 2024 at Yemilab. Additionally, Yemilab includes a cylindrical pit with a volume of approximately 6,300 m$^3$, designed as a multipurpose laboratory for next-generation experiments involving neutrinos, dark matter, and related research. This article provides a focused overview of the construction and structure of Yemilab., Comment: 12 pages, 3 figures, 1 table

Higher-order and fractional discrete time crystals (DTCs) are exotic phases of matter where the discrete time translation symmetry is broken into higher-order and non-integer category. Generation of these unique DTCs has been widely studied theoretically in different systems. However, no current experimental methods can probe these higher-order and fractional DTCs in any quantum many-body systems. We demonstrate an experimental approach to observe higher-order and fractional DTCs in Floquet-driven Rydberg atomic gases. We have discovered multiple $n$-DTCs with integer values of $n$ = 2, 3, and 4, and others ranging up to 14, along with fractional $n$-DTCs with $n$ values beyond the integers. The system response can transition between adjacent integer DTCs, during which the fractional DTCs are investigated. Study of higher-order and fractional DTCs expands fundamental knowledge of non-equilibrium dynamics and is promising for discovery of more complex temporal symmetries beyond the single discrete time translation symmetry., Comment: 17 pages, 10 figures, to be published in Nature Communications