Search

Large Language Models (LLMs) have significantly transformed our daily life and established a new paradigm in natural language processing (NLP). However, the predominant pretraining of LLMs on extensive web-based texts remains insufficient for advanced scientific discovery, particularly in chemistry. The scarcity of specialized chemistry data, coupled with the complexity of multi-modal data such as 2D graph, 3D structure and spectrum, present distinct challenges. Although several studies have reviewed Pretrained Language Models (PLMs) in chemistry, there is a conspicuous absence of a systematic survey specifically focused on chemistry-oriented LLMs. In this paper, we outline methodologies for incorporating domain-specific chemistry knowledge and multi-modal information into LLMs, we also conceptualize chemistry LLMs as agents using chemistry tools and investigate their potential to accelerate scientific research. Additionally, we conclude the existing benchmarks to evaluate chemistry ability of LLMs. Finally, we critically examine the current challenges and identify promising directions for future research. Through this comprehensive survey, we aim to assist researchers in staying at the forefront of developments in chemistry LLMs and to inspire innovative applications in the field., Comment: COLING2025,We maintain an up-to-date Github repository at: https://github.com/OpenDFM/LLM4Chemistry

Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.

In this paper, we consider the following Hardy-type mean field equation \[ \left\{ {\begin{array}{*{20}{c}} { - \Delta u-\frac{1}{(1-|x|^2)^2} u = \lambda e^u}, & {\rm in} \ \ B_1,\\ {\ \ \ \ u = 0,} &\ {\rm on}\ \partial B_1, \end{array}} \right. \] \[\] where $\lambda>0$ is small and $B_1$ is the standard unit disc of $\mathbb{R}^2$. Applying the moving plane method of hyperbolic space and the accurate expansion of heat kernel on hyperbolic space, we establish the radial symmetry and Brezis-Merle lemma for solutions of Hardy-type mean field equation. Meanwhile, we also derive the quantitative results for solutions of Hardy-type mean field equation, which improves significantly the compactness results for classical mean-field equation obtained by Brezis-Merle and Li-Shafrir. Furthermore, applying the local Pohozaev identity from scaling, blow-up analysis and a contradiction argument, we prove that the solutions are unique when $\lambda$ is sufficiently close to 0., Comment: 37 pages. arXiv admin note: text overlap with arXiv:2412.16890

In this paper, we first establish the quantitative properties for positive solutions to the Moser-Trudinger equations in the two-dimensional Poincar\'e disk $\mathbb{B}^2$: \begin{equation*}\label{mt1} \left\{ \begin{aligned} &-\Delta_{\mathbb{B}^2}u=\lambda ue^{u^2},\ x\in\mathbb{B}^2, &u\to0,\ \text{when}\ \rho(x)\to\infty, &||\nabla_{\mathbb{B}^2} u||_{L^2(\mathbb{B}^2)}^2\leq M_0, \end{aligned} \right. \end{equation*} where $0<\lambda<\frac{1}{4}=\inf\limits_{u\in W^{1,2}(\mathbb{B}^2)\backslash\{0\}}\frac{\|\nabla_{\mathbb{B}^2}u\|_{L^2(\mathbb{B}^2)}^2}{\|u\|_{L^2(\mathbb{B}^2)}^2}$, $\rho(x)$ denotes the geodesic distance between $x$ and the origin and $M_0$ is a fixed large positive constant (see Theorem 1.1). Furthermore, by doing a delicate expansion for Dirichlet energy $\|\nabla_{\mathbb{B}^2}u\|_{L^2(\mathbb{B}^2)}^2$ when $\lambda$ approaches to $0,$ we prove that there exists $\Lambda^\ast>4\pi$ such that the Moser-Trudinger functional $F(u)=\int_{\mathbb{B}^2}\left(e^{u^2}-1\right) dV_{\mathbb{B}^2}$ under the constraint $\int_{\mathbb{B}^2}|\nabla_{\mathbb{B}^2}u|^2 dV_{\mathbb{B}^2}=\Lambda$ has at least one positive critical point for $\Lambda\in(4\pi,\Lambda^{\ast})$ up to some M\"{o}bius transformation. Finally, when $\lambda\rightarrow 0$, by doing a more accurate expansion for $u$ near the origin and away from the origin, applying a local Pohozaev identity around the origin and the uniqueness of the Cauchy initial value problem for ODE,Cauchy-initial uniqueness for ODE, we prove that the Moser-Trudinger equation only has one positive solution when $\lambda$ is close to $0.$ During the process of the proofs, we overcome some new difficulties which involves the decay properties of the positive solutions, as well as some precise expansions for the solutions both near the origin and away from the origin., Comment: 54 pages

Conjugate heat transfer is a challenging fluid-structure coupling problem that can significantly influence flame stabilization and thermoacoustic instabilities. To properly capture combustion phenomena that involve conjugate heat transfer, careful modeling of chemical reactions in the fluid domain and heat transfer in the solid body is necessary and remains an active research topic. To this end, we derived a strongly-coupled method with a monolithic weak formulation to investigate the conjugate heat transfer between an anchored flame and a thermal conductive cylinder by means of linear stability analysis and resolvent analysis. We conduct parameter continuation with the Damkohler number to construct a bifurcation diagram and identify multiple baseflow states, including blow-off, anchored flame, and flashback. Linear stability analysis reveals the presence of a single unstable, non-oscillatory eigenmode for the base states on the anchored flame branch. This eigenmode plays a pivotal role in driving the bifurcation. Subsequently, resolvent analysis is performed to examine the amplification behavior of the fluid-solid coupled system under external forcing, showing that heat fluctuations are maximized when heat transfer between the fluid and solid is minimized.

The rise of "bedroom producers" has democratized music creation, while challenging producers to objectively evaluate their work. To address this, we present AI TrackMate, an LLM-based music chatbot designed to provide constructive feedback on music productions. By combining LLMs' inherent musical knowledge with direct audio track analysis, AI TrackMate offers production-specific insights, distinguishing it from text-only approaches. Our framework integrates a Music Analysis Module, an LLM-Readable Music Report, and Music Production-Oriented Feedback Instruction, creating a plug-and-play, training-free system compatible with various LLMs and adaptable to future advancements. We demonstrate AI TrackMate's capabilities through an interactive web interface and present findings from a pilot study with a music producer. By bridging AI capabilities with the needs of independent producers, AI TrackMate offers on-demand analytical feedback, potentially supporting the creative process and skill development in music production. This system addresses the growing demand for objective self-assessment tools in the evolving landscape of independent music production., Comment: Accepted for the NeurIPS 2024 Creative AI Track

Large Language Models (LLMs) have extended their capabilities beyond language generation to interact with external systems through tool calling, offering powerful potential for real-world applications. However, the phenomenon of tool hallucinations, which occur when models improperly select or misuse tools, presents critical challenges that can lead to flawed task execution and increased operational costs. This paper investigates the concept of reliable tool calling and highlights the necessity of addressing tool hallucinations. We systematically categorize tool hallucinations into two main types: tool selection hallucination and tool usage hallucination. To mitigate these issues, we propose a reliability-focused alignment framework that enhances the model's ability to accurately assess tool relevance and usage. By proposing a suite of evaluation metrics and evaluating on StableToolBench, we further demonstrate the effectiveness of our framework in mitigating tool hallucination and improving the overall system reliability of LLM tool calling.

The increasing context window size in Large Language Models (LLMs), such as the GPT and LLaMA series, has improved their ability to tackle complex, long-text tasks, but at the cost of inference efficiency, particularly regarding memory and computational complexity. Existing methods, including selective token retention and window-based attention, improve efficiency but risk discarding important tokens needed for future text generation. In this paper, we propose an approach that enhances LLM efficiency without token loss by reducing the memory and computational load of less important tokens, rather than discarding them.We address two challenges: 1) investigating the distribution of important tokens in the context, discovering recent tokens are more important than distant tokens in context, and 2) optimizing resources for distant tokens by sharing attention scores across layers. The experiments show that our method saves $35\%$ KV cache without compromising the performance., Comment: preprint

The increasing complexity and cost of clinical trials, particularly in the context of oncology and advanced therapies, pose significant challenges for drug development. This study evaluates the predictive capabilities of large language models (LLMs) such as GPT-3.5, GPT-4, and HINT in determining clinical trial outcomes. By leveraging a curated dataset of trials from ClinicalTrials.gov, we compare the models' performance using metrics including balanced accuracy, specificity, recall, and Matthews Correlation Coefficient (MCC). Results indicate that GPT-4o demonstrates robust performance in early trial phases, achieving high recall but facing limitations in specificity. Conversely, the HINT model excels in recognizing negative outcomes, particularly in later trial phases, offering a balanced approach across diverse endpoints. Oncology trials, characterized by high complexity, remain challenging for all models. Additionally, trial duration and disease categories influence predictive performance, with longer durations and complex diseases such as neoplasms reducing accuracy. This study highlights the complementary strengths of LLMs and HINT, providing insights into optimizing predictive tools for clinical trial design and risk management. Future advancements in LLMs are essential to address current gaps in handling negative outcomes and complex domains.

The retinal fundus images are utilized extensively in the diagnosis, and their quality can directly affect the diagnosis results. However, due to the insufficient dataset and algorithm application, current fundus image quality assessment (FIQA) methods are not powerful enough to meet ophthalmologists` demands. In this paper, we address the limitations of datasets and algorithms in FIQA. First, we establish a new FIQA dataset, Fundus Quality Score(FQS), which includes 2246 fundus images with two labels: a continuous Mean Opinion Score varying from 0 to 100 and a three-level quality label. Then, we propose a FIQA Transformer-based Hypernetwork (FTHNet) to solve these tasks with regression results rather than classification results in conventional FIQA works. The FTHNet is optimized for the FIQA tasks with extensive experiments. Results on our FQS dataset show that the FTHNet can give quality scores for fundus images with PLCC of 0.9423 and SRCC of 0.9488, significantly outperforming other methods with fewer parameters and less computation complexity.We successfully build a dataset and model addressing the problems of current FIQA methods. Furthermore, the model deployment experiments demonstrate its potential in automatic medical image quality control. All experiments are carried out with 10-fold cross-validation to ensure the significance of the results., Comment: 11 pages, 7 figures

Current mobile assistants are limited by dependence on system APIs or struggle with complex user instructions and diverse interfaces due to restricted comprehension and decision-making abilities. To address these challenges, we propose MobA, a novel Mobile phone Agent powered by multimodal large language models that enhances comprehension and planning capabilities through a sophisticated two-level agent architecture. The high-level Global Agent (GA) is responsible for understanding user commands, tracking history memories, and planning tasks. The low-level Local Agent (LA) predicts detailed actions in the form of function calls, guided by sub-tasks and memory from the GA. Integrating a Reflection Module allows for efficient task completion and enables the system to handle previously unseen complex tasks. MobA demonstrates significant improvements in task execution efficiency and completion rate in real-life evaluations, underscoring the potential of MLLM-empowered mobile assistants., Comment: 27 pages, 6 figures, and 5 tables. We will release our source code in a few days

Large language models (LLMs) have demonstrated remarkable performance, particularly in multilingual contexts. While recent studies suggest that LLMs can transfer skills learned in one language to others, the internal mechanisms behind this ability remain unclear. We observed that the neuron activation patterns of LLMs exhibit similarities when processing the same language, revealing the existence and location of key linguistic regions. Additionally, we found that neuron activation patterns are similar when processing sentences with the same semantic meaning in different languages. This indicates that LLMs map semantically identical inputs from different languages into a "Lingua Franca", a common semantic latent space that allows for consistent processing across languages. This semantic alignment becomes more pronounced with training and increased model size, resulting in a more language-agnostic activation pattern. Moreover, we found that key linguistic neurons are concentrated in the first and last layers of LLMs, becoming denser in the first layers as training progresses. Experiments on BLOOM and LLaMA2 support these findings, highlighting the structural evolution of multilingual LLMs during training and scaling up. This paper provides insights into the internal workings of LLMs, offering a foundation for future improvements in their cross-lingual capabilities., Comment: 16 pages, 11 figures, 4 tables

Accurate and affordable indoor 3D reconstruction is critical for effective robot navigation and interaction. Traditional LiDAR-based mapping provides high precision but is costly, heavy, and power-intensive, with limited ability for novel view rendering. Vision-based mapping, while cost-effective and capable of capturing visual data, often struggles with high-quality 3D reconstruction due to sparse point clouds. We propose ES-Gaussian, an end-to-end system using a low-altitude camera and single-line LiDAR for high-quality 3D indoor reconstruction. Our system features Visual Error Construction (VEC) to enhance sparse point clouds by identifying and correcting areas with insufficient geometric detail from 2D error maps. Additionally, we introduce a novel 3DGS initialization method guided by single-line LiDAR, overcoming the limitations of traditional multi-view setups and enabling effective reconstruction in resource-constrained environments. Extensive experimental results on our new Dreame-SR dataset and a publicly available dataset demonstrate that ES-Gaussian outperforms existing methods, particularly in challenging scenarios. The project page is available at https://chenlu-china.github.io/ES-Gaussian/., Comment: This preprint has been withdrawn due to concerns regarding the originality of certain technical elements, as well as its basis in a company project report that was intended solely for internal discussions. To avoid any potential misunderstandings, we have decided to withdraw this submission from public access. We apologize for any confusion this may have caused

Similarity search finds objects that are similar to a given query object based on a similarity metric. As the amount and variety of data continue to grow, similarity search in metric spaces has gained significant attention. Metric spaces can accommodate any type of data and support flexible distance metrics, making similarity search in metric spaces beneficial for many real-world applications, such as multimedia retrieval, personalized recommendation, trajectory analytics, data mining, decision planning, and distributed servers. However, existing studies mostly focus on indexing metric spaces on a single machine, which faces efficiency and scalability limitations with increasing data volume and query amount. Recent advancements in similarity search turn towards distributed methods, while they face challenges including inefficient local data management, unbalanced workload, and low concurrent search efficiency. To this end, we propose DIMS, an efficient Distributed Index for similarity search in Metric Spaces. First, we design a novel three-stage heterogeneous partition to achieve workload balance. Then, we present an effective three-stage indexing structure to efficiently manage objects. We also develop concurrent search methods with filtering and validation techniques that support efficient distributed similarity search. Additionally, we devise a cost-based optimization model to balance communication and computation cost. Extensive experiments demonstrate that DIMS significantly outperforms existing distributed similarity search approaches.

This paper presents a method to evaluate the alignment between the decision-making logic of Large Language Models (LLMs) and human cognition in a case study on legal LLMs. Unlike traditional evaluations on language generation results, we propose to evaluate the correctness of the detailed decision-making logic of an LLM behind its seemingly correct outputs, which represents the core challenge for an LLM to earn human trust. To this end, we quantify the interactions encoded by the LLM as primitive decision-making logic, because recent theoretical achievements have proven several mathematical guarantees of the faithfulness of the interaction-based explanation. We design a set of metrics to evaluate the detailed decision-making logic of LLMs. Experiments show that even when the language generation results appear correct, a significant portion of the internal inference logic contains notable issues.

Text summarization tasks commonly employ Pre-trained Language Models (PLMs) to fit diverse standard datasets. While these PLMs excel in automatic evaluations, they frequently underperform in human evaluations, indicating a deviation between their generated summaries and human summarization preferences. This discrepancy is likely due to the low quality of fine-tuning datasets and the limited availability of high-quality human-annotated data that reflect true human preference. To address this challenge, we introduce a novel human summarization preference alignment framework AlignSum. This framework consists of three parts: Firstly, we construct a Data Pymarid with extractive, abstractive, and human-annotated summary data. Secondly, we conduct the Gaussian Resampling to remove summaries with extreme lengths. Finally, we implement the two-stage hierarchical fine-tuning with Data Pymarid after Gaussian Resampling. We apply AlignSum to PLMs on the human-annotated CNN/DailyMail and BBC XSum datasets. Experiments show that with AlignSum, PLMs like BART-Large surpass 175B GPT-3 in both automatic and human evaluations. This demonstrates that AlignSum significantly enhances the alignment of language models with human summarization preferences., Comment: EMNLP2024 Findings, code at: https://github.com/csyanghan/AlignSum

Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0., Comment: 12 pages, 1 figure, 9 tables. Technical Report, accepted by NeurIPS 2024 Workshop FM4Science

In this work, we aim to establish a detailed description of the physical mechanisms of intrinsic thermoacoustic modes and their interplay with duct acoustic modes. The biglobal linear stability analysis of an anchored laminar flame in an acoustic duct is carried out by exploiting the linearized compressible reactive flow equations. The pure intrinsic thermoacoustic mode is first identified and characterized under conditions where the eigenfrequencies of duct acoustic modes are sufficiently high and no mode interplay is present. Parameter variations, considering the inflow Mach number and acoustic reflection coefficients, are conducted to study the behaviors of eigenmode trajectories in the proximity of an exceptional point. Near the exceptional point, trajectory veering and mode switching are observed, and the extreme sensitivity renders the mode trajectory highly dependent on the choices of parameter and parameter variation path. One important finding is that different parameter variation paths can lead to inconsistent results in mode origin identification. Hence, we propose to characterize the thermoacoustic modes based on the flow structures. Specifically, the Helmholtz decomposition is employed to extract the potential and solenoidal components of the thermoacoustic modes. Intrinsic thermoacoustic modes and duct acoustic modes exhibit distinct flow structures, which are clearly distinguishable through the decomposed fields. Near the exceptional point, thermoacoustic modes display characteristics common to both intrinsic and duct modes. The results from compressible reactive flow analysis reveal that the flow structure of eigenmodes offers new insights into intrinsic thermoacoustic modes and mode interplay near the exceptional point.

Retrieval-augmented generation (RAG) has emerged as a popular solution to mitigate the hallucination issues of large language models. However, existing studies on RAG seldom address the issue of predictive uncertainty, i.e., how likely it is that a RAG model's prediction is incorrect, resulting in uncontrollable risks in real-world applications. In this work, we emphasize the importance of risk control, ensuring that RAG models proactively refuse to answer questions with low confidence. Our research identifies two critical latent factors affecting RAG's confidence in its predictions: the quality of the retrieved results and the manner in which these results are utilized. To guide RAG models in assessing their own confidence based on these two latent factors, we develop a counterfactual prompting framework that induces the models to alter these factors and analyzes the effect on their answers. We also introduce a benchmarking procedure to collect answers with the option to abstain, facilitating a series of experiments. For evaluation, we introduce several risk-related metrics and the experimental results demonstrate the effectiveness of our approach. Our code and benchmark dataset are available at https://github.com/ict-bigdatalab/RC-RAG.

Rapid developments of AI tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI., Comment: 19 pages, 7 figures, 11 tables

In this paper, we investigate the phase behaviors and dynamics of self-propelled particles with active reorientation in double-well potential. We observe the self-propelled particles exhibit flocking and clustering in an asymmetric potential trap. By MD simulations, we obtain a phase diagram of flocking with active reorientation and potential asymmetry as parameters. We compare the responses of inactive and active particles to the potential. It shows that active reorientation of particles amplifies the degree of aggregation on one side in the asymmetric potential well. Furthermore, we calculate the mean squared displacement and identify distinct diffusion regimes. These results highlight active particles with active reorientation exhibit greater sensitivity in double-well potentials., Comment: 7 pages, 8 figures

The hydrodynamic interactions among bacterial cell bodies, flagella, and surrounding boundaries are essential for understanding bacterial motility in complex environments. In this study, we demonstrate that each slender flagellum can be modeled as a series of spheres, and that the interactions between these spheres can be accurately characterized using a resistance matrix. This approach allows us to effectively and efficiently evaluate the propulsive effects of the flagella. Notably, our investigation into bacterial motility near a colloidal sphere reveals significant discrepancies between results derived from the twin multipole moment and those obtained through resistive force theory. Consequently, neglecting the hydrodynamic interactions among cell bodies, flagella, and colloidal spheres may lead to substantial inaccuracies. Our model simplifies bacteria into a series of spheres, making it well-suited for examining bacterial motility near spherical boundaries, as well as the nonlinear deformation dynamics of elastic flagella.

We introduce a novel self-supervised deep clustering approach tailored for unstructured data without requiring prior knowledge of the number of clusters, termed Adaptive Self-supervised Robust Clustering (ASRC). In particular, ASRC adaptively learns the graph structure and edge weights to capture both local and global structural information. The obtained graph enables us to learn clustering-friendly feature representations by an enhanced graph auto-encoder with contrastive learning technique. It further leverages the clustering results adaptively obtained by robust continuous clustering (RCC) to generate prototypes for negative sampling, which can further contribute to promoting consistency among positive pairs and enlarging the gap between positive and negative samples. ASRC obtains the final clustering results by applying RCC to the learned feature representations with their consistent graph structure and edge weights. Extensive experiments conducted on seven benchmark datasets demonstrate the efficacy of ASRC, demonstrating its superior performance over other popular clustering models. Notably, ASRC even outperforms methods that rely on prior knowledge of the number of clusters, highlighting its effectiveness in addressing the challenges of clustering unstructured data.

The persistent challenge of formulating ontic structuralism in a rigorous manner, which prioritizes structures over the entities they contain, calls for a transformation of traditional logical frameworks. I argue that Univalent Foundations (UF), which feature the axiom that all isomorphic structures are identical, offer such a foundation and are more attractive than other proposed structuralist frameworks. Furthermore, I delve into the significance in the case of the hole argument and, very briefly, the nature of symmetries.

Symmetries and isomorphisms play similar conceptual roles when we consider how models represent physical situations, but they are formally distinct, as two models related by symmetries are not typically isomorphic. I offer a rigorous categorical strategy that formulate symmetries as isomorphisms between models and apply it to classical electromagnetism, and evaluate its philosophical significance in relation to the recent debate between `sophistication' and `reduction'. In addition to traditional spacetime models, I also consider algebraic models, in which case we can use the method of natural operators to address the problem of ontological nonperspicuity faced by the categorical strategy. Finally, I briefly expound on the significance of symmetries as isomorphisms in the framework of Univalent Foundations, in which isomorphic structures are formally identified.

Plane adjustment (PA) is crucial for many 3D applications, involving simultaneous pose estimation and plane recovery. Despite recent advancements, it remains a challenging problem in the realm of multi-view point cloud registration. Current state-of-the-art methods can achieve globally optimal convergence only with good initialization. Furthermore, their high time complexity renders them impractical for large-scale problems. To address these challenges, we first exploit a novel optimization strategy termed \textit{Bi-Convex Relaxation}, which decouples the original problem into two simpler sub-problems, reformulates each sub-problem using a convex relaxation technique, and alternately solves each one until the original problem converges. Building on this strategy, we propose two algorithmic variants for solving the plane adjustment problem, namely \textit{GlobalPointer} and \textit{GlobalPointer++}, based on point-to-plane and plane-to-plane errors, respectively. Extensive experiments on both synthetic and real datasets demonstrate that our method can perform large-scale plane adjustment with linear time complexity, larger convergence region, and robustness to poor initialization, while achieving similar accuracy as prior methods. The code is available at https://github.com/wu-cvgl/GlobalPointer., Comment: Accepted to ECCV 2024. The first two authors contributed equally to this work. Code: https://github.com/wu-cvgl/GlobalPointer

Data science and engineering workflows often span multiple stages, from warehousing to orchestration, using tools like BigQuery, dbt, and Airbyte. As vision language models (VLMs) advance in multimodal understanding and code generation, VLM-based agents could potentially automate these workflows by generating SQL queries, Python code, and GUI operations. This automation can improve the productivity of experts while democratizing access to large-scale data analysis. In this paper, we introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering workflows, featuring 494 real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks, derived from real-world use cases, evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems. To balance realistic simulation with evaluation simplicity, we devote significant effort to developing automatic configurations for task setup and carefully crafting evaluation metrics for each task. Furthermore, we supplement multimodal agents with comprehensive documents of these enterprise data software systems. Our empirical evaluation reveals that existing state-of-the-art LLM/VLM-based agents do not reliably automate full data workflows (14.0% success). Even with step-by-step guidance, these agents still underperform in tasks that require fine-grained, knowledge-intensive GUI actions (16.2%) and involve remote cloud-hosted workspaces (10.6%). We hope that Spider2-V paves the way for autonomous multimodal agents to transform the automation of data science and engineering workflow. Our code and data are available at https://spider2-v.github.io., Comment: 34 pages, 14 figures, 10 tables

Urban redevelopment often involves a delicate balance between enhancing regional connectivity and preserving local social fabric. Through a case study in Guangzhou, China, we argue that demolishing a historic street to construct a new subway station shows competing interests between local government's priority to facilitate spatial connectivity and locals' priority to maintain a place for social interaction and memories. We measure the social costs of the new subway station through a network lens, focusing on the loss of social ties and memories and low travel benefits of the new station for the local populations. We find that 1) the demolition will remove many small businesses that support locals' daily activities, social ties, and memories, and 2) the new station reduces travel distance and increases route options for passengers from other areas of the city more than locals nearby the demolition site. Our results contribute to a network-based framework and methodology to understand and contest inequality in expanding transportation network infrastructure in cities.

Retrieval-augmented Large Language Models (LLMs) have reshaped traditional query-answering systems, offering unparalleled user experiences. However, existing retrieval techniques often struggle to handle multi-modal query contexts. In this paper, we present an interactive Multi-modal Query Answering (MQA) system, empowered by our newly developed multi-modal retrieval framework and navigation graph index, integrated with cutting-edge LLMs. It comprises five core components: Data Preprocessing, Vector Representation, Index Construction, Query Execution, and Answer Generation, all orchestrated by a dedicated coordinator to ensure smooth data flow from input to answer generation. One notable aspect of MQA is its utilization of contrastive learning to assess the significance of different modalities, facilitating precise measurement of multi-modal information similarity. Furthermore, the system achieves efficient retrieval through our advanced navigation graph index, refined using computational pruning techniques. Another highlight of our system is its pluggable processing framework, allowing seamless integration of embedding models, graph indexes, and LLMs. This flexibility provides users diverse options for gaining insights from their multi-modal knowledge base. A preliminary video introduction of MQA is available at https://youtu.be/xvUuo2ZIqWk., Comment: This demo paper has been accepted by VLDB 2024

Large language models have demonstrated their capabilities in storyline creation and human-like character role-playing. Current language model agents mainly focus on reasonable behaviors from the level of individuals, and their behaviors might be hard to constraint on the level of the whole storyline. In this paper we introduce IBSEN, a director-actor coordinate agent framework that generates drama scripts and makes the plot played by agents more controllable. The director agent writes plot outlines that the user desires to see, instructs the actor agents to role-play their characters, and reschedules the plot when human players participate in the scenario to ensure the plot is progressing towards the objective. To evaluate the framework, we create a novel drama plot that involves several actor agents and check the interactions between them under the instruction of the director agent. Evaluation results show that our framework could generate complete, diverse drama scripts from only a rough outline of plot objectives, meanwhile maintaining the characteristics of characters in the drama. Our codes and prompts are available at https://github.com/OpenDFM/ibsen., Comment: Accepted by ACL 2024 Main

The emergence of Vision Language Models (VLMs) has brought unprecedented advances in understanding multimodal information. The combination of textual and visual semantics in VLMs is highly complex and diverse, making the safety alignment of these models challenging. Furthermore, due to the limited study on the safety alignment of VLMs, there is a lack of large-scale, high-quality datasets. To address these limitations, we propose a Safety Preference Alignment dataset for Vision Language Models named SPA-VL. In terms of breadth, SPA-VL covers 6 harmfulness domains, 13 categories, and 53 subcategories, and contains 100,788 samples of the quadruple (question, image, chosen response, rejected response). In terms of depth, the responses are collected from 12 open- (e.g., QwenVL) and closed-source (e.g., Gemini) VLMs to ensure diversity. The experimental results indicate that models trained with alignment techniques on the SPA-VL dataset exhibit substantial improvements in harmlessness and helpfulness while maintaining core capabilities. SPA-VL, as a large-scale, high-quality, and diverse dataset, represents a significant milestone in ensuring that VLMs achieve both harmlessness and helpfulness. We have made our code https://github.com/EchoseChen/SPA-VL-RLHF and SPA-VL dataset url https://huggingface.co/datasets/sqrti/SPA-VL publicly available.

Large language models have ushered in a new era of artificial intelligence research. However, their substantial training costs hinder further development and widespread adoption. In this paper, inspired by the redundancy in the parameters of large language models, we propose a novel training paradigm: Evolving Subnetwork Training (EST). EST samples subnetworks from the layers of the large language model and from commonly used modules within each layer, Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP). By gradually increasing the size of the subnetworks during the training process, EST can save the cost of training. We apply EST to train GPT2 model and TinyLlama model, resulting in 26.7\% FLOPs saving for GPT2 and 25.0\% for TinyLlama without an increase in loss on the pre-training dataset. Moreover, EST leads to performance improvements in downstream tasks, indicating that it benefits generalization. Additionally, we provide intuitive theoretical studies based on training dynamics and Dropout theory to ensure the feasibility of EST. Our code is available at https://github.com/OpenDFM/EST., Comment: Accepted to ICML 2024

Building generalist agents that can handle diverse tasks and evolve themselves across different environments is a long-term goal in the AI community. Large language models (LLMs) are considered a promising foundation to build such agents due to their generalized capabilities. Current approaches either have LLM-based agents imitate expert-provided trajectories step-by-step, requiring human supervision, which is hard to scale and limits environmental exploration; or they let agents explore and learn in isolated environments, resulting in specialist agents with limited generalization. In this paper, we take the first step towards building generally-capable LLM-based agents with self-evolution ability. We identify a trinity of ingredients: 1) diverse environments for agent exploration and learning, 2) a trajectory set to equip agents with basic capabilities and prior knowledge, and 3) an effective and scalable evolution method. We propose AgentGym, a new framework featuring a variety of environments and tasks for broad, real-time, uni-format, and concurrent agent exploration. AgentGym also includes a database with expanded instructions, a benchmark suite, and high-quality trajectories across environments. Next, we propose a novel method, AgentEvol, to investigate the potential of agent self-evolution beyond previously seen data across tasks and environments. Experimental results show that the evolved agents can achieve results comparable to SOTA models. We release the AgentGym suite, including the platform, dataset, benchmark, checkpoints, and algorithm implementations. The AgentGym suite is available on https://github.com/WooooDyy/AgentGym., Comment: Project site: https://agentgym.github.io

Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pre-training and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging \emph{sparsity} in pre-trained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that Sparsity-Accelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a $45\%$ throughput improvement in continual pre-training and saves $38\%$ training time in supervised fine-tuning in practice. It offers a simple, hardware-agnostic, and easily deployable framework for additional LLM training. Our code is available at https://github.com/OpenDFM/SAT., Comment: Accepted to ACL 2024 Findings

In this paper, we are concerned with the following $n$-Laplacian mean field equation \[ \left\{ {\begin{array}{*{20}{c}} { - \Delta_n u = \lambda e^u} & {\rm in} \ \ \Omega, \\ {\ \ \ \ u = 0} &\ {\rm on}\ \partial \Omega, \end{array}} \right. \] \[\] where $\Omega$ is a smooth bounded domain of $\mathbb{R}^n \ (n\geq 2)$ and $- \Delta_n u =-{\rm div}(|\nabla u|^{n-2}\nabla u)$. We first establish the quantization property of solutions to the above $n$-Laplacian mean field equation. As an application, combining the Pohozaev identity and the capacity estimate, we obtain the sharp constant $C(n)$ of the Moser-Onofri inequality in the $n$-dimensional unit ball $B^n:=B^n(0,1)$, $$\mathop {\inf }\limits_{u \in W_0^{1,n}(B^n)}\frac{1}{ n C_n}\int_{B^n} | \nabla u|^n dx- \ln \int_{B^n} {e^u} dx\geq C(n),$$ which extends the result of Caglioti-Lions-Marchioro-Pulvirenti in \cite{Caglioti} to the case of $n$-dimensional ball. Here $C_n=(\frac{n^2}{n-1})^{n-1} \omega_{n-1}$ and $\omega_{n-1}$ is the surface measure of $B^n$. For the Moser-Onofri inequality in a general bounded domain of $\mathbb{R}^n$, we apply the technique of $n$-harmonic transplantation to give the optimal concentration level of the Moser-Onofri inequality and obtain the criterion for the existence and non-existence of extremals for the Moser-Onofri inequality.

We propose a semiparametric model for dyadic link formations in directed networks. The model contains a set of degree parameters that measure different effects of popularity or outgoingness across nodes, a regression parameter vector that reflects the homophily effect resulting from the nodal attributes or pairwise covariates associated with edges, and a set of latent random noises with unknown distributions. Our interest lies in inferring the unknown degree parameters and homophily parameters. The dimension of the degree parameters increases with the number of nodes. Under the high-dimensional regime, we develop a kernel-based least squares approach to estimate the unknown parameters. The major advantage of our estimator is that it does not encounter the incidental parameter problem for the homophily parameters. We prove consistency of all the resulting estimators of the degree parameters and homophily parameters. We establish high-dimensional central limit theorems for the proposed estimators and provide several applications of our general theory, including testing the existence of degree heterogeneity, testing sparse signals and recovering the support. Simulation studies and a real data application are conducted to illustrate the finite sample performance of the proposed methods., Comment: 28 pages, 3 figures

Recently, Dolbeault-Esteban-Figalli-Frank-Loss [20] established the optimal stability of the first-order Sobolev inequality with dimension-dependent constant. Subsequently, Chen-Lu-Tang [18] obtained the optimal stability for the fractional Sobolev inequality of order $s$ when $0

Recently, Large Language Models (LLMs) have been demonstrated to possess impressive capabilities in a variety of domains and tasks. We investigate the issue of prompt design in the multi-turn text-to-SQL task and attempt to enhance the LLMs' reasoning capacity when generating SQL queries. In the conversational context, the current SQL query can be modified from the preceding SQL query with only a few operations due to the context dependency. We introduce our method called CoE-SQL which can prompt LLMs to generate the SQL query based on the previously generated SQL query with an edition chain. We also conduct extensive ablation studies to determine the optimal configuration of our approach. Our approach outperforms different in-context learning baselines stably and achieves state-of-the-art performances on two benchmarks SParC and CoSQL using LLMs, which is also competitive to the SOTA fine-tuned models.