Search

Large Language Models for Code (LLMs4Code) have been found to exhibit outstanding performance in the software engineering domain, especially the remarkable performance in coding tasks. However, even the most advanced LLMs4Code can inevitably contain incorrect or outdated code knowledge. Due to the high cost of training LLMs4Code, it is impractical to re-train the models for fixing these problematic code knowledge. Model editing is a new technical field for effectively and efficiently correcting erroneous knowledge in LLMs, where various model editing techniques and benchmarks have been proposed recently. Despite that, a comprehensive study that thoroughly compares and analyzes the performance of the state-of-the-art model editing techniques for adapting the knowledge within LLMs4Code across various code-related tasks is notably absent. To bridge this gap, we perform the first systematic study on applying state-of-the-art model editing approaches to repair the inaccuracy of LLMs4Code. To that end, we introduce a benchmark named CLMEEval, which consists of two datasets, i.e., CoNaLa-Edit (CNLE) with 21K+ code generation samples and CodeSearchNet-Edit (CSNE) with 16K+ code summarization samples. With the help of CLMEEval, we evaluate six advanced model editing techniques on three LLMs4Code: CodeLlama (7B), CodeQwen1.5 (7B), and Stable-Code (3B). Our findings include that the external memorization-based GRACE approach achieves the best knowledge editing effectiveness and specificity (the editing does not influence untargeted knowledge), while generalization (whether the editing can generalize to other semantically-identical inputs) is a universal challenge for existing techniques. Furthermore, building on in-depth case analysis, we introduce an enhanced version of GRACE called A-GRACE, which incorporates contrastive learning to better capture the semantics of the inputs., Comment: Accepted by ICSE2025. The code is available at: https://github.com/xpq-tech/code-llmedit.git

We propose a tensor-based criterion for benign landscape in phase retrieval and establish boundedness of gradient trajectories. This implies that gradient descent will converge to a global minimum for almost every initial point.

Majorana zero modes (MZMs) are the most intensively studied non-Abelian anyons. The Dirac fermion zero modes in topological insulators can be interpreted as the symmetry-protected "doubling" of the MZMs, suggesting an intrinsic connection between the quantum statistics of the two types of zero modes. Here we find that the minimal Kitaev chain model provides a unified characterization of the non-Abelian braiding statistics of both the MZMs and Dirac fermion zero modes under different parameter regimes. In particular, we introduce a minimal tri-junction setting based on the minimal Kitaev chain model and show it facilitates the unified scheme of braiding Dirac fermion zero modes, as well as the MZMs in the assistance of a Dirac mode. This unified minimal model unveils that the non-Abelian braiding of the MZMs can be continuously extended to the realm of Dirac fermion zero modes. The present study reveals deeper insights into the non-Abelian statistics and enables a broader search for non-Abelian anyons beyond the scope of MZMs., Comment: 6 pages, 3 figures

Large Language Models (LLMs) have recently demonstrated remarkable coding capabilities. However, assessing code generation based on well-formed properties and aligning it with developer preferences remains challenging. In this paper, we explore two key questions under the new challenge of code preference learning: (i) How do we train models to predict meaningful preferences for code? and (ii) How do human and LLM preferences align with verifiable code properties and developer code tastes? To this end, we propose CodeFavor, a framework for training pairwise code preference models from synthetic evolution data, including code commits and code critiques. To evaluate code preferences, we introduce CodePrefBench, a benchmark comprising 1364 rigorously curated code preference tasks to cover three verifiable properties-correctness, efficiency, and security-along with human preference. Our evaluation shows that CodeFavor holistically improves the accuracy of model-based code preferences by up to 28.8%. Meanwhile, CodeFavor models can match the performance of models with 6-9x more parameters while being 34x more cost-effective. We also rigorously validate the design choices in CodeFavor via a comprehensive set of controlled experiments. Furthermore, we discover the prohibitive costs and limitations of human-based code preference: despite spending 23.4 person-minutes on each task, 15.1-40.3% of tasks remain unsolved. Compared to model-based preference, human preference tends to be more accurate under the objective of code correctness, while being sub-optimal for non-functional objectives.

Precision gravitational wave measurement transforms research beyond general relativity and cosmology. Advances are made by applying quantum enhanced interferometry into the LIGO, Virgo and KAGRA detectors. Here, we develop an atomic sensor that employs a $p$-orbital Bose-Einstein condensate in an optical lattice to project gravitational wave signals into an orbital squeezed state. This entangled state couples linearly to the spacetime distortion signals received via a Michelson interferometer. Simulation data show that this sensor improves sensitivity over LIGO's quantum noise by approximately one order of magnitude and detection volume by $\sim 10^3$ in key frequency regimes. Additionally, it reduces the required laser power by five orders of magnitude. These results suggest that atomic orbital squeezing offers a compelling alternative to conventional techniques, offering a qualitatively different avenue for gravitational wave-based detection of dark matter, black holes, and the equation of state in neutron stars., Comment: 7+55 pages, 4+15 figures, 1 table

Knowledge editing has emerged as an efficient technology for updating the knowledge of large language models (LLMs), attracting increasing attention in recent years. However, there is a lack of effective measures to prevent the malicious misuse of this technology, which could lead to harmful edits in LLMs. These malicious modifications could cause LLMs to generate toxic content, misleading users into inappropriate actions. In front of this risk, we introduce a new task, Knowledge Editing Type Identification (KETI), aimed at identifying different types of edits in LLMs, thereby providing timely alerts to users when encountering illicit edits. As part of this task, we propose KETIBench, which includes five types of harmful edits covering most popular toxic types, as well as one benign factual edit. We develop four classical classification models and three BERT-based models as baseline identifiers for both open-source and closed-source LLMs. Our experimental results, across 42 trials involving two models and three knowledge editing methods, demonstrate that all seven baseline identifiers achieve decent identification performance, highlighting the feasibility of identifying malicious edits in LLMs. Additional analyses reveal that the performance of the identifiers is independent of the reliability of the knowledge editing methods and exhibits cross-domain generalization, enabling the identification of edits from unknown sources. All data and code are available in https://github.com/xpq-tech/KETI. Warning: This paper contains examples of toxic text., Comment: Under review

Secure pairing is crucial for ensuring the trustworthy deployment and operation of Internet of Things (IoT) devices. However, traditional pairing methods are often unsuitable for IoT devices due to their lack of conventional user interfaces, such as keyboards. Proximity-based pairing approaches are usable but vulnerable to exploitation by co-located malicious devices. While methods based on a user's physical operations (such as shaking) on IoT devices offer greater security, they typically rely on inertial sensors to sense the operations, which most IoT devices lack. We introduce a novel technique called Universal Operation Sensing, enabling IoT devices to sense the user's physical operations without the need for inertial sensors. With this technique, users can complete the pairing process within seconds using simple actions such as pressing a button or twisting a knob, whether they are holding a smartphone or wearing a smartwatch. Moreover, we reveal an inaccuracy issue in the fuzzy commitment protocol, which is frequently used for pairing. To address it, we propose an accurate pairing protocol, which does not use fuzzy commitment and incurs zero information loss. The comprehensive evaluation shows that it is secure, usable and efficient.

Linear control models have gained extensive application in vehicle control due to their simplicity, ease of use, and support for stability analysis. However, these models lack adaptability to the changing environment and multi-objective settings. Reinforcement learning (RL) models, on the other hand, offer adaptability but suffer from a lack of interpretability and generalization capabilities. This paper aims to develop a family of RL-based controllers enhanced by physics-informed policies, leveraging the advantages of both physics-based models (data-efficient and interpretable) and RL methods (flexible to multiple objectives and fast computing). We propose the Physics-Enhanced Residual Policy Learning (PERPL) framework, where the physics component provides model interpretability and stability. The learning-based Residual Policy adjusts the physics-based policy to adapt to the changing environment, thereby refining the decisions of the physics model. We apply our proposed model to decentralized control to mixed traffic platoon of Connected and Automated Vehicles (CAVs) and Human-driven Vehicles (HVs) using a constant time gap (CTG) strategy for cruising and incorporating actuator and communication delays. Experimental results demonstrate that our method achieves smaller headway errors and better oscillation dampening than linear models and RL alone in scenarios with artificially extreme conditions and real preceding vehicle trajectories. At the macroscopic level, overall traffic oscillations are also reduced as the penetration rate of CAVs employing the PERPL scheme increases.

Recent studies and industry advancements indicate that modular vehicles (MVs) have the potential to enhance transportation systems through their ability to dock and split en route. Although various applications of MVs have been explored across different domains, their use in logistics remains relatively underexplored. This study examines the application of MVs in cargo delivery to reduce costs. We model the delivery problem for MVs as a variant of the Vehicle Routing Problem, referred to as the Modular Vehicle Routing Problem (MVRP). In the MVRP, MVs can either serve customers independently or dock with other MVs to form a platoon, thereby reducing the average cost per unit. To tackle this problem, we first developed a Mixed Integer Linear Programming model, solvable by commercial optimization solvers. Given the problem's computational complexity, we also designed a Tabu Search (TS) algorithm with specialized neighborhood operators tailored for the MVRP. To escape local optima, multi-start and shaking strategies were incorporated into the TS algorithm. Additionally, we explored potential applications in logistics through various MVRP variants. The results of the numerical experiments indicate that the proposed algorithm successfully identifies all optimal solutions found by the MILP model in small-size benchmark instances, while also demonstrating good convergence speed in large-size benchmark instances. Comparative experiments show that the MVRP approach can reduce costs by approximately 5\% compared to traditional delivery methods. The code and data used in this study will be made available upon the acceptance of this paper.

The automated vehicle (AV) equipped with the Adaptive Cruise Control (ACC) system is expected to reduce the fuel consumption for the intelligent transportation system. This paper presents the Advanced ACC-Micro (AA-Micro) model, a new energy consumption model based on micro trajectory data, calibrated and verified by empirical data. Utilizing a commercial AV equipped with the ACC system as the test platform, experiments were conducted at the Columbus 151 Speedway, capturing data from multiple ACC and Human-Driven (HV) test runs. The calibrated AA-Micro model integrates features from traditional energy consumption models and demonstrates superior goodness of fit, achieving an impressive 90% accuracy in predicting ACC system energy consumption without overfitting. A comprehensive statistical evaluation of the AA-Micro model's applicability and adaptability in predicting energy consumption and vehicle trajectories indicated strong model consistency and reliability for ACC vehicles, evidenced by minimal variance in RMSE values and uniform RSS distributions. Conversely, significant discrepancies were observed when applying the model to HV data, underscoring the necessity for specialized models to accurately predict energy consumption for HV and ACC systems, potentially due to their distinct energy consumption characteristics.

Advancements in autonomous driving have increasingly focused on end-to-end (E2E) systems that manage the full spectrum of driving tasks, from environmental perception to vehicle navigation and control. This paper introduces V2X-VLM, an innovative E2E vehicle-infrastructure cooperative autonomous driving (VICAD) framework with Vehicle-to-Everything (V2X) systems and large vision-language models (VLMs). V2X-VLM is designed to enhance situational awareness, decision-making, and ultimate trajectory planning by integrating multimodel data from vehicle-mounted cameras, infrastructure sensors, and textual information. The contrastive learning method is further employed to complement VLM by refining feature discrimination, assisting the model to learn robust representations of the driving environment. Evaluations on the DAIR-V2X dataset show that V2X-VLM outperforms state-of-the-art cooperative autonomous driving methods, while additional tests on corner cases validate its robustness in real-world driving conditions.

We study proximal random reshuffling for minimizing the sum of locally Lipschitz functions and a proper lower semicontinuous convex function without assuming coercivity or the existence of limit points. The algorithmic guarantees pertaining to near approximate stationarity rely on a new tracking lemma linking the iterates to trajectories of conservative fields. One of the novelties in the analysis consists in handling conservative fields with unbounded values.

Motivated by the emergent reasoning capabilities of Vision Language Models (VLMs) and their potential to improve the comprehensibility of autonomous driving systems, this paper introduces a closed-loop autonomous driving controller called VLM-MPC, which combines the Model Predictive Controller (MPC) with VLM to evaluate how model-based control could enhance VLM decision-making. The proposed VLM-MPC is structured into two asynchronous components: The upper layer VLM generates driving parameters (e.g., desired speed, desired headway) for lower-level control based on front camera images, ego vehicle state, traffic environment conditions, and reference memory; The lower-level MPC controls the vehicle in real-time using these parameters, considering engine lag and providing state feedback to the entire system. Experiments based on the nuScenes dataset validated the effectiveness of the proposed VLM-MPC across various environments (e.g., night, rain, and intersections). The results demonstrate that the VLM-MPC consistently maintains Post Encroachment Time (PET) above safe thresholds, in contrast to some scenarios where the VLM-based control posed collision risks. Additionally, the VLM-MPC enhances smoothness compared to the real-world trajectories and VLM-based control. By comparing behaviors under different environmental settings, we highlight the VLM-MPC's capability to understand the environment and make reasoned inferences. Moreover, we validate the contributions of two key components, the reference memory and the environment encoder, to the stability of responses through ablation tests.

Given a $C^{1,1}_\mathrm{loc}$ lower bounded function $f:\mathbb{R}^n\rightarrow \mathbb{R}$ definable in an o-minimal structure on the real field, we show that the singular perturbation $\epsilon \searrow 0$ in the heavy ball system \begin{equation} \label{eq:P_eps} \tag{$P_\epsilon$} \epsilon\ddot{x}_\epsilon(t) + \gamma\dot{x}_\epsilon(t) + \nabla f(x_\epsilon(t)) = 0, ~~~ \forall t \geqslant 0, ~~~ x_\epsilon(0) = x_0, ~~~ \dot{x}_\epsilon(0) = \dot{x}_0, \end{equation} preserves boundedness of solutions, where $\gamma>0$ is the friction and $(x_0,\dot{x}_0) \in \mathbb{R}^n \times \mathbb{R}^n$ is the initial condition. This complements the work of Attouch, Goudou, and Redont which deals with finite time horizons. In other words, this work studies the asymptotic behavior of a ball rolling on a surface subject to gravitation and friction, without assuming convexity nor coercivity.

We consider an atom-cavity system having long-range atomic interactions mediated by cavity modes. It has been shown that quantum simulations of spin models with this system can naturally be used to solve number partition problems. Here, we present encoding schemes for numerous NP-complete problems, encompassing the majority of Karp's 21 NP-complete problems. We find a number of such computation problems can be encoded by the atom-cavity system at a linear cost of atom number. There are still certain problems that cannot be encoded by the atom-cavity as efficiently, such as quadratic unconstrained binary optimization (QUBO), and the Hamiltonian cycle. For these problems, we provide encoding schemes with a quadratic or quartic cost in the atom number. We expect this work to provide important guidance to search for the practical quantum advantage of the atom-cavity system in solving NP-complete problems. Moreover, the encoding schemes we develop here may also be adopted in other optical systems for solving NP-complete problems, where a similar form of Mattis-type spin glass Hamiltonian as in the atom-cavity system can be implemented., Comment: 25 pages, 1 table

Our study delves into Multimodal Entity Linking, aligning the mention in multimodal information with entities in knowledge base. Existing methods are still facing challenges like ambiguous entity representations and limited image information utilization. Thus, we propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding. The LLM, such as BLIP-2, extracts information relevant to entities in the image, which can facilitate improved extraction of entity features and linking them with the dynamic entity representations provided by ChatGPT. The experiments demonstrate that our proposed DIM method outperforms the majority of existing methods on the three original datasets, and achieves state-of-the-art (SOTA) on the dynamically enhanced datasets (Wiki+, Rich+, Diverse+). For reproducibility, our code and collected datasets are released on \url{https://github.com/season1blue/DIM}., Comment: Published on PRCV24

Developing quantum algorithms adaptive to specific constraints of near-term devices is an essential step towards practical quantum advantage. In a recent work [Phys. Rev. Lett. 131, 103601(2023)], we show cold atoms in an optical cavity can be built as a universal quantum optimizer with programmable all-to-all interactions, and the effective Hamiltonian for atoms directly encodes number partitioning problems (NPPs). Here, we numerically investigate the performance of quantum annealing (QA) and quantum approximate optimization algorithm (QAOA) to find the solution of NPP that is encoded in the ground state of atomic qubits. We find the success probability of the standard QA decays rapidly with the problem size. The optimized annealing path or inhomogeneous driving fields only lead to mild improvement on the success probability. Similarly, the standard QAOA always gets trapped in a false local minimum, and there is no significant performance improvement as we increase the depth of the quantum circuit. Inspired by the counterdiabatic driving, we propose an adaptive ansatz of QAOA which releases the parameter freedom of the NPP Hamiltonian to match higher-order counterdiabatic terms. Through numerical simulations, we find that our adaptive QAOA can achieve the optimal solution within very small circuit depth. It is thus worth paying the extra optimization cost of additional parameters for improving QAOA performance. Therefore, our adaptive QAOA provides a promising choice for programmable atom-cavity systems to demonstrate competitive computational power within its quantum coherence time.

In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly on small open-sourced LLMs. In this work, we propose a training framework that significantly improves self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating a number of explanations and refinement trajectories and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass@1 by up to 15.92% and pass@10 by 9.30% over four benchmarks. RL training brings additional up to 3.54% improvement on pass@1 and 2.55% improvement on pass@10. The trained LLMs show iterative refinement ability, and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs in source code.

Worldwide geolocalization aims to locate the precise location at the coordinate level of photos taken anywhere on the Earth. It is very challenging due to 1) the difficulty of capturing subtle location-aware visual semantics, and 2) the heterogeneous geographical distribution of image data. As a result, existing studies have clear limitations when scaled to a worldwide context. They may easily confuse distant images with similar visual contents, or cannot adapt to various locations worldwide with different amounts of relevant data. To resolve these limitations, we propose G3, a novel framework based on Retrieval-Augmented Generation (RAG). In particular, G3 consists of three steps, i.e., Geo-alignment, Geo-diversification, and Geo-verification to optimize both retrieval and generation phases of worldwide geolocalization. During Geo-alignment, our solution jointly learns expressive multi-modal representations for images, GPS and textual descriptions, which allows us to capture location-aware semantics for retrieving nearby images for a given query. During Geo-diversification, we leverage a prompt ensembling method that is robust to inconsistent retrieval performance for different image queries. Finally, we combine both retrieved and generated GPS candidates in Geo-verification for location prediction. Experiments on two well-established datasets IM2GPS3k and YFCC4k verify the superiority of G3 compared to other state-of-the-art methods. Our code and data are available online for reproduction., Comment: Accepted to NeurIPS2024

Multimodal aspect-based sentiment analysis (MABSA) aims to understand opinions in a granular manner, advancing human-computer interaction and other fields. Traditionally, MABSA methods use a joint prediction approach to identify aspects and sentiments simultaneously. However, we argue that joint models are not always superior. Our analysis shows that joint models struggle to align relevant text tokens with image patches, leading to misalignment and ineffective image utilization. In contrast, a pipeline framework first identifies aspects through MATE (Multimodal Aspect Term Extraction) and then aligns these aspects with image patches for sentiment classification (MASC: Multimodal Aspect-Oriented Sentiment Classification). This method is better suited for multimodal scenarios where effective image use is crucial. We present three key observations: (a) MATE and MASC have different feature requirements, with MATE focusing on token-level features and MASC on sequence-level features; (b) the aspect identified by MATE is crucial for effective image utilization; and (c) images play a trivial role in previous MABSA methods due to high noise. Based on these observations, we propose a pipeline framework that first predicts the aspect and then uses translation-based alignment (TBA) to enhance multimodal semantic consistency for better image utilization. Our method achieves state-of-the-art (SOTA) performance on widely used MABSA datasets Twitter-15 and Twitter-17. This demonstrates the effectiveness of the pipeline approach and its potential to provide valuable insights for future MABSA research. For reproducibility, the code and checkpoint will be released., Comment: Code will be released upon publication

Automated Vehicles (AVs) promise significant advances in transportation. Critical to these improvements is understanding AVs' longitudinal behavior, relying heavily on real-world trajectory data. Existing open-source trajectory datasets of AV, however, often fall short in refinement, reliability, and completeness, hindering effective performance metrics analysis and model development. This study addresses these challenges by creating a Unified Longitudinal TRAjectory dataset for AVs (Ultra-AV) to analyze their microscopic longitudinal driving behaviors. This dataset compiles data from 13 distinct sources, encompassing various AV types, test sites, and experiment scenarios. We established a three-step data processing: 1. extraction of longitudinal trajectory data, 2. general data cleaning, and 3. data-specific cleaning to obtain the longitudinal trajectory data and car-following trajectory data. The validity of the processed data is affirmed through performance evaluations across safety, mobility, stability, and sustainability, along with an analysis of the relationships between variables in car-following models. Our work not only furnishes researchers with standardized data and metrics for longitudinal AV behavior studies but also sets guidelines for data collection and model development., Comment: NA

This paper reviews the NTIRE 2024 RAW Image Super-Resolution Challenge, highlighting the proposed solutions and results. New methods for RAW Super-Resolution could be essential in modern Image Signal Processing (ISP) pipelines, however, this problem is not as explored as in the RGB domain. Th goal of this challenge is to upscale RAW Bayer images by 2x, considering unknown degradations such as noise and blur. In the challenge, a total of 230 participants registered, and 45 submitted results during thee challenge period. The performance of the top-5 submissions is reviewed and provided here as a gauge for the current state-of-the-art in RAW Image Super-Resolution., Comment: CVPR 2024 - NTIRE Workshop

Large language models (LLMs) have shown impressive effectiveness in various software engineering tasks, including automated program repair (APR). In this study, we take a deep dive into automated bug fixing utilizing LLMs. In contrast to many deep learning-based APR methods that assume known bug locations, rely on line-level localization tools, or address bug prediction and fixing in one step, our approach uniquely employs LLMs to predict bug location at the token level and subsequently utilizes them for bug fixing. This methodological separation of bug localization and fixing using different LLMs enables effective integration of diverse contextual information and improved incorporation of inductive biases. We introduce Toggle: Token-Granulated Bug Localization and Repair, a comprehensive program repair framework that integrates a bug localization model, an adjustment unit, and a bug-fixing model. Toggle takes a buggy function as input and generates a complete corrected function. We investigate various styles of prompting to the bug fixing model to identify the most effective prompts that better utilize the inductive bias and significantly outperform others. Toggle achieves the new state-of-the-art (SOTA) performance on the CodeXGLUE code refinement benchmark, and exhibits better and comparable performance on several other widely-used APR datasets, including Defects4J.

Dynamical quantum phase transitions, encompassing phenomena like many-body localization transitions and measurement-induced phase transitions, are often characterized and identified through the analysis of quantum entanglement. Here, we highlight that the dynamical phases defined by entanglement are further enriched by complexity. We investigate both the entanglement and Krylov complexity for fermions on regular graphs, which can be implemented by systems like $^6$Li atoms confined by optical tweezers. Our investigations unveil that while entanglement follows volume laws on both types of regular graphs with degree $d = 2$ and $d = 3$, the Krylov complexity exhibits distinctive behaviors. We analyze both free fermions and interacting fermions models. In the absence of interaction, both numerical results and theoretical analysis confirm that the dimension of the Krylov space scales as $D\sim N$ for regular graphs of degree $d = 2$ with $N$ sites, and we have $D\sim N^2$ for $d = 3$. The qualitative distinction also persists in interacting fermions on regular graphs. For interacting fermions, our theoretical analyses find the dimension scales as $D\sim 4^{N^\alpha}$ for regular graphs of $d = 2$ with $0.38\leq\alpha\leq0.59$, whereas it scales as $D\sim 4^N$ for $d = 3$. The distinction in the complexity of quantum dynamics for fermions on graphs with different connectivity can be probed in experiments by measuring the out-of-time-order correlators.

Multimodal entity linking (MEL) aims to utilize multimodal information (usually textual and visual information) to link ambiguous mentions to unambiguous entities in knowledge base. Current methods facing main issues: (1)treating the entire image as input may contain redundant information. (2)the insufficient utilization of entity-related information, such as attributes in images. (3)semantic inconsistency between the entity in knowledge base and its representation. To this end, we propose DWE+ for multimodal entity linking. DWE+ could capture finer semantics and dynamically maintain semantic consistency with entities. This is achieved by three aspects: (a)we introduce a method for extracting fine-grained image features by partitioning the image into multiple local objects. Then, hierarchical contrastive learning is used to further align semantics between coarse-grained information(text and image) and fine-grained (mention and visual objects). (b)we explore ways to extract visual attributes from images to enhance fusion feature such as facial features and identity. (c)we leverage Wikipedia and ChatGPT to capture the entity representation, achieving semantic enrichment from both static and dynamic perspectives, which better reflects the real-world entity semantics. Experiments on Wikimel, Richpedia, and Wikidiverse datasets demonstrate the effectiveness of DWE+ in improving MEL performance. Specifically, we optimize these datasets and achieve state-of-the-art performance on the enhanced datasets. The code and enhanced datasets are released on https://github.com/season1blue/DWET, Comment: under review on TOIS. arXiv admin note: substantial text overlap with arXiv:2312.11816

Current LiDAR-based Vehicle-to-Everything (V2X) multi-agent perception systems have shown the significant success on 3D object detection. While these models perform well in the trained clean weather, they struggle in unseen adverse weather conditions with the domain gap. In this paper, we propose a Domain Generalization based approach, named V2X-DGW, for LiDAR-based 3D object detection on multi-agent perception system under adverse weather conditions. Our research aims to not only maintain favorable multi-agent performance in the clean weather but also promote the performance in the unseen adverse weather conditions by learning only on the clean weather data. To realize the Domain Generalization, we first introduce the Adaptive Weather Augmentation (AWA) to mimic the unseen adverse weather conditions, and then propose two alignments for generalizable representation learning: Trust-region Weather-invariant Alignment (TWA) and Agent-aware Contrastive Alignment (ACA). To evaluate this research, we add Fog, Rain, Snow conditions on two publicized multi-agent datasets based on physics-based models, resulting in two new datasets: OPV2V-w and V2XSet-w. Extensive experiments demonstrate that our V2X-DGW achieved significant improvements in the unseen adverse weathers.

This paper introduces an online physical enhanced residual learning (PERL) framework for Connected Autonomous Vehicles (CAVs) platoon, aimed at addressing the challenges posed by the dynamic and unpredictable nature of traffic environments. The proposed framework synergistically combines a physical model, represented by Model Predictive Control (MPC), with data-driven online Q-learning. The MPC controller, enhanced for centralized CAV platoons, employs vehicle velocity as a control input and focuses on multi-objective cooperative optimization. The learning-based residual controller enriches the MPC with prior knowledge and corrects residuals caused by traffic disturbances. The PERL framework not only retains the interpretability and transparency of physics-based models but also significantly improves computational efficiency and control accuracy in real-world scenarios. The experimental results present that the online Q-learning PERL controller, in comparison to the MPC controller and PERL controller with a neural network, exhibits significantly reduced position and velocity errors. Specifically, the PERL's cumulative absolute position and velocity errors are, on average, 86.73% and 55.28% lower than the MPC's, and 12.82% and 18.83% lower than the neural network-based PERL's, in four tests with different reference trajectories and errors. The results demonstrate our advanced framework's superior accuracy and quick convergence capabilities, proving its effectiveness in maintaining platoon stability under diverse conditions.

This paper presents a comprehensive review of trajectory data of Advanced Driver Assistance System equipped-vehicle, with the aim of precisely model of Autonomous Vehicles (AVs) behavior. This study emphasizes the importance of trajectory data in the development of AV models, especially in car-following scenarios. We introduce and evaluate several datasets: the OpenACC Dataset, the Connected & Autonomous Transportation Systems Laboratory Open Dataset, the Vanderbilt ACC Dataset, the Central Ohio Dataset, and the Waymo Open Dataset. Each dataset offers unique insights into AV behaviors, yet they share common challenges in terms of data availability, processing, and standardization. After a series of data cleaning, outlier removal and statistical analysis, this paper transforms datasets of varied formats into a uniform standard, thereby improving their applicability for modeling AV car-following behavior. Key contributions of this study include: 1. the transformation of all datasets into a unified standard format, enhancing their utility for broad research applications; 2. a comparative analysis of these datasets, highlighting their distinct characteristics and implications for car-following model development; 3. the provision of guidelines for future data collection projects, along with the open-source release of all processed data and code for use by the research community., Comment: 6 pages, 2 figures

The general capabilities of large language models (LLMs) make them the infrastructure for various AI applications, but updating their inner knowledge requires significant resources. Recent model editing is a promising technique for efficiently updating a small amount of knowledge of LLMs and has attracted much attention. In particular, local editing methods, which directly update model parameters, are more suitable for updating a small amount of knowledge. Local editing methods update weights by computing least squares closed-form solutions and identify edited knowledge by vector-level matching in inference, which achieve promising results. However, these methods still require a lot of time and resources to complete the computation. Moreover, vector-level matching lacks reliability, and such updates disrupt the original organization of the model's parameters. To address these issues, we propose an detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching and adds them to the subject word embeddings in Transformer input. To get these editing embeddings, we propose optimizing then suppressing fusion method, which first optimizes learnable embedding vectors for the editing target and then suppresses the Knowledge Embedding Dimensions (KEDs) to obtain final editing embeddings. We thus propose SWEA$\oplus$OS method for editing factual knowledge in LLMs. We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$\oplus$OS on the \textsc{CounterFact} and zsRE datasets. To further validate the reasoning ability of SWEA$\oplus$OS in editing knowledge, we evaluate it on the more complex \textsc{RippleEdits} benchmark. The results demonstrate that SWEA$\oplus$OS possesses SOTA reasoning ability., Comment: Under review; Our code is available at https://github.com/xpq-tech/SWEA

Search engines are crucial as they provide an efficient and easy way to access vast amounts of information on the internet for diverse information needs. User queries, even with a specific need, can differ significantly. Prior research has explored the resilience of ranking models against typical query variations like paraphrasing, misspellings, and order changes. Yet, these works overlook how diverse demographics uniquely formulate identical queries. For instance, older individuals tend to construct queries more naturally and in varied order compared to other groups. This demographic diversity necessitates enhancing the adaptability of ranking models to diverse query formulations. To this end, in this paper, we propose a framework that integrates a novel rewriting pipeline that rewrites queries from various demographic perspectives and a novel framework to enhance ranking robustness. To be specific, we use Chain of Thought (CoT) technology to utilize Large Language Models (LLMs) as agents to emulate various demographic profiles, then use them for efficient query rewriting, and we innovate a robust Multi-gate Mixture of Experts (MMoE) architecture coupled with a hybrid loss function, collectively strengthening the ranking models' robustness. Our extensive experimentation on both public and industrial datasets assesses the efficacy of our query rewriting approach and the enhanced accuracy and robustness of the ranking model. The findings highlight the sophistication and effectiveness of our proposed model.

The precise estimation of macroscopic traffic parameters, such as travel time and fuel consumption, is essential for the optimization of traffic management systems. Despite its importance, the comprehensive acquisition of vehicle trajectory data for the calculation of these macroscopic measures presents a challenge. To bridge this gap, this study aims to calibrate car-following models capable of predicting both microscopic measures and macroscopic measures. We conduct a numerical analysis to trace the cumulative process of model prediction errors across various measurements, and our findings indicate that macroscopic measures encapsulate the accumulation of model errors. By incorporating macroscopic measures into vehicle model calibration, we can mitigate the impact of noise on microscopic data measurements. We compare three car-following model calibration methods: MiC (using microscopic measurements), MaC (using macroscopic measurements), and BiC (using both microscopic and macroscopic measurements): utilizing real-world trajectory data. The BiC method emerges as the most successful in reconstructing vehicle trajectories and accurately estimating travel time and fuel consumption, whereas the MiC method leads to overfitting and inaccurate macro-measurement predictions. This study underscores the importance of bi-scale calibration for precise traffic and energy consumption predictions, laying the groundwork for future research aimed at enhancing traffic management strategies.

There has been much recent progress in controlling $p$-orbital degrees of freedom in optical lattices, for example with lattice shaking, sublattice swapping, and lattice potential programming. Here, we present a protocol of preparing lowest Landau level (LLL) states of cold atoms by adiabatically compressing $p$-orbital Bose-Einstein condensates confined in two-dimensional optical lattices. The system starts from a chiral $p+ip$ Bose-Einstein condensate (BEC) state, which acquires finite angular momentum by spontaneous symmetry breaking. Such chiral BEC states have been achieved in recent optical lattice experiments for cold atoms loaded in the $p$-bands. Through an adiabatic adjustment of the lattice potential, we compress the three-dimensional BEC into a two-dimensional system, in which the orbital degrees of freedom continuously morph into LLL states. This process is enforced by the discrete rotation symmetry of the lattice potential. The final quantum state inherits large angular momentum from the original chiral $p+ip$ state, with one quantized unit per particle. We investigate the quantum many-body ground state of interacting bosons in the LLL considering contact repulsion. This leads to an exotic gapped BEC state. Our theory can be readily tested in experiments for the required techniques are all accessible to the current optical lattice experiments.

This review paper explores Multimodal Large Language Models (MLLMs), which integrate Large Language Models (LLMs) like GPT-4 to handle multimodal data such as text and vision. MLLMs demonstrate capabilities like generating image narratives and answering image-based questions, bridging the gap towards real-world human-computer interactions and hinting at a potential pathway to artificial general intelligence. However, MLLMs still face challenges in processing the semantic gap in multimodality, which may lead to erroneous generation, posing potential risks to society. Choosing the appropriate modality alignment method is crucial, as improper methods might require more parameters with limited performance improvement. This paper aims to explore modality alignment methods for LLMs and their existing capabilities. Implementing modality alignment allows LLMs to address environmental issues and enhance accessibility. The study surveys existing modal alignment methods in MLLMs into four groups: (1) Multimodal Converters that change data into something LLMs can understand; (2) Multimodal Perceivers to improve how LLMs perceive different types of data; (3) Tools Assistance for changing data into one common format, usually text; and (4) Data-Driven methods that teach LLMs to understand specific types of data in a dataset. This field is still in a phase of exploration and experimentation, and we will organize and update various existing research methods for multimodal information alignment.

Query expansion, pivotal in search engines, enhances the representation of user information needs with additional terms. While existing methods expand queries using retrieved or generated contextual documents, each approach has notable limitations. Retrieval-based methods often fail to accurately capture search intent, particularly with brief or ambiguous queries. Generation-based methods, utilizing large language models (LLMs), generally lack corpus-specific knowledge and entail high fine-tuning costs. To address these gaps, we propose a novel zero-shot query expansion framework utilizing LLMs for mutual verification. Specifically, we first design a query-query-document generation method, leveraging LLMs' zero-shot reasoning ability to produce diverse sub-queries and corresponding documents. Then, a mutual verification process synergizes generated and retrieved documents for optimal expansion. Our proposed method is fully zero-shot, and extensive experiments on three public benchmark datasets are conducted to demonstrate its effectiveness over existing methods. Our code is available online at https://github.com/Applied-Machine-Learning-Lab/MILL to ease reproduction., Comment: Accepted to NAACL 2024

Automated vehicle (AV) shuttles are emerging mobility technologies that have been widely piloted and deployed. Public attitude is critical to the deployment progress and the overall social benefits of automated vehicle (AV) technologies. The AV shuttle demonstration was regarded as a good way for possible attitude improvements. However, not all existing AV shuttle technologies are mature and reliable enough. Some frustrating uncomfort issues in AV shuttle demonstrations may adversely affect public attitudes of AV technologies. Studying the impact of the comfort of the demonstration on attitudinal change can help us provide guidance for future demonstrations. Thanks to the AV shuttle project piloted at Dunedin, Florida, this paper interviewed 161 people before and after taking an open road AV shuttle ride. In addition to the participant's demographic information, driving-related information and psychological-related information, we collected the AV shuttles' operation status (e.g., abrupt brake). A series of discrete outcome models were employed to estimate the factors influencing people's initial opinions before the AV shuttle ride and people's attitudes change after the ride. The most important finding is that an unsatisfactory riding experience could negatively affect participants' attitudes toward AV shuttles. Particularly, the number of abrupt brakes raises more concerns when the participants are drivers who share the rights of way with the AV shuttle compared to when the participants are pedestrians or AV shuttle riders. Moreover, the participants' attitudes toward sharing public space with AV shuttles as pedestrians and human drivers exhibited more negative changes compared with simply riding on AV shuttles. These results improve our understanding of public attitudes toward the existing AV technologies and help identify new opportunities to improve AV technology adoption.

We investigate a model many-body system of spinless Fermi gas in two dimensions, where the bare two-body interaction is repulsive and takes the form of a soft-core disk potential. We obtain the zero temperature phase diagram of this model by numerical functional renormalization group (FRG), which retains the effective interaction vertices in all channels to provide a detailed picture of how Cooper pairing emerges under the renormalization flow. The repulsion drives the system to a series of superfluid states with higher angular momentum paring, for example in the $f$- and $h$-wave channels instead of the $p$-wave channel. This is in sharp contrast to the original Kohn-Luttinger mechanism where pairing of very large angular momenta and exponentially small transition temperature was predicted. We trace the stabilization and enhancement of $f$- and $h$-wave pairing back to the momentum dependence of the bare interaction. A perturbative calculation is carried out to show that while the second order Kohn-Luttinger diagrams provide a qualitative understanding of the onsets of the various superfluid phases, they are unable to accurately capture the phase boundaries predicted by FRG. Our findings suggest that tuning the shape of the interaction potential offers a promising route to achieve stronger ``pairing glue" and to realize nontrivial superfluid phases in repulsive Fermi gases beyond the scope of the original Kohn-Luttinger analysis., Comment: Published version, with updated title, abstract, and introduction

In vehicle trajectory prediction, physics models and data-driven models are two predominant methodologies. However, each approach presents its own set of challenges: physics models fall short in predictability, while data-driven models lack interpretability. Addressing these identified shortcomings, this paper proposes a novel framework, the Physics-Enhanced Residual Learning (PERL) model. PERL integrates the strengths of physics-based and data-driven methods for traffic state prediction. PERL contains a physics model and a residual learning model. Its prediction is the sum of the physics model result and a predicted residual as a correction to it. It preserves the interpretability inherent to physics-based models and has reduced data requirements compared to data-driven methods. Experiments were conducted using a real-world vehicle trajectory dataset. We proposed a PERL model, with the Intelligent Driver Model (IDM) as its physics car-following model and Long Short-Term Memory (LSTM) as its residual learning model. We compare this PERL model with the physics car-following model, data-driven model, and other physics-informed neural network (PINN) models. The result reveals that PERL achieves better prediction with a small dataset, compared to the physics model, data-driven model, and PINN model. Second, the PERL model showed faster convergence during training, offering comparable performance with fewer training samples than the data-driven model and PINN model. Sensitivity analysis also proves comparable performance of PERL using another residual learning model and a physics car-following model.