Search

Prompt learning has become a prevalent strategy for adapting vision-language foundation models (VLMs) such as CLIP to downstream tasks. With the emergence of large language models (LLMs), recent studies have explored the potential of using category-related descriptions to enhance prompt effectiveness. However, conventional descriptions lack explicit structured information necessary to represent the interconnections among key elements like entities or attributes with relation to a particular category. Since existing prompt tuning methods give little consideration to managing structured knowledge, this paper advocates leveraging LLMs to construct a graph for each description to prioritize such structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), enabling simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and global-level prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Finally, by enhancing multi-granularity knowledge generation, redesigning the relationship-driven attention re-weighting module, and incorporating consistent constraints on the hierarchical text encoder, we propose HPT++, which further improves the performance of HPT. Our experiments are conducted across a wide range of evaluation settings, including base-to-new generalization, cross-dataset evaluation, and domain generalization. Extensive results and ablation studies demonstrate the effectiveness of our methods, which consistently outperform existing SOTA methods., Comment: 19 pages, 7 figures, 7 tables. arXiv admin note: substantial text overlap with arXiv:2312.06323

Video temporal grounding is an emerging topic aiming to identify specific clips within videos. In addition to pre-trained video models, contemporary methods utilize pre-trained vision-language models (VLM) to capture detailed characteristics of diverse scenes and objects from video frames. However, as pre-trained on images, VLM may struggle to distinguish action-sensitive patterns from static objects, making it necessary to adapt them to specific data domains for effective feature representation over temporal grounding. We address two primary challenges to achieve this goal. Specifically, to mitigate high adaptation costs, we propose an efficient preliminary in-domain fine-tuning paradigm for feature adaptation, where downstream-adaptive features are learned through several pretext tasks. Furthermore, to integrate action-sensitive information into VLM, we introduce Action-Cue-Injected Temporal Prompt Learning (ActPrompt), which injects action cues into the image encoder of VLM for better discovering action-sensitive patterns. Extensive experiments demonstrate that ActPrompt is an off-the-shelf training framework that can be effectively applied to various SOTA methods, resulting in notable improvements. The complete code used in this study is provided in the supplementary materials., Comment: 9 pages, 5 figures

In this work, we present a reward-driven automated curriculum reinforcement learning approach for interaction-aware self-driving at unsignalized intersections, taking into account the uncertainties associated with surrounding vehicles (SVs). These uncertainties encompass the uncertainty of SVs' driving intention and also the quantity of SVs. To deal with this problem, the curriculum set is specifically designed to accommodate a progressively increasing number of SVs. By implementing an automated curriculum selection mechanism, the importance weights are rationally allocated across various curricula, thereby facilitating improved sample efficiency and training outcomes. Furthermore, the reward function is meticulously designed to guide the agent towards effective policy exploration. Thus the proposed framework could proactively address the above uncertainties at unsignalized intersections by employing the automated curriculum learning technique that progressively increases task difficulty, and this ensures safe self-driving through effective interaction with SVs. Comparative experiments are conducted in $Highway\_Env$, and the results indicate that our approach achieves the highest task success rate, attains strong robustness to initialization parameters of the curriculum selection module, and exhibits superior adaptability to diverse situational configurations at unsignalized intersections. Furthermore, the effectiveness of the proposed method is validated using the high-fidelity CARLA simulator., Comment: 8 pages, 6 figures

Background: Colonoscopy, a crucial diagnostic tool in gastroenterology, depends heavily on superior bowel preparation. ChatGPT, a large language model with emergent intelligence which also exhibits potential in medical applications. This study aims to assess the accuracy and consistency of ChatGPT in using the Boston Bowel Preparation Scale (BBPS) for colonoscopy assessment. Methods: We retrospectively collected 233 colonoscopy images from 2020 to 2023. These images were evaluated using the BBPS by 3 senior endoscopists and 3 novice endoscopists. Additionally, ChatGPT also assessed these images, having been divided into three groups and undergone specific Fine-tuning. Consistency was evaluated through two rounds of testing. Results: In the initial round, ChatGPT's accuracy varied between 48.93% and 62.66%, trailing the endoscopists' accuracy of 76.68% to 77.83%. Kappa values for ChatGPT was between 0.52 and 0.53, compared to 0.75 to 0.87 for the endoscopists. Conclusion: While ChatGPT shows promise in bowel preparation scoring, it currently does not match the accuracy and consistency of experienced endoscopists. Future research should focus on in-depth Fine-tuning.

The interface-enhanced superconductivity in monolayer iron selenide (FeSe) films on SrTiO3 has been actively pursued in the past decade. Although a synergistic effect between interfacial charge transfer and interfacial electron-phonon coupling (EPC) is proposed to be responsible for the mechanism, the microscopic nature of the interfacial EPC in the enhancement of superconductivity remains highly controversial. Herein we experimentally reveal that a coherent optical phonon mode at 4.2 THz from the SrTiO3 substrate couples to FeSe electrons and modulates the quasiparticle relaxations using ultrafast pump-probe spectroscopy. This mode originates from the antiferrodistortive (AFD) transition in SrTiO3 and is significantly stronger in the presence of monolayer FeSe than that in purely FeTe-capped Nb-doped SrTiO3. Pump fluence and temperature-dependent spectroscopy measurements suggest that SrTiO3 substrate facilitates the stabilization of FeSe structure and possibly prevents the occurrence of nematic phase transition, supporting that SrTiO3 substrate modifies the electronic structure of monolayer FeSe through a strong interfacial EPC strength as large as 0.77. Our results provide unprecedented direct evidence that the strong coupling of SrTiO3 coherent phonon to FeSe electrons is indeed responsible for the high-temperature superconductivity in monolayer FeSe and SrTiO3 heterostructure.

Prompt learning has become a prevalent strategy for adapting vision-language foundation models to downstream tasks. As large language models (LLMs) have emerged, recent studies have explored the use of category-related descriptions as input to enhance prompt effectiveness. Nevertheless, conventional descriptions fall short of structured information that effectively represents the interconnections among entities or attributes linked to a particular category. To address this limitation and prioritize harnessing structured knowledge, this paper advocates for leveraging LLMs to build a graph for each description to model the entities and attributes describing the category, as well as their correlations. Preexisting prompt tuning methods exhibit inadequacies in managing this structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), which enables simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and global-level prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Extensive experiments demonstrate that our HPT shows strong effectiveness and generalizes much better than existing SOTA methods. Our code is available at https://github.com/Vill-Lab/2024-AAAI-HPT., Comment: AAAI2024

Microcavity exciton polaritons (polaritons) as part-light part-matter quasiparticles, garner significant attention for non-equilibrium Bose-Einstein condensation at elevated temperatures. Recently, halide perovskites have emerged as promising room-temperature polaritonic platforms thanks to their large exciton binding energies and superior optical properties. However, currently, inducing room-temperature non-equilibrium polariton condensation in perovskite microcavities requires optical pulsed excitations with high excitation densities. Herein, we demonstrate continuous-wave optically pumped polariton condensation with an exceptionally low threshold of ~0.6 W cm-2 and a narrow linewidth of ~1 meV. Polariton condensation is unambiguously demonstrated by characterizing the nonlinear behavior and coherence properties. We also identify a microscopic mechanism involving the potential landscape in the perovskite microcavity, where numerous discretized energy levels arising from the hybridization of adjacent potential minima enhance the polariton relaxation, facilitating polariton condensate formation. Our findings lay the foundation for the next-generation energy-efficient polaritonic devices operating at room temperature., Comment: 16 pages, 4 figures

We develop and demonstrate how to use the GUGA-based MRCISD with Core-Valence Separation approximation (CVS) to compute the core-excited states. Firstly, perform a normal SCF or valence MCSCF calculation to optimize the molecular orbitals. Secondly, rotate the optimized target core orbitals and append to the active space, form an extended CVS active space, and perform a CVS-MCSCF calculation for core-excited states. Finally, construct the CVS-MRCI expansion space, and perform a CVS-MRCI calculation to optimize the CI coefficients based on the variational method. The CVS approximation with GUGA-based methods can be implemented by flexible truncation of the Distinct Row Table (DRT). Eliminating the valence-excited configurations from the CVS-MRCI expansion space can prevent variational collapse in the Davidson iteration diagonalization. The accuracy of the CVS-MRCI scheme was investigated for excitation energies and compared with that of the CVS-MCSCF method. The results show that CVS-MRCI is capable of reproducing well-matched vertical core excitation energies that are consistent with experiments, by combining large basis sets and a rational reference space. The calculation results also highlight the fact that the dynamic correlation between electrons makes an undeniable contribution in core-excited states.

Cloth-Changing Person Re-Identification (CC-ReID) is a common and realistic problem since fashion constantly changes over time and people's aesthetic preferences are not set in stone. While most existing cloth-changing ReID methods focus on learning cloth-agnostic identity representations from coarse semantic cues (e.g. silhouettes and part segmentation maps), they neglect the continuous shape distributions at the pixel level. In this paper, we propose Continuous Surface Correspondence Learning (CSCL), a new shape embedding paradigm for cloth-changing ReID. CSCL establishes continuous correspondences between a 2D image plane and a canonical 3D body surface via pixel-to-vertex classification, which naturally aligns a person image to the surface of a 3D human model and simultaneously obtains pixel-wise surface embeddings. We further extract fine-grained shape features from the learned surface embeddings and then integrate them with global RGB features via a carefully designed cross-modality fusion module. The shape embedding paradigm based on 2D-3D correspondences remarkably enhances the model's global understanding of human body shape. To promote the study of ReID under clothing change, we construct 3D Dense Persons (DP3D), which is the first large-scale cloth-changing ReID dataset that provides densely annotated 2D-3D correspondences and a precise 3D mesh for each person image, while containing diverse cloth-changing cases over all four seasons. Experiments on both cloth-changing and cloth-consistent ReID benchmarks validate the effectiveness of our method., Comment: Accepted by ACM MM 2023

Unsignalized intersections are typically considered as one of the most representative and challenging scenarios for self-driving vehicles. To tackle autonomous driving problems in such scenarios, this paper proposes a curriculum proximal policy optimization (CPPO) framework with stage-decaying clipping. By adjusting the clipping parameter during different stages of training through proximal policy optimization (PPO), the vehicle can first rapidly search for an approximate optimal policy or its neighborhood with a large parameter, and then converges to the optimal policy with a small one. Particularly, the stage-based curriculum learning technology is incorporated into the proposed framework to improve the generalization performance and further accelerate the training process. Moreover, the reward function is specially designed in view of different curriculum settings. A series of comparative experiments are conducted in intersection-crossing scenarios with bi-lane carriageways to verify the effectiveness of the proposed CPPO method. The results show that the proposed approach demonstrates better adaptiveness to different dynamic and complex environments, as well as faster training speed over baseline methods., Comment: 7 pages, 4 figures

In this work, we propose a novel learning-based model predictive control (MPC) framework for motion planning and control of urban self-driving. In this framework, instantaneous references and cost functions of online MPC are learned from raw sensor data without relying on any oracle or predicted states of traffic. Moreover, driving safety conditions are latently encoded via the introduction of a learnable instantaneous reference vector. In particular, we implement a deep reinforcement learning (DRL) framework for policy search, where practical and lightweight raw observations are processed to reason about the traffic and provide the online MPC with instantaneous references. The proposed approach is validated in a high-fidelity simulator, where our development manifests remarkable adaptiveness to complex and dynamic traffic. Furthermore, sim-to-real deployments are also conducted to evaluate the generalizability of the proposed framework in various real-world applications. Also, we provide the open-source code and video demonstrations at the project website: https://latent-mpc.github.io/.

Bulk photovoltaic effect, which arises from crystal symmetry-driven charge carrier separation, is an intriguing physical phenomenon that has attracted extensive interest in photovoltaic application due to its junction-free photovoltaic and potential to surpass Shockley-Queisser limit. Whereas conventional ferroelectric materials mostly suffer from extremely low photocurrent density and weak photovoltaic response at visible light wavelengths. Emerging two-dimensional ferroelectric semiconductors with coupled visible light absorption and spontaneous polarization characteristics are a promising alternative for making functional photoferroelectrics. Herein, we report the experimental demonstration of the bulk photovoltaic effect behavior based on the 2D ferroelectric semiconductor {$\alpha$-InSe caused by an out-of-plane polarization induced depolarization field. The {$\alpha$-InSe device exhibits enhanced bulk photovoltaic response in the visible light spectrum owing to its narrow bandgap. It was demonstrated that the generated photovoltaic current density was nearly two orders of magnitude greater than conventional bulk ferroelectric materials. These findings highlight the potential of 2D ferroelectric semiconductor materials for bulk photovoltaic applications in a broad spectral region.

Spin or valley degrees of freedom in condensed matter have been proposed as efficient information carriers towards next generation spintronics. It is therefore crucial to develop effective strategies to generate and control spin or valley-locked spin currents, e.g., by exploiting the spin Hall or valley Hall effects. However, the scattering, and rapid dephasing of electrons pose major challenges to achieve macroscopic coherent spin currents and realistic spintronic or valleytronic devices, specifically at room temperature, where strong thermal fluctuations could further obscure the spin flow. Exciton polaritons in semiconductor microcavities being the quantum superposition of excitons and photons, are believed to be promising platforms for spin-dependent optoelectronic or, in short, spin-optronic devices. Long-range spin current flows of exciton polaritons may be controlled through the optical spin Hall effect. However, this effect could neither be unequivocally observed at room temperature nor be exploited for realistic polariton spintronic devices due to the presence of strong thermal fluctuations or large linear spin splittings. Here, we report the observation of room temperature optical spin Hall effect of exciton polaritons with the spin current flow over a distance as large as 60 um in a hybrid organic-inorganic FAPbBr3 perovskite microcavity. We show direct evidence of the long-range coherence at room temperature in the flow of exciton polaritons, and the spin current carried by them. By harnessing the long-range spin-Hall transport of exciton polaritons, we have demonstrated two novel room temperature polaritonic devices, namely the NOT gate and the spin-polarized beam splitter, advancing the frontier of room-temperature polaritonics in perovskite microcavities., Comment: 21 pages, 5 figures

We develop a scheme of generating highly indistinguishable single photons from an active quantum Su-Schrieffer-Heeger chain made from a collection of noisy quantum emitters. Surprisingly, the single photon emission spectrum of the active quantum chain is extremely narrow compared to that of a single emitter or topologically trivial chain. Moreover, this effect becomes dramatically strong close to the non-trivial-to-trivial phase transition point. Using this effect, we show that the single photon linewidth of a long topological quantum chain can become arbitrarily narrow, constituting an ideal source of indistinguishable single photons. Finally, taking specific examples of actual quantum emitters, we provide a microscopic and quantitative analysis of our model and analyze the most important parameters in view of the experimental realization., Comment: 17 pages, 10 figures, 1 table

Exciton-polaritons resulting from the strong exciton-photon interaction stimulate the development of novel coherent light sources with low threshold, long range spatial, and temporal coherence to circumvent the ever increasing energy demands of optical communications. Polaritons from bound states in the continuum (BICs) are promising for Bose-Einstein condensation owing to their infinite quality factors that enlarge photon lifetimes and benefit polariton accumulations. However, BIC polariton condensation remains limited to cryogenic temperatures ascribed to the small exciton binding energies of conventional material platforms. Herein, we demonstrated a room-temperature BIC polaritonic platform based on halide perovskite air-hole photonic crystals, profiting from the non-radiative BIC states and stable excitons of cesium lead bromide. BIC polariton condensation was achieved near the dispersion minimum that generates directional vortex beam emission with long-range coherence. Our work provides a significant basis for the applications of polariton condensates for integrated photonic and topological circuits., Comment: 16 pages,4 figures

Lane change in dense traffic typically requires the recognition of an appropriate opportunity for maneuvers, which remains a challenging problem in self-driving. In this work, we propose a chance-aware lane-change strategy with high-level model predictive control (MPC) through curriculum reinforcement learning (CRL). In our proposed framework, full-state references and regulatory factors concerning the relative importance of each cost term in the embodied MPC are generated by a neural policy. Furthermore, effective curricula are designed and integrated into an episodic reinforcement learning (RL) framework with policy transfer and enhancement, to improve the convergence speed and ensure a high-quality policy. The proposed framework is deployed and evaluated in numerical simulations of dense and dynamic traffic. It is noteworthy that, given a narrow chance, the proposed approach generates high-quality lane-change maneuvers such that the vehicle merges into the traffic flow with a high success rate of 96%. Finally, our framework is validated in the high-fidelity simulator under dense traffic, demonstrating satisfactory practicality and generalizability., Comment: The paper has been accepted by ICRA 2024

Prompt learning is one of the most effective and trending ways to adapt powerful vision-language foundation models like CLIP to downstream datasets by tuning learnable prompt vectors with very few samples. However, although prompt learning achieves excellent performance over in-domain data, it still faces the major challenge of generalizing to unseen classes and domains. Some existing prompt learning methods tackle this issue by adaptively generating different prompts for different tokens or domains but neglecting the ability of learned prompts to generalize to unseen domains. In this paper, we propose a novel prompt learning paradigm that directly generates \emph{domain invariant} prompt that can be generalized to unseen domains, called MetaPrompt. Specifically, a dual-modality prompt tuning network is proposed to generate prompts for input from both image and text modalities. With a novel asymmetric contrastive loss, the representation from the original pre-trained vision-language model acts as supervision to enhance the generalization ability of the learned prompt. More importantly, we propose a meta-learning-based prompt tuning algorithm that explicitly constrains the task-specific prompt tuned for one domain or class to also achieve good performance in another domain or class. Extensive experiments on 11 datasets for base-to-new generalization and 4 datasets for domain generalization demonstrate that our method consistently and significantly outperforms existing methods., Comment: 12 pages, 6 figures, 5 tables