Search

In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms. To address this challenge, we aim to achieve both high model performance and practical deployability. Specifically, we focus on Vision-and-Language Navigation (VLN), a core task in Embodied AI. This paper introduces a two-stage knowledge distillation framework, producing a student model, MiniVLN, and showcasing the significant potential of distillation techniques in developing lightweight models. The proposed method aims to capture fine-grained knowledge during the pretraining phase and navigation-specific knowledge during the fine-tuning phase. Our findings indicate that the two-stage distillation approach is more effective in narrowing the performance gap between the teacher model and the student model compared to single-stage distillation. On the public R2R and REVERIE benchmarks, MiniVLN achieves performance on par with the teacher model while having only about 12% of the teacher model's parameter count.

Vision-and-Language Navigation (VLN) tasks require an agent to follow textual instructions to navigate through 3D environments. Traditional approaches use supervised learning methods, relying heavily on domain-specific datasets to train VLN models. Recent methods try to utilize closed-source large language models (LLMs) like GPT-4 to solve VLN tasks in zero-shot manners, but face challenges related to expensive token costs and potential data breaches in real-world applications. In this work, we introduce Open-Nav, a novel study that explores open-source LLMs for zero-shot VLN in the continuous environment. Open-Nav employs a spatial-temporal chain-of-thought (CoT) reasoning approach to break down tasks into instruction comprehension, progress estimation, and decision-making. It enhances scene perceptions with fine-grained object and spatial knowledge to improve LLM's reasoning in navigation. Our extensive experiments in both simulated and real-world environments demonstrate that Open-Nav achieves competitive performance compared to using closed-source LLMs.

This study investigates the 5$f$-electron-conduction electron hybridization process in the heavy fermion superconductor UPd$_2$Al$_3$ using a combination of angle-resolved photoemission spectroscopy (ARPES) and time-resolved photoemission spectroscopy (tr-PES). ARPES measurements reveal the formation of a hybridization gap at a temperature of approximately 75 K, which becomes more pronounced as the temperature decreases. Notably, the persistence of a flat U 5$f$ band at temperatures well above the hybridization onset challenges conventional understanding. Our findings demonstrate a non-monotonic temperature dependence of the quasiparticle relaxation time, with an anomalous decrease at 20 K, suggesting complex electronic and magnetic interactions. These findings provide detailed insights into the 5$f$-electron hybridization process in UPd$_2$Al$_3$, with significant implications for the understanding of heavy fermion superconductivity and the role of 5$f$-electron hybridization in uranium-based materials., Comment: 5 pages, 4 figures

Let T be the singular integral operator with variable kernel defined by $Tf(x)= p.v. \int_{\mathbb{R}^{n}}K(x,x-y)f(y)\mathrm{d}y$ and $D^{\gamma}(0\leq\gamma\leq1)$ be the fractional differentiation operator, where $K(x,z)=\frac{\Omega(x,z')}{|z|^{n}}$, $z'=\frac{z}{|z|},~~z\neq0$. Let $~T^{\ast}~$and $~T^\sharp~$ be the adjoint of $T$ and the pseudo-adjoint of $T$, respectively. In this paper, via the expansion of spherical harmonics and the estimates of the convolution operators $T_{m,j}$, we shall prove some boundedness results for $TD^{\gamma}-D^{\gamma}T$ and $(T^{\ast}-T^{\sharp})D^{\gamma}$ under natural regularity assumptions on the exponent function on a class of generalized Herz-Morrey spaces with weight and variable exponent, which extend some known results. Moreover, various norm characterizations for the product $T_{1}T_{2}$ and the pseudo-product $T_{1}\circ T_{2}$ are also established.

The vector charmonium states can be directly produced at the $e^+e^{-}$ annihilation process. Among them, $Y(4230)$ and $Y(4360)$ splitting from the previously discovered $Y(4260)$ are not easily arranged into the conventional charmonium spectrum, while the recent studies indicated that they have strong couplings to $D\bar{D}_1$ and $D^*\bar{D}_1$. In this work, we investigate the productions of $Y(4230)$ and $Y(4360)$ as the heavy quark spin symmetry doublet hadronic molecules of $D\bar{D}_1$ and $D^*\bar{D}_1$ in $B$ decays via the triangle diagram mechanism. In particular, we propose that the decay constants of $Y(4230)$ and $Y(4360)$ extracted in $B$ decays are helpful to clarify their nature., Comment: Typos corrected

Current text-video retrieval methods mainly rely on cross-modal matching between queries and videos to calculate their similarity scores, which are then sorted to obtain retrieval results. This method considers the matching between each candidate video and the query, but it incurs a significant time cost and will increase notably with the increase of candidates. Generative models are common in natural language processing and computer vision, and have been successfully applied in document retrieval, but their application in multimodal retrieval remains unexplored. To enhance retrieval efficiency, in this paper, we introduce a model-based video indexer named T2VIndexer, which is a sequence-to-sequence generative model directly generating video identifiers and retrieving candidate videos with constant time complexity. T2VIndexer aims to reduce retrieval time while maintaining high accuracy. To achieve this goal, we propose video identifier encoding and query-identifier augmentation approaches to represent videos as short sequences while preserving their semantic information. Our method consistently enhances the retrieval efficiency of current state-of-the-art models on four standard datasets. It enables baselines with only 30\%-50\% of the original retrieval time to achieve better retrieval performance on MSR-VTT (+1.0%), MSVD (+1.8%), ActivityNet (+1.5%), and DiDeMo (+0.2%). The code is available at https://github.com/Lilidamowang/T2VIndexer-generativeSearch.

In the present work, we propose to explore the productions of the bottom and bottom-strange mesons in the high-energy pion and kaon-induced reactions on a proton target. The cross sections are evaluated with an effective Lagrangian constructed by the heavy-quark limit and chiral symmetry. Our estimations show that at $P_\pi=80$ GeV, the cross sections for $B(5279)$, $B^\ast (5325)$, $B_0^\ast (5738)$, $B_1^\prime (5757)$, $B_1(5721)$ and $B_2^\ast (5747)$ production processes are estimated to be $3.19 \sim 86.26$, $1.86\sim 51.29$, $0.87 \sim 24.25$, $0.84 \sim 23.14$, $162.35 \sim 4477.66$, and $57.16 \sim 1604.43$ nb, respectively, where uncertainties arise from the model parameter. In addition, the cross sections for the corresponding bottom-strange mesons production processes are very similar. Moreover, our estimations indicate that the ratios of these cross sections are almost independent on the model parameters. In particular, the cross-section ratios related to the states in the same doublets are of order one, which is consistent with the expectation of heavy-quark limit. The cross sections related to the states in the $T$ doublets are about two orders larger than those related to the states in the $S$ doublets.

In the present work, we investigate the production of $X(4630)$ and $Y(4626)$ in $B^+$ and $B_s^0$ decays, where $X(4630)$ and $Y(4626)$ are considered as the $C-$ parity pigeon pair in the $D_{s}^{\ast+} D_{s1}(2536)^-$ molecular frame. The branching fractions of $B^+ \to K^+ X(4630)/Y(4626)$ and $B_s^0 \to \eta X(4630)/Y(4626)$ have been evaluated using an effective Lagrangian approach, which are of the order of $10^{-5}$ and the ratios of these branching fractions are almost independent on the model parameter. Based on the present estimations, we propose to search $Y(4626)$ in the process $B^+ \to K^+ J/\psi \eta^{(\prime)}$, which should be accessible by the LHCb and Belle II Collaborations.

Real-world navigation often involves dealing with unexpected obstructions such as closed doors, moved objects, and unpredictable entities. However, mainstream Vision-and-Language Navigation (VLN) tasks typically assume instructions perfectly align with the fixed and predefined navigation graphs without any obstructions. This assumption overlooks potential discrepancies in actual navigation graphs and given instructions, which can cause major failures for both indoor and outdoor agents. To address this issue, we integrate diverse obstructions into the R2R dataset by modifying both the navigation graphs and visual observations, introducing an innovative dataset and task, R2R with UNexpected Obstructions (R2R-UNO). R2R-UNO contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research. Experiments on R2R-UNO reveal that state-of-the-art VLN methods inevitably encounter significant challenges when facing such mismatches, indicating that they rigidly follow instructions rather than navigate adaptively. Therefore, we propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction to help agents effectively adapt to obstructed environments. Empirical results show that ObVLN not only maintains robust performance in unobstructed scenarios but also achieves a substantial performance advantage with unexpected obstructions., Comment: Accepted to MM 2024

Vision-and-language pretraining (VLP) in the medical field utilizes contrastive learning on image-text pairs to achieve effective transfer across tasks. Yet, current VLP approaches with the masked modelling strategy face two challenges when applied to the medical domain. First, current models struggle to accurately reconstruct key pathological features due to the scarcity of medical data. Second, most methods only adopt either paired image-text or image-only data, failing to exploit the combination of both paired and unpaired data. To this end, this paper proposes a XLIP (Masked modelling for medical Language-Image Pre-training) framework to enhance pathological learning and feature learning via unpaired data. First, we introduce the attention-masked image modelling (AttMIM) and entity-driven masked language modelling module (EntMLM), which learns to reconstruct pathological visual and textual tokens via multi-modal feature interaction, thus improving medical-enhanced features. The AttMIM module masks a portion of the image features that are highly responsive to textual features. This allows XLIP to improve the reconstruction of highly similar image data in medicine efficiency. Second, our XLIP capitalizes unpaired data to enhance multimodal learning by introducing disease-kind prompts. The experimental results show that XLIP achieves SOTA for zero-shot and fine-tuning classification performance on five datasets. Our code will be available at https://github.com/White65534/XLIP

Recent work shows that documents from encyclopedias serve as helpful auxiliary information for zero-shot learning. Existing methods align the entire semantics of a document with corresponding images to transfer knowledge. However, they disregard that semantic information is not equivalent between them, resulting in a suboptimal alignment. In this work, we propose a novel network to extract multi-view semantic concepts from documents and images and align the matching rather than entire concepts. Specifically, we propose a semantic decomposition module to generate multi-view semantic embeddings from visual and textual sides, providing the basic concepts for partial alignment. To alleviate the issue of information redundancy among embeddings, we propose the local-to-semantic variance loss to capture distinct local details and multiple semantic diversity loss to enforce orthogonality among embeddings. Subsequently, two losses are introduced to partially align visual-semantic embedding pairs according to their semantic relevance at the view and word-to-patch levels. Consequently, we consistently outperform state-of-the-art methods under two document sources in three standard benchmarks for document-based zero-shot learning. Qualitatively, we show that our model learns the interpretable partial association., Comment: Accepted to ACM International Conference on Multimedia (MM) 2024

Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs to generalize navigational reasoning and diverse language understanding. However, a significant discrepancy in agent performance is observed when integrating LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous downstream specialist models. Furthermore, the inherent capacity of language to interpret and facilitate communication in agent interactions is often underutilized in these integrations. In this work, we strive to bridge the divide between VLN-specialized models and LLM-based navigation paradigms, while maintaining the interpretative prowess of LLMs in generating linguistic navigational reasoning. By aligning visual content in a frozen LLM, we encompass visual observation comprehension for LLMs and exploit a way to incorporate LLMs and navigation policy networks for effective action predictions and navigational reasoning. We demonstrate the data efficiency of the proposed methods and eliminate the gap between LM-based agents and state-of-the-art VLN specialists., Comment: Accepted to ECCV 2024

Vision-and-Language Navigation (VLN) has gained increasing attention over recent years and many approaches have emerged to advance their development. The remarkable achievements of foundation models have shaped the challenges and proposed methods for VLN research. In this survey, we provide a top-down review that adopts a principled framework for embodied planning and reasoning, and emphasizes the current methods and future opportunities leveraging foundation models to address VLN challenges. We hope our in-depth discussions could provide valuable resources and insights: on one hand, to milestone the progress and explore opportunities and potential roles for foundation models in this field, and on the other, to organize different challenges and solutions in VLN to foundation model researchers., Comment: Authors contributed equally to this work, and supervisors contributed equal advising to this work

We propose and discuss a paradigm that allows for expressing \emph{data-parallel} rendering with the classically non-parallel ANARI API. We propose this as a new standard for data-parallel sci-vis rendering, describe two different implementations of this paradigm, and use multiple sample integrations into existing apps to show how easy it is to adopt this paradigm, and what can be gained from doing so.

One of the key arguments for building robots that have similar form factors to human beings is that we can leverage the massive human data for training. Yet, doing so has remained challenging in practice due to the complexities in humanoid perception and control, lingering physical gaps between humanoids and humans in morphologies and actuation, and lack of a data pipeline for humanoids to learn autonomous skills from egocentric vision. In this paper, we introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. This policy transfers to the real world and allows humanoid robots to follow human body and hand motion in real time using only a RGB camera, i.e. shadowing. Through shadowing, human operators can teleoperate humanoids to collect whole-body data for learning different tasks in the real world. Using the data collected, we then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously by imitating human skills. We demonstrate the system on our customized 33-DoF 180cm humanoid, autonomously completing tasks such as wearing a shoe to stand up and walk, unloading objects from warehouse racks, folding a sweatshirt, rearranging objects, typing, and greeting another robot with 60-100% success rates using up to 40 demonstrations. Project website: https://humanoid-ai.github.io/, Comment: project website: https://humanoid-ai.github.io/

Patients with mental disorders often exhibit risky abnormal actions, such as climbing walls or hitting windows, necessitating intelligent video behavior monitoring for smart healthcare with the rising Internet of Things (IoT) technology. However, the development of vision-based Human Action Recognition (HAR) for these actions is hindered by the lack of specialized algorithms and datasets. In this paper, we innovatively propose to build a vision-based HAR dataset including abnormal actions often occurring in the mental disorder group and then introduce a novel Scene-Motion-aware Action Recognition Technology framework, named SMART, consisting of two technical modules. First, we propose a scene perception module to extract human motion trajectory and human-scene interaction features, which introduces additional scene information for a supplementary semantic representation of the above actions. Second, the multi-stage fusion module fuses the skeleton motion, motion trajectory, and human-scene interaction features, enhancing the semantic association between the skeleton motion and the above supplementary representation, thus generating a comprehensive representation with both human motion and scene information. The effectiveness of our proposed method has been validated on our self-collected HAR dataset (MentalHAD), achieving 94.9% and 93.1% accuracy in un-seen subjects and scenes and outperforming state-of-the-art approaches by 6.5% and 13.2%, respectively. The demonstrated subject- and scene- generalizability makes it possible for SMART's migration to practical deployment in smart healthcare systems for mental disorder patients in medical settings. The code and dataset will be released publicly for further research: https://github.com/Inowlzy/SMART.git.

Current Vision-and-Language Navigation (VLN) tasks mainly employ textual instructions to guide agents. However, being inherently abstract, the same textual instruction can be associated with different visual signals, causing severe ambiguity and limiting the transfer of prior knowledge in the vision domain from the user to the agent. To fill this gap, we propose Vision-and-Language Navigation with Multi-modal Prompts (VLN-MP), a novel task augmenting traditional VLN by integrating both natural language and images in instructions. VLN-MP not only maintains backward compatibility by effectively handling text-only prompts but also consistently shows advantages with different quantities and relevance of visual prompts. Possible forms of visual prompts include both exact and similar object images, providing adaptability and versatility in diverse navigation scenarios. To evaluate VLN-MP under a unified framework, we implement a new benchmark that offers: (1) a training-free pipeline to transform textual instructions into multi-modal forms with landmark images; (2) diverse datasets with multi-modal instructions for different downstream tasks; (3) a novel module designed to process various image prompts for seamless integration with state-of-the-art VLN models. Extensive experiments on four VLN benchmarks (R2R, RxR, REVERIE, CVDN) show that incorporating visual prompts significantly boosts navigation performance. While maintaining efficiency with text-only prompts, VLN-MP enables agents to navigate in the pre-explore setting and outperform text-based models, showing its broader applicability., Comment: IJCAI 2024

The vision-and-language navigation (VLN) task necessitates an agent to perceive the surroundings, follow natural language instructions, and act in photo-realistic unseen environments. Most of the existing methods employ the entire image or object features to represent navigable viewpoints. However, these representations are insufficient for proper action prediction, especially for the REVERIE task, which uses concise high-level instructions, such as ''Bring me the blue cushion in the master bedroom''. To address enhancing representation, we propose an augmented commonsense knowledge model (ACK) to leverage commonsense information as a spatio-temporal knowledge graph for improving agent navigation. Specifically, the proposed approach involves constructing a knowledge base by retrieving commonsense information from ConceptNet, followed by a refinement module to remove noisy and irrelevant knowledge. We further present ACK which consists of knowledge graph-aware cross-modal and concept aggregation modules to enhance visual representation and visual-textual data alignment by integrating visible objects, commonsense knowledge, and concept history, which includes object and knowledge temporal information. Moreover, we add a new pipeline for the commonsense-based decision-making process which leads to more accurate local action prediction. Experimental results demonstrate our proposed model noticeably outperforms the baseline and archives the state-of-the-art on the REVERIE benchmark.

We present a novel task called online video editing, which is designed to edit \textbf{streaming} frames while maintaining temporal consistency. Unlike existing offline video editing assuming all frames are pre-established and accessible, online video editing is tailored to real-life applications such as live streaming and online chat, requiring (1) fast continual step inference, (2) long-term temporal modeling, and (3) zero-shot video editing capability. To solve these issues, we propose Streaming Video Diffusion (SVDiff), which incorporates the compact spatial-aware temporal recurrence into off-the-shelf Stable Diffusion and is trained with the segment-level scheme on large-scale long videos. This simple yet effective setup allows us to obtain a single model that is capable of executing a broad range of videos and editing each streaming frame with temporal coherence. Our experiments indicate that our model can edit long, high-quality videos with remarkable results, achieving a real-time inference speed of 15.2 FPS at a resolution of 512x512.

Recent advancements in Multimodal Large Language Models (MM-LLMs) have demonstrated promising potential in terms of generalization and robustness when applied to different modalities. While previous works have already achieved 3D human motion generation using various approaches including language modeling, they mostly % are mostly carefully designed use specialized architecture and are restricted to single-human motion generation. Inspired by the success of MM-LLMs, we propose MotionLLM, a simple and general framework that can achieve single-human, multi-human motion generation, and motion captioning by fine-tuning pre-trained LLMs. Specifically, we encode and quantize motions into discrete LLM-understandable tokens, which results in a unified vocabulary consisting of both motion and text tokens. With only 1--3% parameters of the LLMs trained by using adapters, our single-human motion generation achieves comparable results to those diffusion models and other trained-from-scratch transformer-based models. Additionally, we show that our approach is scalable and flexible, allowing easy extension to multi-human motion generation through autoregressive generation of single-human motions. Project page: https://knoxzhao.github.io/MotionLLM, Comment: Project page: https://knoxzhao.github.io/MotionLLM

We present a functional generative approach to extract risk-neutral densities from market prices of options. Specifically, we model the log-returns on the time-to-maturity continuum as a stochastic curve driven by standard normal. We then use neural nets to represent the term structures of the location, the scale, and the higher-order moments, and impose stringent conditions on the learning process to ensure the neural net-based curve representation is free of static arbitrage. This specification is structurally clear in that it separates the modeling of randomness from the modeling of the term structures of the parameters. It is data adaptive in that we use neural nets to represent the shape of the stochastic curve. It is also generative in that the functional form of the stochastic curve, although parameterized by neural nets, is an explicit and deterministic function of the standard normal. This explicitness allows for the efficient generation of samples to price options across strikes and maturities, without compromising data adaptability. We have validated the effectiveness of this approach by benchmarking it against a comprehensive set of baseline models. Experiments show that the extracted risk-neutral densities accommodate a diverse range of shapes. Its accuracy significantly outperforms the extensive set of baseline models--including three parametric models and nine stochastic process models--in terms of accuracy and stability. The success of this approach is attributed to its capacity to offer flexible term structures for risk-neutral skewness and kurtosis.

Recent advancements in social bot detection have been driven by the adoption of Graph Neural Networks. The social graph, constructed from social network interactions, contains benign and bot accounts that influence each other. However, previous graph-based detection methods that follow the transductive message-passing paradigm may not fully utilize hidden graph information and are vulnerable to adversarial bot behavior. The indiscriminate message passing between nodes from different categories and communities results in excessively homogeneous node representations, ultimately reducing the effectiveness of social bot detectors. In this paper, we propose SEBot, a novel multi-view graph-based contrastive learning-enabled social bot detector. In particular, we use structural entropy as an uncertainty metric to optimize the entire graph's structure and subgraph-level granularity, revealing the implicitly existing hierarchical community structure. And we design an encoder to enable message passing beyond the homophily assumption, enhancing robustness to adversarial behaviors of social bots. Finally, we employ multi-view contrastive learning to maximize mutual information between different views and enhance the detection performance through multi-task learning. Experimental results demonstrate that our approach significantly improves the performance of social bot detection compared with SOTA methods., Comment: KDD 2024

Detecting social bots has evolved into a pivotal yet intricate task, aimed at combating the dissemination of misinformation and preserving the authenticity of online interactions. While earlier graph-based approaches, which leverage topological structure of social networks, yielded notable outcomes, they overlooked the inherent dynamicity of social networks -- In reality, they largely depicted the social network as a static graph and solely relied on its most recent state. Due to the absence of dynamicity modeling, such approaches are vulnerable to evasion, particularly when advanced social bots interact with other users to camouflage identities and escape detection. To tackle these challenges, we propose BotDGT, a novel framework that not only considers the topological structure, but also effectively incorporates dynamic nature of social network. Specifically, we characterize a social network as a dynamic graph. A structural module is employed to acquire topological information from each historical snapshot. Additionally, a temporal module is proposed to integrate historical context and model the evolving behavior patterns exhibited by social bots and legitimate users. Experimental results demonstrate the superiority of BotDGT against the leading methods that neglected the dynamic nature of social networks in terms of accuracy, recall, and F1-score., Comment: IJCAI 2024

Novel view synthesis aims to generate new view images of a given view image collection. Recent attempts address this problem relying on 3D geometry priors (e.g., shapes, sizes, and positions) learned from multi-view images. However, such methods encounter the following limitations: 1) they require a set of multi-view images as training data for a specific scene (e.g., face, car or chair), which is often unavailable in many real-world scenarios; 2) they fail to extract the geometry priors from single-view images due to the lack of multi-view supervision. In this paper, we propose a Geometry-enhanced NeRF (G-NeRF), which seeks to enhance the geometry priors by a geometry-guided multi-view synthesis approach, followed by a depth-aware training. In the synthesis process, inspired that existing 3D GAN models can unconditionally synthesize high-fidelity multi-view images, we seek to adopt off-the-shelf 3D GAN models, such as EG3D, as a free source to provide geometry priors through synthesizing multi-view data. Simultaneously, to further improve the geometry quality of the synthetic data, we introduce a truncation method to effectively sample latent codes within 3D GAN models. To tackle the absence of multi-view supervision for single-view images, we design the depth-aware training approach, incorporating a depth-aware discriminator to guide geometry priors through depth maps. Experiments demonstrate the effectiveness of our method in terms of both qualitative and quantitative results., Comment: CVPR 2024 Accepted Paper

Current vision-language pre-training (VLP) methodologies predominantly depend on paired image-text datasets, a resource that is challenging to acquire in radiology due to privacy considerations and labelling complexities. Data augmentation provides a practical solution to overcome the issue of data scarcity, however, most augmentation methods exhibit a limited focus, prioritising either image or text augmentation exclusively. Acknowledging this limitation, our objective is to devise a framework capable of concurrently augmenting medical image and text data. We design a Pairwise Augmentation (PairAug) approach that contains an Inter-patient Augmentation (InterAug) branch and an Intra-patient Augmentation (IntraAug) branch. Specifically, the InterAug branch of our approach generates radiology images using synthesised yet plausible reports derived from a Large Language Model (LLM). The generated pairs can be considered a collection of new patient cases since they are artificially created and may not exist in the original dataset. In contrast, the IntraAug branch uses newly generated reports to manipulate images. This process allows us to create new paired data for each individual with diverse medical conditions. Our extensive experiments on various downstream tasks covering medical image classification zero-shot and fine-tuning analysis demonstrate that our PairAug, concurrently expanding both image and text data, substantially outperforms image-/text-only expansion baselines and advanced medical VLP baselines. Our code is released at \url{https://github.com/YtongXie/PairAug}., Comment: Accepted to CVPR2024

In this paper,three external fields interacting with the four-level Y-type atomic system described by the density-matrix approach is investigated .The results show that the left-handedness with zero absorption are achieved.And the zero absorption property displays the possibility of manipulation with varying the phase and the intensity of the coupling field. The zero absorption property may be used to amplify the evanescent waves that have been lost in the imaging by traditional lenses.Our scheme proposes an approach to obtain negative refractive medium with zero absorption and the possibility to enhance the imaging resolution in realizing "superlenses"., Comment: 8 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:2403.00042; text overlap with arXiv:2403.00848

In this study, we investigate the electronic structure of a spin-triplet superconductor candidate CeRh$_2$As$_2$ using high-resolution angle-resolved photoemission spectroscopy and density functional theory calculations. Notably, Fermi surface nesting hints at connections to magnetic excitation or quadrupole density wave phenomena, elucidating the superconducting mechanisms. Measured band structures reveal primarily localized 4f electrons, with minor itinerant contributions. Additionally, a transition from localized to itinerant behavior and significant c-f hybridization anisotropy underscore the role of f-electrons in shaping electronic properties. These findings deepen our understanding of CeRh$_2$As$_2$'s unconventional superconductivity and magnetism. Further exploration promises advances in superconductivity research., Comment: 6 pages, 3 figures

Multimodal large language models (MLLMs) have attracted widespread interest and have rich applications. However, the inherent attention mechanism in its Transformer structure requires quadratic complexity and results in expensive computational overhead. Therefore, in this work, we propose VL-Mamba, a multimodal large language model based on state space models, which have been shown to have great potential for long-sequence modeling with fast inference and linear scaling in sequence length. Specifically, we first replace the transformer-based backbone language model such as LLama or Vicuna with the pre-trained Mamba language model. Then, we empirically explore how to effectively apply the 2D vision selective scan mechanism for multimodal learning and the combinations of different vision encoders and variants of pretrained Mamba language models. The extensive experiments on diverse multimodal benchmarks with competitive performance show the effectiveness of our proposed VL-Mamba and demonstrate the great potential of applying state space models for multimodal learning tasks.

Temporal misalignment (time offset) between sensors is common in low cost visual-inertial odometry (VIO) systems. Such temporal misalignment introduces inconsistent constraints for state estimation, leading to a significant positioning drift especially in high dynamic motion scenarios. In this article, we focus on online temporal calibration to reduce the positioning drift caused by the time offset for high dynamic motion VIO. For the time offset observation model, most existing methods rely on accurate state estimation or stable visual tracking. For the prediction model, current methods oversimplify the time offset as a constant value with white Gaussian noise. However, these ideal conditions are seldom satisfied in real high dynamic scenarios, resulting in the poor performance. In this paper, we introduce online time offset modeling networks (TON) to enhance real-time temporal calibration. TON improves the accuracy of time offset observation and prediction modeling. Specifically, for observation modeling, we propose feature velocity observation networks to enhance velocity computation for features in unstable visual tracking conditions. For prediction modeling, we present time offset prediction networks to learn its evolution pattern. To highlight the effectiveness of our method, we integrate the proposed TON into both optimization-based and filter-based VIO systems. Simulation and real-world experiments are conducted to demonstrate the enhanced performance of our approach. Additionally, to contribute to the VIO community, we will open-source the code of our method on: https://github.com/Franky-X/FVON-TPN.

2-D(Two-dimensional) isotropic negative refractive index (NRI) is explicitly realized via the orthogonal signal and coupling standing-wave fields coupling the N-type four-level atomic system. Under some key parameters of the dense vapor media, the atomic system exhibits isotropic NRI with simultaneous negative permittivity and permeability (i.e. Left-handedness) in the 2-D x-y plane. Compared with other 2-D NRI schemes, the coherent atomic vapor media in our scheme may be an ideal 2-D isotropic NRI candidate and has some potential advantages, significance or applications in the further investigation., Comment: 6 pages, 4 figures

Two-dimensional (2D) atom localization via the spontaneously generated coherence (SGC) and detunings associated with the probe and standing-wave driving fields in a three-level V-type atomic system are investigated. In the gain process, two equal and tunable peak maxima of position distribution in the plane via the detunings are observed. However, one decreasing and the other increasing peak maxima in the absorption process via the SGC are achieved in the quadrants I and III of the x-y plane. A better resolution and more novelty for the 2D atom localization in our scheme are obtained., Comment: 8 pages, 5 figures

Left-handedness with three zero-absorption windows is achieved in a triple-quantum dot (TQD) system. With the typical parameters of a GaAs/AlGaAs heterostructure, the simultaneous negative relative electric permittivity and magnetic permeability are obtained by the adjustable incoherent pumping field and two inter-dot tunnelings. What's more, three zero-absorption windows in the left-handedness frequency bands are observed in the TQD system. The left-handedness with zero-absorption in solid state heterostructure may solve not only the challenge in the photonic resonant scheme for left-handed materials (LHMs) but also the application limitation of the negative refractive materials with large amount of absorption., Comment: 5 pages, 4 figures

In recent years, Neural Radiance Fields (NeRFs) have demonstrated significant potential in encoding highly-detailed 3D geometry and environmental appearance, positioning themselves as a promising alternative to traditional explicit representation for 3D scene reconstruction. However, the predominant reliance on RGB imaging presupposes ideal lighting conditions: a premise frequently unmet in robotic applications plagued by poor lighting or visual obstructions. This limitation overlooks the capabilities of infrared (IR) cameras, which excel in low-light detection and present a robust alternative under such adverse scenarios. To tackle these issues, we introduce Thermal-NeRF, the first method that estimates a volumetric scene representation in the form of a NeRF solely from IR imaging. By leveraging a thermal mapping and structural thermal constraint derived from the thermal characteristics of IR imaging, our method showcasing unparalleled proficiency in recovering NeRFs in visually degraded scenes where RGB-based methods fall short. We conduct extensive experiments to demonstrate that Thermal-NeRF can achieve superior quality compared to existing methods. Furthermore, we contribute a dataset for IR-based NeRF applications, paving the way for future research in IR NeRF reconstruction.

Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL., Comment: Accepted at CVPR2024. Pre-print before final camera-ready version

Nonreciprocity, which denotes the asymmetric or even unidirectional transmission of light, constitutes the cornerstone of modern photonic circuits. In the realm of photonic devices, it has been widely utilized in isolators, circulators and so on. Recent topology in artificial materials, an unprecedented degree of freedom, has been proposed to solve the effect of impurities on nonreciprocal transmission. However, in view of the bulk-edge correspondence, the spatial width of the transmission channel with uniform field distribution is quite narrow and needs further exploration. In this paper, we proposed a one-way valley-locked waveguide with a large channel in an all-dielectric photonic crystal. Quite different from the topological edge modes, the unidirectional property of our waveguide comes from the bulk modes with valley-lock, which can fully utilize the whole dimension of the structure with an efficiency of 100%. Additionally, the electrical field is uniformly distributed across the entire channel, which opens a new avenue for low-loss nonreciprocity devices., Comment: 16 pages and 5 figures

This paper attempts a probe into negative refraction without absorption by means of an incoherent pump field and a strong coherent field coupling the dense four-level atomic system.With the application of the incoherent pump field to manipulate the populations in atomic levels and the variable strong coherent field to create quantum coherence, the constraint condition of two equal transition frequencies responding to the probe field in the atomic system isn't required.And these lead to the propagation transparency and strong magnetic response of the probe field,left-handedness with vanishing absorption in the atomic system.However,an excessive coherent field intensity would increase the absorption., Comment: 8 pages, 4 figures

The growing demand for personalized decision-making has led to a surge of interest in estimating the Conditional Average Treatment Effect (CATE). The intersection of machine learning and causal inference has yielded various effective CATE estimators. However, deploying these estimators in practice is often hindered by the absence of counterfactual labels, making it challenging to select the desirable CATE estimator using conventional model selection procedures like cross-validation. Existing approaches for CATE estimator selection, such as plug-in and pseudo-outcome metrics, face two inherent challenges. Firstly, they are required to determine the metric form and the underlying machine learning models for fitting nuisance parameters or plug-in learners. Secondly, they lack a specific focus on selecting a robust estimator. To address these challenges, this paper introduces a novel approach, the Distributionally Robust Metric (DRM), for CATE estimator selection. The proposed DRM not only eliminates the need to fit additional models but also excels at selecting a robust CATE estimator. Experimental studies demonstrate the efficacy of the DRM method, showcasing its consistent effectiveness in identifying superior estimators while mitigating the risk of selecting inferior ones.

What factors fundamentally determine the value of superconducting transition temperature (Tc) in high temperature superconductors has been the subject of intense debate. Following the establishment of an empirical law known as Homes'law, there is a growing consensus in the community that the Tc value of the cuprate superconductors is closely linked to its superfluid density and conductivity. However, all the data supporting this empirical law have been obtained from the ambient-pressure superconductors. In this study, we present the first high-pressure results about the connection of these two quantities with Tc, through the studies on the Bi1.74Pb0.38Sr1.88CuO6+delta and Bi2Sr2CaCu2O8+delta, in which the value of their high-pressure resistivity (the reciprocal of conductivity) is achieved by adopting our newly established method, while the value of superfluid density is extracted using the Homes'law. We highlight that the Tc values are strongly linked the two joint response factors of magnetic field and electric field, i.e. superfluid density and conductivity, respectively, implying that the physics governing the determination of Tc is influenced by the intrinsic electromagnetic fields of the system., Comment: 11 pages, 3 figures

The Silicon Charge Detector (SCD) is a subdetector of the High Energy Cosmic Radiation Detection payload. The dynamic range of the silicon microstrip detector can be extended by the capacitive coupling effect, which is related to the interstrip capacitance and the coupling capacitance. A detector prototype with several sets of parameters was designed and tested in the ion beams at the CERN Super Proton Synchrotron. The capacitive coupling fractions with readout strip and floating strip incidences were studied using the beam test data and SPICE simulation.

Fusion-based place recognition is an emerging technique jointly utilizing multi-modal perception data, to recognize previously visited places in GPS-denied scenarios for robots and autonomous vehicles. Recent fusion-based place recognition methods combine multi-modal features in implicit manners. While achieving remarkable results, they do not explicitly consider what the individual modality affords in the fusion system. Therefore, the benefit of multi-modal feature fusion may not be fully explored. In this paper, we propose a novel fusion-based network, dubbed EINet, to achieve explicit interaction of the two modalities. EINet uses LiDAR ranges to supervise more robust vision features for long time spans, and simultaneously uses camera RGB data to improve the discrimination of LiDAR point clouds. In addition, we develop a new benchmark for the place recognition task based on the nuScenes dataset. To establish this benchmark for future research with comprehensive comparisons, we introduce both supervised and self-supervised training schemes alongside evaluation protocols. We conduct extensive experiments on the proposed benchmark, and the experimental results show that our EINet exhibits better recognition performance as well as solid generalization ability compared to the state-of-the-art fusion-based place recognition approaches. Our open-source code and benchmark are released at: https://github.com/BIT-XJY/EINet.

The resistivity of a superconductor in its normal state plays a critical role in determining its superconducting ground state. However, measuring the resistivity of a material under high pressure has long presented a significant technical challenge due to pressure-induced changes in the crystallographic directions, especially for samples with anisotropic layered structures like high-Tc superconductors and other intriguing quantum materials. Here, we are the first to propose a novel and effective method for determining high-pressure resistivity, which relies on the ambient-pressure resistivity, initial sample sizes, lattice parameters, high-pressure resistance, and lattice parameters measured from the same sample. Its validity has been confirmed through our investigations of pressurized copper-oxide superconductors, which demonstrates that this method provides new possibilities for researchers conducting high-pressure studies related to resistivity of these materials., Comment: 8 pages, 1 figure

Vision-and-language navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavor to showcase the capability of VLMs to achieve state-of-the-art level navigation performance without any maps, odometers, or depth inputs. Following human instruction, NaVid only requires an on-the-fly video stream from a monocular RGB camera equipped on the robot to output the next-step action. Our formulation mimics how humans navigate and naturally gets rid of the problems introduced by odometer noises, and the Sim2Real gaps from map or depth inputs. Moreover, our video-based approach can effectively encode the historical observations of robots as spatio-temporal contexts for decision making and instruction following. We train NaVid with 510k navigation samples collected from continuous environments, including action-planning and instruction-reasoning samples, along with 763k large-scale web data. Extensive experiments show that NaVid achieves state-of-the-art performance in simulation environments and the real world, demonstrating superior cross-dataset and Sim2Real transfer. We thus believe our proposed VLM approach plans the next step for not only the navigation agents but also this research field., Comment: Accepted by Robotics: Science and Systems (RSS 2024)

In the present work, we investigate the production of the newly observed $P^\Lambda_{\psi s}(4338)$ state in $\Xi_b^-$ decay, where the $P^\Lambda_{\psi s}(4338)$ is assigned as a $\Xi_c \bar{D}$ molecular state. By using an effective Lagrangian approach, we evaluate the branching fractions of $\Xi_b^-\rightarrow P^\Lambda_{\psi s}(4338)K^-$ via the triangle loop mechanism. The branching fractions of $\Xi_b^-\rightarrow P^\Lambda_{\psi s}(4338)K^-$ are in the order of $10^{-4}$; the result is compared with our previous work of $\Xi_b^-\rightarrow P^\Lambda_{\psi s}(4459)K^-$. We also predict the ratio of $P^\Lambda_{\psi s}(4459)$ and $P^\Lambda_{\psi s}(4338)$ productions in the decay $\Xi_b^- \to P^\Lambda_{\psi s} K^- \to J/\psi \Lambda K^-$. The predicted branching fractions and their ratios could be tested experimentally, which may be helpful for understanding the molecular picture of $P^\Lambda_{\psi s}(4338)$ as well as other hidden-charm pentaquark states with strangeness. Moreover, the experimental potential of observing $P^\Lambda_{\psi s}(4338)$ in the $\Xi^-_b\to K^- J/\psi \Lambda$ is discussed., Comment: 7 pages, 5 figures

In financial modeling problems, non-Gaussian tails exist widely in many circumstances. Among them, the accurate estimation of risk-neutral distribution (RND) from option prices is of great importance for researchers and practitioners. A precise RND can provide valuable information regarding the market's expectations, and can further help empirical asset pricing studies. This paper presents a parsimonious parametric approach to extract RNDs of underlying asset returns by using a generative machine learning model. The model incorporates the asymmetric heavy tails property of returns with a clever design. To calibrate the model, we design a Monte Carlo algorithm that has good capability with the assistance of modern machine learning computing tools. Numerically, the model fits Heston option prices well and captures the main shapes of implied volatility curves. Empirically, using S\&P 500 index option prices, we demonstrate that the model outperforms some popular parametric density methods under mean absolute error. Furthermore, the skewness and kurtosis of RNDs extracted by our model are consistent with intuitive expectations. More generally, the proposed methodology is widely applicable in data fitting and probabilistic forecasting.

The atomic layer deposition(ALD) coating lengthens the lifetime of microchannel plates(MCP), which are used as the electron amplifier of the photomultiplier tubes(PMT). In the Jinping Neutrino Experiment, the newly developed 8-inch MCP-PMT achieves high collection efficiency by coating with high secondary emission materials. The resulting single electron response(SER) charge distribution deviates from the Gaussian distribution in large charge regions.To understand the nature of the jumbo-charged SER, we designed a voltage-division experiment to quantify the dependence of the MCP gain on the energy of incident electrons. Combining the relationship with the Furman probabilistic model, we reproduced the SER charge spectra by an additional amplification stage on the input electrode of the first MCP. Our results favor a Gamma-Tweedie mixture to describe the SER charge spectra of the MCP-PMTs.

This paper presents a algebraic analysis of electromagnetic chirality-induced negative refractive index in a four-level atomic medium. According to analyze mathematically its argument of the complex refractive index for one circular polarization, it found that the negative refractive index without simultaneously negative permittivity and permeability can be obtained when the argument is in the second quadrant of the cartesian coordinate system, and that the probe field coupling to two equal transition frequencies in the atomic level doesn't require. This undoubtedly reduced stringent conditions to negative refractive index by quantum optics. As an application, our scheme may possibly give a novel approach to obtain negative refractive index by electromagnetic chirality-inducing., Comment: 8 pages, 5 figures

We perform the quantum yields in a multi-band quantum dot (QD) photocell via doping an intermediate band (IB) between the conduction band (CB) and valence band (VB). Under two different sub-band gap layouts, the output power has a prominent enhancement than the single-band gap photocell and the achieved peak photo-to-charge efficiency reaches to 74.9% as compared to the limit efficiency of 63.2% via the IB approach in the theoretical solar cell prototype. The achieved quantum yields reveal the potential to improve efficiency by some effective theoretical approaches in the QD-IB photocell., Comment: 9 pages, 3 figures

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.