Search

3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. Traditional methods depending on supervised learning with 3D point clouds are limited by scarce datasets. Recently zero-shot methods leveraging LLMs have been proposed to address the data issue. While effective, these methods only use object-centric information, limiting their ability to handle complex queries. In this work, we present VLM-Grounder, a novel framework using vision-language models (VLMs) for zero-shot 3D visual grounding based solely on 2D images. VLM-Grounder dynamically stitches image sequences, employs a grounding and feedback scheme to find the target object, and uses a multi-view ensemble projection to accurately estimate 3D bounding boxes. Experiments on ScanRefer and Nr3D datasets show VLM-Grounder outperforms previous zero-shot methods, achieving 51.6% Acc@0.25 on ScanRefer and 48.0% Acc on Nr3D, without relying on 3D geometry or object priors. Codes are available at https://github.com/OpenRobotLab/VLM-Grounder ., Comment: CoRL 2024 Camera Ready. 25 pages. A novel zero-shot 3D visual grounding framework based solely on 2D images

Whole-brain proton magnetic resonance spectroscopic imaging (1H-MRSI) is a non-invasive technique for assessing neurochemical distribution in the brain, offering valuable insights into brain functions and neural diseases. It greatly benefits from the improved SNR at ultrahigh field strengths ($\geq$7T). However, 1H-MRSI still faces several challenges, such as long acquisition time and severe signal contaminations from water and lipids. In this study, 2D and 3D short TR/TE 1H-FID-MRSI sequences using rosette trajectories were developed with spatial resolutions of 4.48$\times$4.48 mm$^2$ and 4.48$\times$4.48$\times$4.50 mm$^3$, respectively. Water signals were suppressed using an optimized Five-variable-Angle-gaussian-pulses-with-ShorT-total-duration of 76 ms (FAST) water suppression scheme, and lipid signals were removed using the L2 regularization method. Metabolic maps of major 1H metabolites were obtained within 5:40 min with 16 averages and 1 average for the 2D and 3D acquisitions, respectively. Excellent inter-session reproducibility was shown, with the coefficients of variance (CV) being lower than 6% for N-Acetyle-L-aspartic acid (NAA), Glutamate (Glu), Choline Chloride and glycerophosphocholine (tCho), Creatine and Phosphocreatine (tCr), and Glycine and Myo-inositol (Gly+Ins). To explore the potential of further accelerating the acquisition, compressed sensing was applied retrospectively to the 3D datasets. The structural similarity index (SSIM) remained above 0.85 and 0.8 until $R = 2$ and $R = 3$ for the metabolite maps of Glu, NAA, tCr, and tCho, indicating the possibility for further reduction of acquisition time to around 2min., Comment: 25 pages, 9 figures

Background: X-nuclei imaging (e.g. 31P MRI) suffers low SNR due to lower gyromagnetic ratios and tissue concentrations than 1H MRI. Enhancing SNR requires high fields, advanced coils, optimized sequences, and sophisticated reconstruction methods. An optimized pipeline is essential for maximizing SNR in X-nuclei imaging. Purpose: To develop a toolbox of reconstruction and processing methods for fast 31P MR imaging, evaluating optimal pipelines for two data types: 31P-GRE and bSSFP-like MRF imaging. Methods: The toolbox includes coil combination (adaptive and whitened-SVD), k-space filtering, reconstruction (Kaiser-Bessel regridding with FFT and NUFFT), and denoising (compressed sensing and MP-PCA). 31P MR datasets were acquired at 7T using spiral encoding with a double-tuned birdcage coil and a 32-channel phased array. GRE parameters: readout 18.62ms, flip angle 59deg, TR 5400ms, TE 0.3ms. MRF used a bSSFP-like scheme: readout 9.12ms, TR 19.8ms, TE 0.3ms. Results: For GRE data, adaptive coil combination with Kaiser-Bessel regridding (AC-KB) achieved highest mean SNRs: 38 (phantom) and 3.4 (in vivo), outperforming others. For MRF data, whitened-SVD with Kaiser-Bessel regridding yielded an SNR of 18, surpassing AC-KB's 16. MP-PCA denoised GRE effectively; compressed sensing suited MRF data. Conclusion: We demonstrated a toolbox for optimizing fast 31P MR imaging pipelines. Optimal methods depend on acquisition and data. Kaiser-Bessel regridding outperforms NUFFT. Adaptive coil combination and MP-PCA denoising are preferred for GRE data; whitened-SVD and compressed sensing are superior for MRF data. Careful selection of processing methods is crucial, as conventional 1H MRI techniques may not suffice. The toolbox facilitates future investigation for other X-nuclei datasets., Comment: 20 pages, 9 figures

Natural language question answering (QA) over structured data sources such as tables and knowledge graphs (KGs) have been widely investigated, for example with Large Language Models (LLMs). The main solutions include question to formal query parsing and retrieval-based answer generation. However, current methods of the former often suffer from weak generalization, failing to dealing with multiple sources simultaneously, while the later is limited in trustfulness. In this paper, we propose UnifiedTQA, a trustful QA framework that can simultaneously support multiple types of structured data in a unified way. To this end, it adopts an LLM-friendly and unified knowledge representation method called Condition Graph (CG), and uses an LLM and demonstration-based two-level method for CG querying. For enhancement, it is also equipped with dynamic demonstration retrieval. We have evaluated UnifiedTQA with 5 benchmarks covering 3 types of structured data. It outperforms 2 existing unified structured data QA methods and in comparison with the baselines that are specific to a data type, it achieves state-of-the-art on 2 of them. Further more, we demonstrates potential of our method for more general QA tasks, QA over mixed structured data and QA across structured data.

Purpose: Phosphorus Magnetic Resonance Spectroscopy (31P MRS) enables non-invasive assessment of energy metabolism, yet its application is hindered by sensitivity limitations. To overcome this, often high magnetic fields are used, leading to challenges such as spatial B_1^+ inhomogeneity and therefore the need for accurate flip angle determination in accelerated acquisitions with short repetition times (T_R). In response to these challenges, we propose a novel short T_R and look-up table-based Double-Angle Method for fast 3D 31P B_1^+ mapping (fDAM). Methods: Our method incorporates 3D weighted stack of spiral gradient echo acquisitions and a frequency-selective pulse to enable efficient B_1^+ mapping based on the phosphocreatine signal at 7T. Protocols were optimised using simulations and validated through phantom experiments. The method was validated in phantom experiments and skeletal muscle applications using a birdcage 1H/31P volume coil. Results: The results of fDAM were compared to the classical DAM (cDAM). A good correlation (r=0.94) was obtained between the two B_1^+ maps. A 3D 31P B_1^+ mapping in the human calf muscle was achieved in about 10 min using a birdcage volume coil, with a 20% extended coverage relative to that of the cDAM (24 min). fDAM also enabled the first full brain coverage 31P 3D B_1^+ mapping in approx. 10 min using a 1 Tx/ 32 Rx coil. Conclusion: fDAM is an efficient method for 31P 3D B_1^+ mapping, showing promise for future applications in rapid 31P MRSI.

Knowledge graph (KG) completion aims to find out missing triples in a KG. Some tasks, such as link prediction and instance completion, have been proposed for KG completion. They are triple-level tasks with some elements in a missing triple given to predict the missing element of the triple. However, knowing some elements of the missing triple in advance is not always a realistic setting. In this paper, we propose a novel graph-level automatic KG completion task called Triple Set Prediction (TSP) which assumes none of the elements in the missing triples is given. TSP is to predict a set of missing triples given a set of known triples. To properly and accurately evaluate this new task, we propose 4 evaluation metrics including 3 classification metrics and 1 ranking metric, considering both the partial-open-world and the closed-world assumptions. Furthermore, to tackle the huge candidate triples for prediction, we propose a novel and efficient subgraph-based method GPHT that can predict the triple set fast. To fairly compare the TSP results, we also propose two types of methods RuleTensor-TSP and KGE-TSP applying the existing rule- and embedding-based methods for TSP as baselines. During experiments, we evaluate the proposed methods on two datasets extracted from Wikidata following the relation-similarity partial-open-world assumption proposed by us, and also create a complete family data set to evaluate TSP results following the closed-world assumption. Results prove that the methods can successfully generate a set of missing triples and achieve reasonable scores on the new task, and GPHT performs better than the baselines with significantly shorter prediction time. The datasets and code for experiments are available at https://github.com/zjukg/GPHT-for-TSP., Comment: Paper accepted by TKDE in 2024

LiDAR-camera extrinsic calibration (LCEC) is crucial for data fusion in intelligent vehicles. Offline, target-based approaches have long been the preferred choice in this field. However, they often demonstrate poor adaptability to real-world environments. This is largely because extrinsic parameters may change significantly due to moderate shocks or during extended operations in environments with vibrations. In contrast, online, target-free approaches provide greater adaptability yet typically lack robustness, primarily due to the challenges in cross-modal feature matching. Therefore, in this article, we unleash the full potential of large vision models (LVMs), which are emerging as a significant trend in the fields of computer vision and robotics, especially for embodied artificial intelligence, to achieve robust and accurate online, target-free LCEC across a variety of challenging scenarios. Our main contributions are threefold: we introduce a novel framework known as MIAS-LCEC, provide an open-source versatile calibration toolbox with an interactive visualization interface, and publish three real-world datasets captured from various indoor and outdoor environments. The cornerstone of our framework and toolbox is the cross-modal mask matching (C3M) algorithm, developed based on a state-of-the-art (SoTA) LVM and capable of generating sufficient and reliable matches. Extensive experiments conducted on these real-world datasets demonstrate the robustness of our approach and its superior performance compared to SoTA methods, particularly for the solid-state LiDARs with super-wide fields of view., Comment: accepted to IEEE Trans. on Intelligent Vehicles (T-IV)

This work introduces Weaver, our first family of large language models (LLMs) dedicated to content creation. Weaver is pre-trained on a carefully selected corpus that focuses on improving the writing capabilities of large language models. We then fine-tune Weaver for creative and professional writing purposes and align it to the preference of professional writers using a suit of novel methods for instruction data synthesis and LLM alignment, making it able to produce more human-like texts and follow more diverse instructions for content creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for different applications and can be dynamically dispatched by a routing agent according to query complexity to balance response quality and computation cost. Evaluation on a carefully curated benchmark for assessing the writing capabilities of LLMs shows Weaver models of all sizes outperform generalist LLMs several times larger than them. Notably, our most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes. Moreover, Weaver natively supports retrieval-augmented generation (RAG) and function calling (tool usage). We present various use cases of these abilities for improving AI-assisted writing systems, including integration of external knowledge bases, tools, or APIs, and providing personalized writing assistance. Furthermore, we discuss and summarize a guideline and best practices for pre-training and fine-tuning domain-specific LLMs.

As the development of academic conferences fosters global scholarly communication, researchers consistently need to obtain accurate and up-to-date information about academic conferences. Since the information is scattered, using an intelligent question-answering system to efficiently handle researchers' queries and ensure awareness of the latest advancements is necessary. Recently, Large Language Models (LLMs) have demonstrated impressive capabilities in question answering, and have been enhanced by retrieving external knowledge to deal with outdated knowledge. However, these methods fail to work due to the lack of the latest conference knowledge. To address this challenge, we develop the ConferenceQA dataset, consisting of seven diverse academic conferences. Specifically, for each conference, we first organize academic conference data in a tree-structured format through a semi-automated method. Then we annotate question-answer pairs and classify the pairs into four different types to better distinguish their difficulty. With the constructed dataset, we further propose a novel method STAR (STructure-Aware Retrieval) to improve the question-answering abilities of LLMs, leveraging inherent structural information during the retrieval process. Experimental results on the ConferenceQA dataset show the effectiveness of our retrieval method. The dataset and code are available at https://github.com/zjukg/ConferenceQA., Comment: 12 pages, 3 figures

This paper reports on the NTIRE 2023 Quality Assessment of Video Enhancement Challenge, which will be held in conjunction with the New Trends in Image Restoration and Enhancement Workshop (NTIRE) at CVPR 2023. This challenge is to address a major challenge in the field of video processing, namely, video quality assessment (VQA) for enhanced videos. The challenge uses the VQA Dataset for Perceptual Video Enhancement (VDPVE), which has a total of 1211 enhanced videos, including 600 videos with color, brightness, and contrast enhancements, 310 videos with deblurring, and 301 deshaked videos. The challenge has a total of 167 registered participants. 61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions. A total of 176 submissions were submitted by 37 participating teams during the final testing phase. Finally, 19 participating teams submitted their models and fact sheets, and detailed the methods they used. Some methods have achieved better results than baseline methods, and the winning methods have demonstrated superior prediction performance.

Since the dynamic characteristics of knowledge graphs, many inductive knowledge graph representation learning (KGRL) works have been proposed in recent years, focusing on enabling prediction over new entities. NeuralKG-ind is the first library of inductive KGRL as an important update of NeuralKG library. It includes standardized processes, rich existing methods, decoupled modules, and comprehensive evaluation metrics. With NeuralKG-ind, it is easy for researchers and engineers to reproduce, redevelop, and compare inductive KGRL methods. The library, experimental methodologies, and model re-implementing results of NeuralKG-ind are all publicly released at https://github.com/zjukg/NeuralKG/tree/ind ., Comment: Accepted by SIGIR2023 Demonstration Track

Mode-locked lasers play the role of the ideal testbeds for studying self-coherent structures, dissipative solitons, with stable spatiotemporal profiles supported by the balance between dispersion and nonlinearity. However, under some conditions, the profile and energy of the solitons can oscillate (breath) periodically. Unlike the previous studies of different scalar mechanisms of breathers emergence, we demonstrate experimentally and theoretically a new vector mechanism where breathing of the output power is a result of the heteroclinic dynamics. The heteroclinic orbit is a trajectory periodically evolving nearby the neighborhood of one of the orthogonal states of polarization with further switching to and evolving nearby the other state of polarization. The dwelling time for the trajectory near each orthogonal state is determined by the cavity anisotropy adjustable with the help of the polarization controller for the pump wave. The obtained results on tunability of breathing dynamics enables unlocking novel techniques of the flexible control and manipulation of the polarization dynamics of dissipative solitons., Comment: 14 pages, 9 figures