Search

This paper is concerned with initial-boundary-value problems of general multi-dimensional hyperbolic relaxation systems with characteristic boundaries. For the characteristic case, we redefine a Generalized Kreiss condition (GKC) which is essentially necessary to have a well-behaved relaxation limit. Under this characteristic GKC and a Shizuta-Kawashima-like condition, we derive reduced boundary conditions for the relaxation limit solving the corresponding equilibrium systems and justify the validity thereof. The key of the derivation is to select an elaborate version of the characteristic GKC by invoking the Shizuta-Kawashima-like condition. In contrast to the existing results, the present one does not assume that the boundary is non-characteristic for either the relaxation or equilibrium systems. In this sense, this paper completes the task in deriving reduced boundary conditions for general linear relaxation systems satisfying the structural stability condition.

In the realm of multimodal research, numerous studies leverage substantial image-text pairs to conduct modal alignment learning, transforming Large Language Models (LLMs) into Multimodal LLMs and excelling in a variety of visual-language tasks. The prevailing methodologies primarily fall into two categories: self-attention-based and cross-attention-based methods. While self-attention-based methods offer superior data efficiency due to their simple MLP architecture, they often suffer from lower computational efficiency due to concatenating visual and textual tokens as input for LLM. Conversely, cross-attention-based methods, although less data-efficient due to additional learnable parameters, exhibit higher computational efficiency by avoiding long sequence input for LLM. To address these trade-offs, we introduce the Data-Efficient and Compute-Efficient Multimodal Large Language Model (EE-MLLM). Without introducing additional modules or learnable parameters, EE-MLLM achieves both data and compute efficiency. Specifically, we modify the original self-attention mechanism in MLLM to a composite attention mechanism. This mechanism has two key characteristics: 1) Eliminating the computational overhead of self-attention within visual tokens to achieve compute efficiency, and 2) Reusing the weights on each layer of LLM to facilitate effective modality alignment between vision and language for data efficiency. Experimental results demonstrate the effectiveness of EE-MLLM across a range of benchmarks, including general-purpose datasets like MMBench and SeedBench, as well as fine-grained tasks such as TextVQA and DocVQA.

A reliable knowledge structure is a prerequisite for building effective adaptive learning systems and intelligent tutoring systems. Pursuing an explainable and trustworthy knowledge structure, we propose a method for constructing causal knowledge networks. This approach leverages Bayesian networks as a foundation and incorporates causal relationship analysis to derive a causal network. Additionally, we introduce a dependable knowledge-learning path recommendation technique built upon this framework, improving teaching and learning quality while maintaining transparency in the decision-making process., Comment: 8 pages, 3 figures, Educational Data Mining 2024, Human-Centric eXplainable AI in Education

This work is concerned with the exponential turnpike property for optimal control problems of particle systems and their mean-field limit. Under the assumption of the strict dissipativity of the cost function, exponential estimates for both optimal states and optimal control are proven. Moreover, we show that all the results for particle systems can be preserved under the limit in the case of infinitely many particles., Comment: 24 pages

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs), and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. The code and models will be made open-source.

Most multi-modal tasks can be formulated into problems of either generation or embedding. Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding. To explore the minimalism of multi-modal paradigms, we attempt to achieve only one model per modality in this work. We propose a Multi-Modal Generative Embedding Model (MM-GEM), whereby the generative and embedding objectives are encapsulated in one Large Language Model. We also propose a PoolAggregator to boost efficiency and enable the ability of fine-grained embedding and generation. A surprising finding is that these two objectives do not significantly conflict with each other. For example, MM-GEM instantiated from ViT-Large and TinyLlama shows competitive performance on benchmarks for multimodal embedding models such as cross-modal retrieval and zero-shot classification, while has good ability of image captioning. Additionally, MM-GEM can seamlessly execute region-level image caption generation and retrieval tasks. Besides, the advanced text model in MM-GEM brings over 5% improvement in Recall@1 for long text and image retrieval.

Humans constantly interact with their surrounding environments. Current human-centric generative models mainly focus on synthesizing humans plausibly interacting with static scenes and objects, while the dynamic human action-reaction synthesis for ubiquitous causal human-human interactions is less explored. Human-human interactions can be regarded as asymmetric with actors and reactors in atomic interaction periods. In this paper, we comprehensively analyze the asymmetric, dynamic, synchronous, and detailed nature of human-human interactions and propose the first multi-setting human action-reaction synthesis benchmark to generate human reactions conditioned on given human actions. To begin with, we propose to annotate the actor-reactor order of the interaction sequences for the NTU120, InterHuman, and Chi3D datasets. Based on them, a diffusion-based generative model with a Transformer decoder architecture called ReGenNet together with an explicit distance-based interaction loss is proposed to predict human reactions in an online manner, where the future states of actors are unavailable to reactors. Quantitative and qualitative results show that our method can generate instant and plausible human reactions compared to the baselines, and can generalize to unseen actor motions and viewpoint changes., Comment: Accepted by CVPR 2024, Project Page: https://liangxuy.github.io/ReGenNet/

The analysis of the ubiquitous human-human interactions is pivotal for understanding humans as social beings. Existing human-human interaction datasets typically suffer from inaccurate body motions, lack of hand gestures and fine-grained textual descriptions. To better perceive and generate human-human interactions, we propose Inter-X, a currently largest human-human interaction dataset with accurate body movements and diverse interaction patterns, together with detailed hand gestures. The dataset includes ~11K interaction sequences and more than 8.1M frames. We also equip Inter-X with versatile annotations of more than 34K fine-grained human part-level textual descriptions, semantic interaction categories, interaction order, and the relationship and personality of the subjects. Based on the elaborate annotations, we propose a unified benchmark composed of 4 categories of downstream tasks from both the perceptual and generative directions. Extensive experiments and comprehensive analysis show that Inter-X serves as a testbed for promoting the development of versatile human-human interaction analysis. Our dataset and benchmark will be publicly available for research purposes., Comment: Project page: https://liangxuy.github.io/inter-x/

Quantum Spin Hall (QSH) insulators possess distinct helical in-gap states, enabling their edge states to act as one-dimensional conducting channels when backscattering is prohibited by time-reversal symmetry. However, it remains challenging to achieve high-performance combinations of nontrivial topological QSH states with superconductivity for applications and requires understanding of the complicated underlying mechanisms. Here, our experimental observations for a novel superconducting phase in the pressurized QSH insulator Ta2Pd3Te5 is reported, and the high-pressure phase maintains its original ambient pressure lattice symmetry up to 45 GPa. Our in-situ high-pressure synchrotron X-ray diffraction, electrical transport, infrared reflectance, and Raman spectroscopy measurements, in combination with rigorous theoretical calculations, provide compelling evidence for the association between the superconducting behavior and the abnormal densified phase. The isostructural transition was found to modify the topology of the Fermi surface directly, accompanied by a fivefold amplification of the density of states at 20 GPa compared to ambient pressure, which synergistically fosters the emergence of robust superconductivity. A profound comprehension of the fascinating properties exhibited by the compressed Ta2Pd3Te5 phase is achieved, highlighting the extraordinary potential of van der Waals (vdW) QSH insulators for exploring and investigating high-performance electronic advanced devices under extreme conditions., Comment: 16pages,4figures

Image captioning requires numerous annotated image-text pairs, resulting in substantial annotation costs. Recently, large models (e.g. diffusion models and large language models) have excelled in producing high-quality images and text. This potential can be harnessed to create synthetic image-text pairs for training captioning models. Synthetic data can improve cost and time efficiency in data collection, allow for customization to specific domains, bootstrap generalization capability for zero-shot performance, and circumvent privacy concerns associated with real-world data. However, existing methods struggle to attain satisfactory performance solely through synthetic data. We identify the issue as generated images from simple descriptions mostly capture a solitary perspective with limited context, failing to align with the intricate scenes prevalent in real-world imagery. To tackle this, we present an innovative pipeline that introduces multi-context data generation. Beginning with an initial text corpus, our approach employs a large language model to extract multiple sentences portraying the same scene from diverse viewpoints. These sentences are then condensed into a single sentence with multiple contexts. Subsequently, we generate intricate images using the condensed captions through diffusion models. Our model is exclusively trained on synthetic image-text pairs crafted through this process. The effectiveness of our pipeline is validated through experimental results in both the in-domain and cross-domain settings, where it achieves state-of-the-art performance on well-known datasets such as MSCOCO, Flickr30k, and NoCaps., Comment: Accepted by AAAI 2024

This work concerns with boundary conditions (BCs) of the linearized moment system for rarefied gases. As the Knudsen number is sufficiently small, we analyze the boundary-layer behaviors of the moment system by resorting to a three-scale asymptotic expansion. The asymptotic analysis casts the flows into the outer solution, the viscous layer and the Knudsen layer. Starting from the BCs of the moment system, we propose a matching requirement and construct BCs for the Navier-Stokes equations. The obtained BCs contain the effect of second-order terms on the velocity slip and temperature jump. For the illustrative case of the Couette flow, we prove the validity of the constructed BCs through the error estimates. Meanwhile, numerical tests are presented to show the performance of the constructed BCs.

This work is concerned with coupling conditions for linear hyperbolic relaxation systems with multiple relaxation times. In the region with small relaxation time, an equilibrium system can be used for computational efficiency. Under the assumption that the relaxation system satisfies the structural stability condition and the interface is non-characteristic, we derive a coupling condition at the interface to couple the two systems in a domain decomposition setting. We prove the validity by the energy estimate and Laplace transform, which shows how the error of the domain decomposition method depends on the smaller relaxation time and the boundary layer effects. In addition, we propose a discontinuous Galerkin (DG) scheme for solving the interface problem with the derived coupling condition and prove the L2 stability. We validate our analysis on the linearized Carleman model and the linearized Grad's moment system and show the effectiveness of the DG scheme., Comment: 30 pages, 2 figures

Inspired by the fact that human eyes continue to develop tracking ability in early and middle childhood, we propose to use tracking as a proxy task for a computer vision system to learn the visual representations. Modelled on the Catch game played by the children, we design a Catch-the-Patch (CtP) game for a 3D-CNN model to learn visual representations that would help with video-related tasks. In the proposed pretraining framework, we cut an image patch from a given video and let it scale and move according to a pre-set trajectory. The proxy task is to estimate the position and size of the image patch in a sequence of video frames, given only the target bounding box in the first frame. We discover that using multiple image patches simultaneously brings clear benefits. We further increase the difficulty of the game by randomly making patches invisible. Extensive experiments on mainstream benchmarks demonstrate the superior performance of CtP against other video pretraining methods. In addition, CtP-pretrained features are less sensitive to domain gaps than those trained by a supervised action recognition task. When both trained on Kinetics-400, we are pleasantly surprised to find that CtP-pretrained representation achieves much higher action classification accuracy than its fully supervised counterpart on Something-Something dataset. Code is available online: github.com/microsoft/CtP., Comment: To appear in CVPR'21. Code available at github.com/microsoft/CtP