Search

Autoregressive models have demonstrated great performance in natural language processing (NLP) with impressive scalability, adaptability and generalizability. Inspired by their notable success in NLP field, autoregressive models have been intensively investigated recently for computer vision, which perform next-token predictions by representing visual data as visual tokens and enables autoregressive modelling for a wide range of vision tasks, ranging from visual generation and visual understanding to the very recent multimodal generation that unifies visual generation and understanding with a single autoregressive model. This paper provides a systematic review of vision autoregressive models, including the development of a taxonomy of existing methods and highlighting their major contributions, strengths, and limitations, covering various vision tasks such as image generation, video generation, image editing, motion generation, medical image analysis, 3D generation, robotic manipulation, unified multimodal generation, etc. Besides, we investigate and analyze the latest advancements in autoregressive models, including thorough benchmarking and discussion of existing methods across various evaluation datasets. Finally, we outline key challenges and promising directions for future research, offering a roadmap to guide further advancements in vision autoregressive models.

This work proposes and analyzes an efficient numerical method for solving the nonlinear Schr\"odinger equation with quasiperiodic potential, where the projection method is applied in space to account for the quasiperiodic structure and the Strang splitting method is used in time.While the transfer between spaces of low-dimensional quasiperiodic and high-dimensional periodic functions and its coupling with the nonlinearity of the operator splitting scheme make the analysis more challenging. Meanwhile, compared to conventional numerical analysis of periodic Schr\"odinger systems, many of the tools and theories are not applicable to the quasiperiodic case. We address these issues to prove the spectral accuracy in space and the second-order accuracy in time. Numerical experiments are performed to substantiate the theoretical findings.

Remote Sensing Image Captioning (RSIC) presents unique challenges and plays a critical role in applications. Traditional RSIC methods often struggle to produce rich and diverse descriptions. Recently, with advancements in VLMs, efforts have emerged to integrate these models into the remote sensing domain and to introduce descriptive datasets specifically designed to enhance VLM training. This paper proposes RS-MoE, a first Mixture of Expert based VLM specifically customized for remote sensing domain. Unlike traditional MoE models, the core of RS-MoE is the MoE Block, which incorporates a novel Instruction Router and multiple lightweight Large Language Models (LLMs) as expert models. The Instruction Router is designed to generate specific prompts tailored for each corresponding LLM, guiding them to focus on distinct aspects of the RSIC task. This design not only allows each expert LLM to concentrate on a specific subset of the task, thereby enhancing the specificity and accuracy of the generated captions, but also improves the scalability of the model by facilitating parallel processing of sub-tasks. Additionally, we present a two-stage training strategy for tuning our RS-MoE model to prevent performance degradation due to sparsity. We fine-tuned our model on the RSICap dataset using our proposed training strategy. Experimental results on the RSICap dataset, along with evaluations on other traditional datasets where no additional fine-tuning was applied, demonstrate that our model achieves state-of-the-art performance in generating precise and contextually relevant captions. Notably, our RS-MoE-1B variant achieves performance comparable to 13B VLMs, demonstrating the efficiency of our model design. Moreover, our model demonstrates promising generalization capabilities by consistently achieving state-of-the-art performance on the Remote Sensing Visual Question Answering (RSVQA) task.

Generalizing to out-of-distribution data while being aware of model fairness is a significant and challenging problem in meta-learning. The goal of this problem is to find a set of fairness-aware invariant parameters of classifier that is trained using data drawn from a family of related training domains with distribution shift on non-sensitive features as well as different levels of dependence between model predictions and sensitive features so that the classifier can achieve good generalization performance on unknown but distinct test domains. To tackle this challenge, existing state-of-the-art methods either address the domain generalization problem but completely ignore learning with fairness or solely specify shifted domains with various fairness levels. This paper introduces an approach to fairness-aware meta-learning that significantly enhances domain generalization capabilities. Our framework, Fairness-Enhanced Meta-Learning for Domain Generalization (FEED), disentangles latent data representations into content, style, and sensitive vectors. This disentanglement facilitates the robust generalization of machine learning models across diverse domains while adhering to fairness constraints. Unlike traditional methods that focus primarily on domain invariance or sensitivity to shifts, our model integrates a fairness-aware invariance criterion directly into the meta-learning process. This integration ensures that the learned parameters uphold fairness consistently, even when domain characteristics vary widely. We validate our approach through extensive experiments across multiple benchmarks, demonstrating not only superior performance in maintaining high accuracy and fairness but also significant improvements over existing state-of-the-art methods in domain generalization tasks., Comment: IEEE International Conference on Big Data 2024

Crystalline defects play a critical role in determining the properties of crystalline solids, underscoring the need for accurate computational methods to study them. Lattice deformation in dislocation simulations, which involves changes in atomic positions, can be described either microscopically by specific atomic configurations or macroscopically by continuum elasticity, each with inherent limitations. The complex amplitude expansion of the phase field crystal (APFC) model provides a mesoscopic approach that bridges these scales. In this paper, we introduce a Fourier pseudospectral method for efficiently solving the APFC model in the context of crystalline defect simulations. This study marks the first application of the Fourier pseudospectral method to the APFC model. The method fully exploits the system's periodicity and facilitates the implementation of periodic boundary conditions, thanks to its high accuracy and computational efficiency. Numerical experiments conducted on two-dimensional triangular lattices and three-dimensional body-centered cubic lattices for edge dislocation geometry optimization have produced strain field images that align well with by continuum elasticity predictions. The findings demonstrate the potential of the APFC model to accurately capture the complex strain fields associated with dislocations at the mesoscopic scales, a key step toward modeling more intricate crystalline defect structures and dynamics.

Recent studies on generalizable object detection have attracted increasing attention with additional weak supervision from large-scale datasets with image-level labels. However, weakly-supervised detection learning often suffers from image-to-box label mismatch, i.e., image-level labels do not convey precise object information. We design Language Hierarchical Self-training (LHST) that introduces language hierarchy into weakly-supervised detector training for learning more generalizable detectors. LHST expands the image-level labels with language hierarchy and enables co-regularization between the expanded labels and self-training. Specifically, the expanded labels regularize self-training by providing richer supervision and mitigating the image-to-box label mismatch, while self-training allows assessing and selecting the expanded labels according to the predicted reliability. In addition, we design language hierarchical prompt generation that introduces language hierarchy into prompt generation which helps bridge the vocabulary gaps between training and testing. Extensive experiments show that the proposed techniques achieve superior generalization performance consistently across 14 widely studied object detection datasets., Comment: NeurIPS 2024 Camera Ready

X-shaped liquid crystalline molecules (XLCMs) are obtained by tethering two flexible end A-blocks and two flexible side B-blocks to a rigid backbone (R). A rich array of ordered structures can be formed from XLCMs, driven by the competition between the interactions between the chemically distinct blocks and the molecular connectivity. Here, we report a theoretical study on the phase behaviour of XLCMs with symmetric and asymmetric side blocks by using the self-consistent field theory (SCFT). A large number of ordered structures, including stable smectic-A, triangle-square, pentagon and giant polygon, are obtained as solutions of the SCFT equations. Phase diagrams of XLCMs as a function of the total length and asymmetric ratio of the side chains are constructed. For XLCMs with symmetric side blocks, the theoretically predicted phase transition sequence is in good agreement with experiments. For XLCMs with a fixed total side chain length, transitions between layered structure to polygonal phases, as well as between different polygonal phases, could be induced by varying the asymmetry of the side chains. The free energy density, domain size, side-chain stretching , and molecular orientation are analyzed to elucidate mechanisms stabilizing the different ordered phases.

Quasicrystals are intriguing ordered structures characterized by the lack of translational symmetry and the existence of rotational symmetry. The tiling of different geometric units such as triangles and squares in two-dimensional space can result in a great variety of quasicrystals that could be realized by the self-assembly of liquid crystalline molecules. In this study, we introduce three self-similar dodecagonal tilings, including a novel Diamond-Square-Triangle pattern, composed of triangular and quadrangular tiles and examine their thermodynamic stability by using the self-consistent field theory applied to T-shaped liquid crystalline molecules. Specifically, we detail the inflation rules for the construction of these dodecagonal tilings and analyze their self-similarity, and show that these tilings can be viewed as projections of higher-dimensional periodic lattice points with projection windows. Using these dodecagonal tilings as initial configurations of the SCFT results in solutions corresponding to quasicrystals that could form from the T-shaped liquid crystalline molecules. The relative stability of these aperiodic phases is analyzed to obtain design rules that could stabilize quasicrystals. Meanwhile, we provide two criteria for distinguishing three dodecagonal quasicrystals and their approximants by analyzing their diffraction peaks. These findings shed new lighten on the discovery of new quasicrystals in soft materials.

The recent experiment shows the superconducting transition temperature in the Ruddlesden-Popper bilayer La$_3$Ni$_2$O$_{7}$ decreases monotonically with increasing pressure above 14 GPa. In order to unravel the underlying mechanism for this unusual dependence, we performed theoretical investigations by combining the density functional theory (DFT) and the unbiased functional renormalization group (FRG). Our DFT calculations show that the Fermi pockets are essentially unchanged with increasing pressure (above 14 GPa), but the bandwidth is enlarged, and particularly the interlayer hopping integral between the nickel $3d_{3z^2-r^2}$ orbitals is enhanced. From the DFT band structure, we construct the bilayer tight-binding model in terms of the nickel $3d_{3z^2-r^2}$ and $3d_{x^2-y^2}$ orbitals. On this basis, we investigate the superconductivity induced by correlation effects by FRG calculations. We find consistently $s_\pm$-wave pairing triggered by spin fluctuations, but the latter are weakened by pressure and lead to a decreasing transition temperature versus pressure, in qualitatively agreement with the experiment. We emphasize that the itinerancy of the $d$-orbitals is important and captured naturally in our FRG calculations, and we argue that the unusual pressure dependence would be unnatural, if not impossible, in the otherwise local-moment picture of the nickel $d$-orbitals. This sheds lights on the pertinent microscopic description of, and more importantly the mechanism of superconductivity in La$_3$Ni$_2$O$_{7}$., Comment: 8 pages, 4 figures

Interparticle interactions with multiple length scales play a pivotal role in the formation and stability of quasicrystals. Choosing a minimal set of length scales to stabilize a given quasicrystal is a challenging problem. To address this challenge, we propose an intelligent screening method (ISM) to design a Landau theory with a minimal number of length scales -- referred to as the minimal Landau theory -- that includes only the essential length scales necessary to stabilize quasicrystals. Based on a generalized multiple-length-scale Landau theory, ISM first evaluates various spectral configurations of candidate structures under a hard constraint. It then identifies the configuration with the lowest free energy. Using this optimal configuration, ISM calculates phase diagrams to explore the thermodynamic stability of desired quasicrystals. ISM can design a minimal Landau theory capable of stabilizing the desired quasicrystals by incrementally increasing the number of length scales. Our application of ISM has not only confirmed known behaviors in 10- and 12-fold quasicrystals but also led to a significant prediction that quasicrystals with 8-, 14-, 16-, and 18-fold symmetry could be stable within three-length-scale Landau models., Comment: 23 pages, 10 figures, 2 long tables

The phase behavior of symmetric diblock copolymers under three-dimensional (3D) soft confinement is investigated using the self-consistent field theory. The soft confinement is realized in binary blends composed AB diblock copolymers and C homopolymers, where the copolymers self-assemble to form a droplet embedded in the homopolymer matrix. The phase behavior of the confined block copolymers is regulated by the degree of confinement and the selectivity of the homopolymers, resulting in a rich variety of novel structures. When the C homopolymers are neutral to the A- and B-blocks, stacked lamellae (SL) are formed where the number of layers increases with the droplet volume, resulting in a morphological transition sequence from Janus particle to square SL. When the C homopolymers are strongly selective to the B-blocks, a series of non-lamellar morphologies, including onion-, hamburger-, cross-, ring-, and cookie-like structures, are observed. A detailed free energy analysis reveals a first-order reversible transformation between SL and onion-like (OL) structures when the selectivity of the homopolymers is changed. Our results provide a comprehensive understanding of how various factors, such as the copolymer concentration, homopolymer chain length, degree of confinement, homopolymer selectivity, affect the self-assembled structures of diblock copolymers under soft 3D confinement., Comment: 9pages,7figures

Recently, deep learning-based hybrid iterative methods (DL-HIM) have emerged as a promising approach for designing fast neural solvers to tackle large-scale sparse linear systems. DL-HIM combine the smoothing effect of simple iterative methods with the spectral bias of neural networks, which allows them to effectively eliminate both high-frequency and low-frequency error components. However, their efficiency may decrease if simple iterative methods can not provide effective smoothing, making it difficult for the neural network to learn mid-frequency and high-frequency components. This paper first establishes a convergence analysis framework for general DL-HIM from a spectral viewpoint, concluding that under reasonable assumptions, DL-HIM exhibit a convergence rate independent of grid size $h$ and physical parameters $\boldsymbol{\mu}$. To meet these assumptions, we design a neural network from an eigen perspective, focusing on learning the eigenvalues and eigenvectors corresponding to error components that simple iterative methods struggle to eliminate. Specifically, the eigenvalues are learned by a meta subnet, while the eigenvectors are approximated using Fourier modes with a transition matrix provided by another meta subnet. The resulting DL-HIM, termed the Fourier Neural Solver (FNS), is expected to achieve a $h, \boldsymbol{\mu}$-independent convergence rate with the guarantee of universal approximation and the discretization invariant property of the meta subnets. We verify the performance of FNS on four types of linear parametric PDEs.

In this work, we study the nucleation of quasicrystals from liquid or periodic crystals by developing an efficient order-order phase transition algorithm, namely the nullspace-preserving saddle search method. Specifically, we focus on nucleation and phase transitions of the decagonal quasicrystal (DQC) based on the Lifshitz-Petrich model. We present the nucleation path of DQC from the liquid and demonstrate one- and two-stage transition paths between DQC and periodic crystals. We provide a perspective of the group-subgroup phase transition and nucleation rates to understand the nucleation and phase transition mechanisms involving DQC. These results reveal the one-step and stepwise modes of symmetry breaking or recovery in the phase transition from DQC, where the stepwise modes are more probable.

The mental health assessment of middle school students has always been one of the focuses in the field of education. This paper introduces a new ensemble learning network based on BERT, employing the concept of enhancing model performance by integrating multiple classifiers. We trained a range of BERT-based learners, which combined using the majority voting method. We collect social network text data of middle school students through China's Weibo and apply the method to the task of classifying emotional tendencies in middle school students' social network texts. Experimental results suggest that the ensemble learning network has a better performance than the base model and the performance of the ensemble learning model, consisting of three single-layer BERT models, is barely the same as a three-layer BERT model but requires 11.58% more training time. Therefore, in terms of balancing prediction effect and efficiency, the deeper BERT network should be preferred for training. However, for interpretability, network ensembles can provide acceptable solutions.

Finding index-1 saddle points is crucial for understanding phase transitions. In this work, we propose a simple yet efficient approach, the spring pair method (SPM), to accurately locate saddle points. Without requiring Hessian information, SPM evolves a single pair of spring-coupled particles on the potential energy surface. By cleverly designing complementary drifting and climbing dynamics based on gradient decomposition, the spring pair converges onto the minimum energy path (MEP) and spontaneously aligns its orientation with the MEP tangent, providing a reliable ascent direction for efficient convergence to saddle points. SPM fundamentally differs from traditional surface walking methods, which rely on the eigenvectors of Hessian that may deviate from the MEP tangent, potentially leading to convergence failure or undesired saddle points. The efficiency of SPM for finding saddle points is verified by ample examples, including high-dimensional Lennard-Jones cluster rearrangement and the Landau energy functional involving quasicrystal phase transitions., Comment: 9 pages, 7 figures

In this work, we investigate [110] symmetric tilt FCC grain boundaries (GBs) by a recently developed approach for quasiperiodic interfaces using the Landau-Brazovskii model. On special tilt angles associated with quadratic algebraic numbers, quasiperiodic GBs exhibit generalized Fibonacci substitution rules. The transition mechanism from quasiperiodic GBs to periodic GBs is explored through sphere offsets and spectral coalescence. We also propose an accurate method to calculate the GB energy for arbitrary tilt angle and analyze the factors affecting the GB energy. The revealing GB energy change is continue along with tilt angle except periodic GBs.

The majority of existing hyperspectral anomaly detection (HAD) methods use the low-rank representation (LRR) model to separate the background and anomaly components, where the anomaly component is optimized by handcrafted sparse priors (e.g., $\ell_{2,1}$-norm). However, this may not be ideal since they overlook the spatial structure present in anomalies and make the detection result largely dependent on manually set sparsity. To tackle these problems, we redefine the optimization criterion for the anomaly component in the LRR model with a self-supervised network called self-supervised anomaly prior (SAP). This prior is obtained by the pretext task of self-supervised learning, which is customized to learn the characteristics of hyperspectral anomalies. Specifically, this pretext task is a classification task to distinguish the original hyperspectral image (HSI) and the pseudo-anomaly HSI, where the pseudo-anomaly is generated from the original HSI and designed as a prism with arbitrary polygon bases and arbitrary spectral bands. In addition, a dual-purified strategy is proposed to provide a more refined background representation with an enriched background dictionary, facilitating the separation of anomalies from complex backgrounds. Extensive experiments on various hyperspectral datasets demonstrate that the proposed SAP offers a more accurate and interpretable solution than other advanced HAD methods.

Self-consistent field theory (SCFT) is one of the most widely-used framework in studying the equilibrium phase behaviors of inhomogenous polymers. For liquid crystalline polymeric systems, the main numerical challenges of solving SCFT encompass efficiently solving plenty of six dimensional partial differential equations (PDEs), precisely determining the subtle energy difference among self-assembled structures, and developing effective iterative methods for nonlinear SCF iteration. To address these challenges, this work introduces a suite of high-order and efficient numerical methods tailored for SCFT of liquid-crystalline polymers. These methods include various advaced PDE solvers, an improved Anderson iteration algorithm to accelerate SCFT calculations, and an optimization technique of adjusting the computational domain during the SCF iterations. Extensive numerical tests demonstrate the efficiency of the proposed methods. Based on these algorithms, we further explore the self-assembly behavior of liquid crystalline polymers through simulations in four, five, and six dimensions, uncovering intricate three-dimensional spatial structures., Comment: 34 pages, 13figures

In this study, we address the challenge of solving elliptic equations with quasiperiodic coefficients. To achieve accurate and efficient computation, we introduce the projection method, which enables the embedding of quasiperiodic systems into higher-dimensional periodic systems. To enhance the computational efficiency, we propose a compressed storage strategy for the stiffness matrix by its multi-level block circulant structure, significantly reducing memory requirements. Furthermore, we design a diagonal preconditioner to efficiently solve the resulting high-dimensional linear system by reducing the condition number of the stiffness matrix. These techniques collectively contribute to the computational effectiveness of our proposed approach. Convergence analysis shows the spectral accuracy of the proposed method. We demonstrate the effectiveness and accuracy of our approach through a series of numerical examples. Moreover, we apply our method to achieve a highly accurate computation of the homogenized coefficients for a quasiperiodic multiscale elliptic equation.

In this paper, we propose a new algorithm, the irrational-window-filter projection method (IWFPM), for solving arbitrary dimensional global quasiperiodic systems. Based on the projection method (PM), IWFPM further utilizes the concentrated distribution of Fourier coefficients to filter out relevant spectral points using an irrational window. Moreover, a corresponding index-shift transform is designed to make the Fast Fourier Transform available. The corresponding error analysis on the function approximation level is also given. We apply IWFPM to 1D, 2D, and 3D quasiperiodic Schr\"odinger eigenproblems to demonstrate its accuracy and efficiency. IWFPM exhibits a significant computational advantage over PM for both extended and localized quantum states. Furthermore, the widespread existence of such spectral point distribution feature can endow IWFPM with significant potential for broader applications in quasiperiodic systems.

Solving high-wavenumber and heterogeneous Helmholtz equations presents a long-standing challenge in scientific computing. In this paper, we introduce a deep learning-enhanced multigrid solver to address this issue. By conducting error analysis on standard multigrid applied to a discrete Helmholtz equation, we devise a strategy to handle errors with different frequencies separately. For error components with frequencies distant from the wavenumber, we perform simple smoothing based on local operations at different levels to eliminate them. On the other hand, to address error components with frequencies near the wavenumber, we utilize another multigrid V-cycle to solve an advection-diffusion-reaction (ADR) equation at a coarse scale. The resulting solver, named Wave-ADR-NS, involves parameters learned through unsupervised training. Numerical results demonstrate that Wave-ADR-NS effectively resolves heterogeneous 2D Helmholtz equation with wavenumber up to 2000. Comparative experiments against classical multigrid preconditioners and existing deep learning-based multigrid preconditioners reveals the superior performance of Wave-ADR-NS.

The fairness-aware online learning framework has emerged as a potent tool within the context of continuous lifelong learning. In this scenario, the learner's objective is to progressively acquire new tasks as they arrive over time, while also guaranteeing statistical parity among various protected sub-populations, such as race and gender, when it comes to the newly introduced tasks. A significant limitation of current approaches lies in their heavy reliance on the i.i.d (independent and identically distributed) assumption concerning data, leading to a static regret analysis of the framework. Nevertheless, it's crucial to note that achieving low static regret does not necessarily translate to strong performance in dynamic environments characterized by tasks sampled from diverse distributions. In this paper, to tackle the fairness-aware online learning challenge in evolving settings, we introduce a unique regret measure, FairSAR, by incorporating long-term fairness constraints into a strongly adapted loss regret framework. Moreover, to determine an optimal model parameter at each time step, we introduce an innovative adaptive fairness-aware online meta-learning algorithm, referred to as FairSAOML. This algorithm possesses the ability to adjust to dynamic environments by effectively managing bias control and model accuracy. The problem is framed as a bi-level convex-concave optimization, considering both the model's primal and dual parameters, which pertain to its accuracy and fairness attributes, respectively. Theoretical analysis yields sub-linear upper bounds for both loss regret and the cumulative violation of fairness constraints. Our experimental evaluation on various real-world datasets in dynamic environments demonstrates that our proposed FairSAOML algorithm consistently outperforms alternative approaches rooted in the most advanced prior online learning methods., Comment: Accepted by TKDD, extended from KDD 2022. arXiv admin note: substantial text overlap with arXiv:2205.11264

We study four double-gyroid (DG) grain boundaries (GBs) with different orientations numerically using the Landau--Brazovskii free energy, including the (422) twin boundary studied recently, a network switching GB, and two tilt GBs. Topological variations and geometric deformations are investigated. It is found that deviations in strut lengths and dihedral angles from the bulk DG substantially exceed changes in strut angles and nodal coplanarity. We also examine the spectra along the contact plane of two grains and utilize them to evaluate the GB widths. Of the four GBs we study, the network switching GB changes to the least extent topologically and geometrically, meanwhile has the lowest energy and the smallest GB width.

The recent discovery of bulk superconductivity in trilayer nickelate La$_4$Ni$_3$O$_{10}$ with the critical temperature $T_c$ near $30$K under high pressure is attracting a new wave of research interest, after the breakthrough of bilayer La$_3$Ni$_2$O$_7$ with $T_c$ near $80$K. The similarities and differences of electronic structure and superconducting mechanism in these two systems are urgent theoretical issues. In this Letter, we study the electronic band structure and construct a minimal trilayer tight-binding model for the high-pressure phase of La$_4$Ni$_3$O$_{10}$ in terms of the nickel $3d_{x^2-y^2}$ and $3d_{3z^2-r^2}$ orbitals, and study the superconducting mechanism due to local Coulomb interactions by the unbiased functional renormalization group. We find antiferromagnetic correlations between the outer layers instead of neighboring ones, apart from the inplane correlations. The effective interaction induces Cooper pairing with the $s_\pm$-wave symmetry, which changes sign across the Fermi pockets. We find $T_c$ in La$_4$Ni$_3$O$_{10}$ is systematically lower than that in La$_3$Ni$_2$O$_7$, and electron doping can enhance $T_c$., Comment: 4 pages, 4 figures, 1 table

Large-vocabulary object detectors (LVDs) aim to detect objects of many categories, which learn super objectness features and can locate objects accurately while applied to various downstream data. However, LVDs often struggle in recognizing the located objects due to domain discrepancy in data distribution and object vocabulary. At the other end, recent vision-language foundation models such as CLIP demonstrate superior open-vocabulary recognition capability. This paper presents KGD, a Knowledge Graph Distillation technique that exploits the implicit knowledge graphs (KG) in CLIP for effectively adapting LVDs to various downstream domains. KGD consists of two consecutive stages: 1) KG extraction that employs CLIP to encode downstream domain data as nodes and their feature distances as edges, constructing KG that inherits the rich semantic relations in CLIP explicitly; and 2) KG encapsulation that transfers the extracted KG into LVDs to enable accurate cross-domain object classification. In addition, KGD can extract both visual and textual KG independently, providing complementary vision and language knowledge for object localization and object classification in detection tasks over various downstream domains. Experiments over multiple widely adopted detection benchmarks show that KGD outperforms the state-of-the-art consistently by large margins.

Camera-only Bird's Eye View (BEV) has demonstrated great potential in environment perception in a 3D space. However, most existing studies were conducted under a supervised setup which cannot scale well while handling various new data. Unsupervised domain adaptive BEV, which effective learning from various unlabelled target data, is far under-explored. In this work, we design DA-BEV, the first domain adaptive camera-only BEV framework that addresses domain adaptive BEV challenges by exploiting the complementary nature of image-view features and BEV features. DA-BEV introduces the idea of query into the domain adaptation framework to derive useful information from image-view and BEV features. It consists of two query-based designs, namely, query-based adversarial learning (QAL) and query-based self-training (QST), which exploits image-view features or BEV features to regularize the adaptation of the other. Extensive experiments show that DA-BEV achieves superior domain adaptive BEV perception performance consistently across multiple datasets and tasks such as 3D object detection and 3D scene segmentation.

Segment Anything Models (SAMs) like SEEM and SAM have demonstrated great potential in learning to segment anything. The core design of SAMs lies with Promptable Segmentation, which takes a handcrafted prompt as input and returns the expected segmentation mask. SAMs work with two types of prompts including spatial prompts (e.g., points) and semantic prompts (e.g., texts), which work together to prompt SAMs to segment anything on downstream datasets. Despite the important role of prompts, how to acquire suitable prompts for SAMs is largely under-explored. In this work, we examine the architecture of SAMs and identify two challenges for learning effective prompts for SAMs. To this end, we propose spatial-semantic prompt learning (SSPrompt) that learns effective semantic and spatial prompts for better SAMs. Specifically, SSPrompt introduces spatial prompt learning and semantic prompt learning, which optimize spatial prompts and semantic prompts directly over the embedding space and selectively leverage the knowledge encoded in pre-trained prompt encoders. Extensive experiments show that SSPrompt achieves superior image segmentation performance consistently across multiple widely adopted datasets.

Accurate cloud recognition and warning are crucial for various applications, including in-flight support, weather forecasting, and climate research. However, recent deep learning algorithms have predominantly focused on detecting cloud regions in satellite imagery, with insufficient attention to the specificity required for accurate cloud recognition. This limitation inspired us to develop the novel FY-4A-Himawari-8 (FYH) dataset, which includes nine distinct cloud categories and uses precise domain adaptation methods to align 70,419 image-label pairs in terms of projection, temporal resolution, and spatial resolution, thereby facilitating the training of supervised deep learning networks. Given the complexity and diversity of cloud formations, we have thoroughly analyzed the challenges inherent to cloud recognition tasks, examining the intricate characteristics and distribution of the data. To effectively address these challenges, we designed a Distribution-aware Interactive-Attention Network (DIAnet), which preserves pixel-level details through a high-resolution branch and a parallel multi-resolution cross-branch. We also integrated a distribution-aware loss (DAL) to mitigate the imbalance across cloud categories. An Interactive Attention Module (IAM) further enhances the robustness of feature extraction combined with spatial and channel information. Empirical evaluations on the FYH dataset demonstrate that our method outperforms other cloud recognition networks, achieving superior performance in terms of mean Intersection over Union (mIoU). The code for implementing DIAnet is available at https://github.com/icey-zhang/DIAnet.

Numerical solving the Schr\"odinger equation with incommensurate potentials presents a great challenge since its solutions could be space-filling quasiperiodic structures without translational symmetry nor decay. In this paper, we propose two high-accuracy numerical methods to solve the time-dependent quasiperiodic Schr\"odinger equation. Concretely, we discretize the spatial variables by the quasiperiodic spectral method and the projection method, and the time variable by the second-order operator splitting method. The corresponding convergence analysis is also presented and shows that the proposed methods both have exponential convergence rate in space and second order accuracy in time, respectively. Meanwhile, we analyse the computational complexity of these numerical algorithms. One- and two-dimensional numerical results verify these convergence conclusions, and demonstrate that the projection method is more efficient.

Traditional computer vision generally solves each single task independently by a dedicated model with the task instruction implicitly designed in the model architecture, arising two limitations: (1) it leads to task-specific models, which require multiple models for different tasks and restrict the potential synergies from diverse tasks; (2) it leads to a pre-defined and fixed model interface that has limited interactivity and adaptability in following user' task instructions. To address them, Visual Instruction Tuning (VIT) has been intensively studied recently, which finetunes a large vision model with language as task instructions, aiming to learn from a wide range of vision tasks described by language instructions a general-purpose multimodal model that can follow arbitrary instructions and thus solve arbitrary tasks specified by the user. This work aims to provide a systematic review of visual instruction tuning, covering (1) the background that presents computer vision task paradigms and the development of VIT; (2) the foundations of VIT that introduce commonly used network architectures, visual instruction tuning frameworks and objectives, and evaluation setups and tasks; (3) the commonly used datasets in visual instruction tuning and evaluation; (4) the review of existing VIT methods that categorizes them with a taxonomy according to both the studied vision task and the method design and highlights the major contributions, strengths, and shortcomings of them; (5) the comparison and discussion of VIT methods over various instruction-following benchmarks; (6) several challenges, open directions and possible future works in visual instruction tuning research.

The endeavor to preserve the generalization of a fair and invariant classifier across domains, especially in the presence of distribution shifts, becomes a significant and intricate challenge in machine learning. In response to this challenge, numerous effective algorithms have been developed with a focus on addressing the problem of fairness-aware domain generalization. These algorithms are designed to navigate various types of distribution shifts, with a particular emphasis on covariate and dependence shifts. In this context, covariate shift pertains to changes in the marginal distribution of input features, while dependence shift involves alterations in the joint distribution of the label variable and sensitive attributes. In this paper, we introduce a simple but effective approach that aims to learn a fair and invariant classifier by simultaneously addressing both covariate and dependence shifts across domains. We assert the existence of an underlying transformation model can transform data from one domain to another, while preserving the semantics related to non-sensitive attributes and classes. By augmenting various synthetic data domains through the model, we learn a fair and invariant classifier in source domains. This classifier can then be generalized to unknown target domains, maintaining both model prediction and fairness concerns. Extensive empirical studies on four benchmark datasets demonstrate that our approach surpasses state-of-the-art methods., Comment: Accepted by KDD 2024 research track

In this work, we propose an efficient nullspace-preserving saddle search (NPSS) method for a class of phase transitions involving translational invariance, where the critical states are often degenerate. The NPSS method includes two stages, escaping from the basin and searching for the index-1 generalized saddle point. The NPSS method climbs upward from the generalized local minimum in segments to overcome the challenges of degeneracy. In each segment, an effective ascent direction is ensured by keeping this direction orthogonal to the nullspace of the initial state in this segment. This method can escape the basin quickly and converge to the transition states. We apply the NPSS method to the phase transitions between crystals, and between crystal and quasicrystal, based on the Landau-Brazovskii and Lifshitz-Petrich free energy functionals. Numerical results show a good performance of the NPSS method.

Extensive experimental studies have shown that numerous ordered phases can be formed via the self-assembly of T-shaped liquid crystalline molecules (TLCMs) composed of a rigid backbone, two flexible end chains and a flexible side chain. However, a comprehensive understanding of the stability and formation mechanisms of these intricately nanostructured phases remains incomplete. Here we fill this gap by carrying out a theoretical study of the phase behaviour of TLCMs. Specifically, we construct phase diagrams of TLCMs by computing the free energy of different ordered phases of the system. Our results reveal that the number of polygonal edges increases as the length of side chain or interaction strength increases, consistent with experimental observations. The theoretical study not only reproduces the experimentally observed phases and phase transition sequences, but also systematically analyzes the stability mechanism of the polygonal phases.

Quasiperiodic systems, related to irrational numbers, are space-filling structures without decay nor translation invariance. How to accurately recover these systems, especially for non-smooth cases, presents a big challenge in numerical computation. In this paper, we propose a new algorithm, finite points recovery (FPR) method, which is available for both smooth and non-smooth cases, to address this challenge. The FPR method first establishes a homomorphism between the lower-dimensional definition domain of the quasiperiodic function and the higher-dimensional torus, then recovers the global quasiperiodic system by employing interpolation technique with finite points in the definition domain without dimensional lifting. Furthermore, we develop accurate and efficient strategies of selecting finite points according to the arithmetic properties of irrational numbers. The corresponding mathematical theory, convergence analysis, and computational complexity analysis on choosing finite points are presented. Numerical experiments demonstrate the effectiveness and superiority of FPR approach in recovering both smooth quasiperiodic functions and piecewise constant Fibonacci quasicrystals. While existing spectral methods encounter difficulties in accurately recovering non-smooth quasiperiodic functions.

Two prevalent types of distributional shifts in machine learning are the covariate shift (as observed across different domains) and the semantic shift (as seen across different classes). Traditional OOD detection techniques typically address only one of these shifts. However, real-world testing environments often present a combination of both covariate and semantic shifts. In this study, we introduce a novel problem, semantic OOD detection across domains, which simultaneously addresses both distributional shifts. To this end, we introduce two regularization strategies: domain generalization regularization, which ensures semantic invariance across domains to counteract the covariate shift, and OOD detection regularization, designed to enhance OOD detection capabilities against the semantic shift through energy bounding. Through rigorous testing on three standard domain generalization benchmarks, our proposed framework showcases its superiority over conventional domain generalization approaches in terms of OOD detection performance. Moreover, it holds its ground by maintaining comparable InD classification accuracy.