Search

The effectiveness and efficiency of modeling complex spectral-spatial relations are both crucial for Hyperspectral image (HSI) classification. Most existing methods based on CNNs and transformers still suffer from heavy computational burdens and have room for improvement in capturing the global-local spectral-spatial feature representation. To this end, we propose a novel lightweight parallel design called lightweight dual-stream Mamba-convolution network (DualMamba) for HSI classification. Specifically, a parallel lightweight Mamba and CNN block are first developed to extract global and local spectral-spatial features. First, the cross-attention spectral-spatial Mamba module is proposed to leverage the global modeling of Mamba at linear complexity. Within this module, dynamic positional embedding is designed to enhance the spatial location information of visual sequences. The lightweight spectral/spatial Mamba blocks comprise an efficient scanning strategy and a lightweight Mamba design to efficiently extract global spectral-spatial features. And the cross-attention spectral-spatial fusion is designed to learn cross-correlation and fuse spectral-spatial features. Second, the lightweight spectral-spatial residual convolution module is proposed with lightweight spectral and spatial branches to extract local spectral-spatial features through residual learning. Finally, the adaptive global-local fusion is proposed to dynamically combine global Mamba features and local convolution features for a global-local spectral-spatial representation. Compared with state-of-the-art HSI classification methods, experimental results demonstrate that DualMamba achieves significant classification accuracy on three public HSI datasets and a superior reduction in model parameters and floating point operations (FLOPs).

In online learning platforms, particularly in rapidly growing computer programming courses, addressing the thousands of students' learning queries requires considerable human cost. The creation of intelligent assistant large language models (LLMs) tailored for programming education necessitates distinct data support. However, in real application scenarios, the data resources for training such LLMs are relatively scarce. Therefore, to address the data scarcity in intelligent educational systems for programming, this paper proposes a new Chinese question-and-answer dataset for Python learners. To ensure the authenticity and reliability of the sources of the questions, we collected questions from actual student questions and categorized them according to various dimensions such as the type of questions and the type of learners. This annotation principle is designed to enhance the effectiveness and quality of online programming education, providing a solid data foundation for developing the programming teaching assists (TA). Furthermore, we conducted comprehensive evaluations of various LLMs proficient in processing and generating Chinese content, highlighting the potential limitations of general LLMs as intelligent teaching assistants in computer programming courses.

Weyl semimetals have been theoretically predicted to become topological metals with anomalous Hall conductivity in amorphous systems. However, measuring the anomalous Hall conductivity in realistic materials, particularly those with multiple pairs of Weyl points, is a significant challenge. If a system respects time-reversal symmetry, then the anomalous Hall conductivity even vanishes. As such, it remains an open question how to probe the Weyl band like topology in amorphous materials. Here, we theoretically demonstrate that, under magnetic fields, a topological metal slab in amorphous systems exhibits three-dimensional quantum Hall effect, even in time-reversal invariant systems, thereby providing a feasible approach to exploring Weyl band like topology in amorphous materials. We unveil the topological origin of the quantized Hall conductance by calculating the Bott index. The index is carried by broadened Landau levels with bulk states spatially localized except at critical transition energies. The topological property also results in edge states localized at distinct hinges on two opposite surfaces., Comment: 17 pages, 6 figures, including the appendix

Estimating the 3D structure of the human body from natural scenes is a fundamental aspect of visual perception. 3D human pose estimation is a vital step in advancing fields like AIGC and human-robot interaction, serving as a crucial technique for understanding and interacting with human actions in real-world settings. However, the current datasets, often collected under single laboratory conditions using complex motion capture equipment and unvarying backgrounds, are insufficient. The absence of datasets on variable conditions is stalling the progress of this crucial task. To facilitate the development of 3D pose estimation, we present FreeMan, the first large-scale, multi-view dataset collected under the real-world conditions. FreeMan was captured by synchronizing 8 smartphones across diverse scenarios. It comprises 11M frames from 8000 sequences, viewed from different perspectives. These sequences cover 40 subjects across 10 different scenarios, each with varying lighting conditions. We have also established an semi-automated pipeline containing error detection to reduce the workload of manual check and ensure precise annotation. We provide comprehensive evaluation baselines for a range of tasks, underlining the significant challenges posed by FreeMan. Further evaluations of standard indoor/outdoor human sensing datasets reveal that FreeMan offers robust representation transferability in real and complex scenes. Code and data are available at https://wangjiongw.github.io/freeman., Comment: CVPR2024 camera ready version. 19 pages, 16 figures. Project page: https://wangjiongw.github.io/freeman/ ; API: https://github.com/wangjiongw/FreeMan_API

In recent years, higher-order topological phases have attracted great interest in various fields of physics. These phases have protected boundary states at lower-dimensional boundaries than the conventional first-order topological phases due to the higher-order bulk-boundary correspondence. In this review, we summarize current research progress on higher-order topological phases in both crystalline and non-crystalline systems. We firstly introduce prototypical models of higher-order topological phases in crystals and their topological characterizations. We then discuss effects of quenched disorder on higher-order topology and demonstrate disorder-induced higher-order topological insulators. We also review the theoretical studies on higher-order topological insulators in amorphous systems without any crystalline symmetry and higher-order topological phases in nonperiodic lattices including quasicrystals, hyperbolic lattices, and fractals, which have no crystalline counterparts. We conclude the review by a summary of experimental realizations of higher-order topological phases and discussions on potential directions for future study., Comment: 52 pages, 27 figures. An invited topical review to Journal of Physics: Condensed Matter. Comments are welcome

Quasicrystals allow for symmetries that are impossible in crystalline materials, such as eight-fold rotational symmetry, enabling the existence of novel higher-order topological insulators in two dimensions without crystalline counterparts. However, it remains an open question whether three-dimensional higher-order topological insulators and Weyl-like semimetals without crystalline counterparts can exist. Here, we demonstrate the existence of a second-order topological insulator by constructing and exploring a three-dimensional model Hamiltonian in a stack of Ammann-Beenker tiling quasicrystalline lattices. The topological phase has eight chiral hinge modes that lead to quantized longitudinal conductances of $4 e^2/h$. We show that the topological phase is characterized by the winding number of the quadrupole moment. We further establish the existence of a second-order topological insulator with time-reversal symmetry, characterized by a $\mathbb{Z}_2$ topological invariant. Finally, we propose a model that exhibits a higher-order Weyl-like semimetal phase, demonstrating both hinge and surface Fermi arcs. Our findings highlight that quasicrystals in three dimensions can give rise to higher-order topological insulators and semimetal phases that are unattainable in crystals., Comment: 7 pages, 6 figures

The quadrupole insulator is a well-known instance of higher-order topological insulators in two dimensions, which possesses midgap corner states in both the energy spectrum and entanglement spectrum. Here, by constructing and exploring a model Hamiltonian under a staggered $\mathbb{Z}_2$ gauge field that respects momentum-glide reflection symmetries, we surprisingly find a quadrupole insulator that lacks zero-energy corner modes in its energy spectrum, despite possessing a nonzero quadrupole moment. Remarkably, the existence of midgap corner modes is found in the entanglement spectrum. Since these midgap states cannot be continuously eliminated, the quadrupole insulator cannot be continuously transformed into a trivial topological insulator, thereby confirming its topological nature. We show that the breakdown of the correspondence between the energy spectrum and entanglement spectrum occurs due to the closure of the edge energy gap when the Hamiltonian is flattened. Finally, we present a model that demonstrates an insulator with corner modes in the energy spectrum even in the absence of the quadrupole moment. In this phase, the entanglement spectrum does not display any midgap states. The results suggest that the bulk-edge correspondence of quadrupole insulators generally manifests in the entanglement spectrum rather than the energy spectrum., Comment: 7 pages, 5 figures

Large language models have emerged as a promising approach towards achieving general-purpose AI agents. The thriving open-source LLM community has greatly accelerated the development of agents that support human-machine dialogue interaction through natural language processing. However, human interaction with the world extends beyond only text as a modality, and other modalities such as vision are also crucial. Recent works on multi-modal large language models, such as GPT-4V and Bard, have demonstrated their effectiveness in handling visual modalities. However, the transparency of these works is limited and insufficient to support academic research. To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs, with a specific focus on facilitating AI agents capable of bridging the gap between ideas and execution, thereby enabling seamless human-AI interaction. Our main contribution is three-fold: 1) We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We outline the detailed methodology of constructing multi-modal instruction tuning datasets and benchmarks for MLLMs, enabling rapid scaling and extension of MLLM research to diverse domains, tasks, and modalities. 3) We provide a primary but potential MLLM training framework optimized for modality extension. We also provide baseline models, comprehensive experimental observations, and analysis to accelerate future research. Our baseline model is trained within 24 A100 GPU hours, framework supports training with V100 and RTX3090 is available thanks to the open-source society., Comment: NeurIPS2023 camera ready ; 37 pages, 33 figures. Code available at https://github.com/OpenLAMM/LAMM ; Project page: https://openlamm.github.io/

While topological phases have been extensively studied in amorphous systems in recent years, it remains unclear whether the random nature of amorphous materials can give rise to higher-order topological phases that have no crystalline counterparts. Here we theoretically demonstrate the existence of higher-order topological insulators in two-dimensional amorphous systems that can host more than six corner modes, such as eight or twelve corner modes. Although individual sample configuration lacks crystalline symmetry, we find that an ensemble of all configurations exhibits an average crystalline symmetry that provides protection for the new topological phases. To characterize the topological phases, we construct two topological invariants. Even though the bulk energy gap in the topological phase vanishes in the thermodynamic limit, we show that the bulk states near zero energy are localized, as supported by the level-spacing statistics and inverse participation ratio. Our findings open an avenue for exploring average symmetry protected higher-order topological phases in amorphous systems without crystalline counterparts., Comment: 17 pages, 7 figures

Set-based face recognition (SFR) aims to recognize the face sets in the unconstrained scenario, where the appearance of same identity may change dramatically with extreme variances (e.g., illumination, pose, expression). We argue that the two crucial issues in SFR, the face quality and burstiness, are both identity-irrelevant and variance-relevant. The quality and burstiness assessment are interfered with by the entanglement of identity, and the face recognition is interfered with by the entanglement of variance. Thus we propose to separate the identity features with the variance features in a light-weighted set-based disentanglement framework. Beyond disentanglement, the variance features are fully utilized to indicate face quality and burstiness in a set, rather than being discarded after training. To suppress face burstiness in the sets, we propose a vocabulary-based burst suppression (VBS) method which quantizes faces with a reference vocabulary. With interword and intra-word normalization operations on the assignment scores, the face burtisness degrees are appropriately estimated. The extensive illustrations and experiments demonstrate the effect of the disentanglement framework with VBS, which gets new state-of-the-art on the SFR benchmarks. The code will be released at https://github.com/Liubinggunzu/set_burstiness., Comment: ACM MM 2022 accepted, code will be released

DNN-based speaker verification (SV) models demonstrate significant performance at relatively high computation costs. Model compression can be applied to reduce the model size for lower resource consumption. The present study exploits weight quantization to compress two widely-used SV models, namely ECAPA-TDNN and ResNet. Experimental results on VoxCeleb show that weight quantization is effective for compressing SV models. The model size can be reduced multiple times without noticeable degradation in performance. Compression of ResNet shows more robust results than ECAPA-TDNN with lower-bitwidth quantization. Analysis of the layer weights suggests that the smooth weight distribution of ResNet may be related to its better robustness. The generalization ability of the quantized model is validated via a language-mismatched SV task. Furthermore, analysis by information probing reveals that the quantized models can retain most of the speaker-relevant knowledge learned by the original models., Comment: Accepted by INTERSPEECH2023

Multi-modal video question answering aims to predict correct answer and localize the temporal boundary relevant to the question. The temporal annotations of questions improve QA performance and interpretability of recent works, but they are usually empirical and costly. To avoid the temporal annotations, we devise a weakly supervised question grounding (WSQG) setting, where only QA annotations are used and the relevant temporal boundaries are generated according to the temporal attention scores. To substitute the temporal annotations, we transform the correspondence between frames and subtitles to Frame-Subtitle (FS) self-supervision, which helps to optimize the temporal attention scores and hence improve the video-language understanding in VideoQA model. The extensive experiments on TVQA and TVQA+ datasets demonstrate that the proposed WSQG strategy gets comparable performance on question grounding, and the FS self-supervision helps improve the question answering and grounding performance on both QA-supervision only and full-supervision settings.

In this paper, we study the problem of shape-programming of incompressible hyperelastic shells through differential growth. The aim of the current work is to determine the growth tensor (or growth functions) that can produce the deformation of a shell to the desired shape. First, a consistent finite-strain shell theory is introduced. The shell equation system is established from the 3D governing system through a series expansion and truncation approach. Based on the shell theory, the problem of shape-programming is studied under the stress-free assumption. For a special case in which the parametric coordinate curves generate a net of curvature lines on the target surface, the sufficient condition to ensure the vanishing of the stress components is analyzed, from which the explicit expression of the growth tensor can be derived. In the general case, we conduct the variable changes and derive the total growth tensor by considering a two-step deformation of the shell. With these obtained results, a general theoretical scheme for shape-programming of thin hyperelastic shells through differential growth is proposed. To demonstrate the feasibility and efficiency of the proposed scheme, several nature-inspired examples are studied. The derived growth tensors in these examples have also been implemented in the numerical simulations to verify their correctness and accuracy. The simulation results show that the target shapes of the shell samples can be recovered completely. The scheme for shape-programming proposed in the current work is helpful in designing and manufacturing intelligent soft devices.

Weakly Supervised Object Localization (WSOL), which aims to localize objects by only using image-level labels, has attracted much attention because of its low annotation cost in real applications. Recent studies leverage the advantage of self-attention in visual Transformer for long-range dependency to re-active semantic regions, aiming to avoid partial activation in traditional class activation mapping (CAM). However, the long-range modeling in Transformer neglects the inherent spatial coherence of the object, and it usually diffuses the semantic-aware regions far from the object boundary, making localization results significantly larger or far smaller. To address such an issue, we introduce a simple yet effective Spatial Calibration Module (SCM) for accurate WSOL, incorporating semantic similarities of patch tokens and their spatial relationships into a unified diffusion model. Specifically, we introduce a learnable parameter to dynamically adjust the semantic correlations and spatial context intensities for effective information propagation. In practice, SCM is designed as an external module of Transformer, and can be removed during inference to reduce the computation cost. The object-sensitive localization ability is implicitly embedded into the Transformer encoder through optimization in the training phase. It enables the generated attention maps to capture the sharper object boundaries and filter the object-irrelevant background area. Extensive experimental results demonstrate the effectiveness of the proposed method, which significantly outperforms its counterpart TS-CAM on both CUB-200 and ImageNet-1K benchmarks. The code is available at https://github.com/164140757/SCM., Comment: Accepted by ECCV2022

In this paper, we propose a multi-layered hyperelastic plate theory of growth within the framework of nonlinear elasticity. First, the 3D governing system for a general multi-layered hyperelastic plate is established, which incorporates the growth effect, and the material and geometrical parameters of the different layers. Then, a series expansion-truncation approach is adopted to eliminate the thickness variables in the 3D governing system. An elaborate calculation scheme is applied to derive the iteration relations of the coefficient functions in the series expansions. Through some further manipulations, a 2D vector plate equation system with the associated boundary conditions is established, which only contains the unknowns in the bottom layer of the plate. To show the efficiency of the current plate theory, three typical examples regarding the growth-induced deformations and instabilities of multi-layered plate samples are studied. Some analytical and numerical solutions to the plate equation are obtained, which can provide accurate predictions on the growth behaviors of the plate samples. Furthermore, the problem of `shape-programming' of multi-layered hyperelastic plates through differential growth is studied. The explicit formulas of shape-programming for some typical multi-layered plates are derived, which involve the fundamental quantities of the 3D target shapes. By using these formulas, the shape evolutions of the plates during the growing processes can be controlled accurately. The results obtained in the current work are helpful for the design of intelligent soft devices with multi-layered plate structures.

For face presentation attack detection (PAD), most of the spoofing cues are subtle, local image patterns (e.g., local image distortion, 3D mask edge and cut photo edges). The representations of existing PAD works with simple global pooling method, however, lose the local feature discriminability. In this paper, the VLAD aggregation method is adopted to quantize local features with visual vocabulary locally partitioning the feature space, and hence preserve the local discriminability. We further propose the vocabulary separation and adaptation method to modify VLAD for cross-domain PADtask. The proposed vocabulary separation method divides vocabulary into domain-shared and domain-specific visual words to cope with the diversity of live and attack faces under the cross-domain scenario. The proposed vocabulary adaptation method imitates the maximization step of the k-means algorithm in the end-to-end training, which guarantees the visual words be close to the center of assigned local features and thus brings robust similarity measurement. We give illustrations and extensive experiments to demonstrate the effectiveness of VLAD with the proposed vocabulary separation and adaptation method on standard cross-domain PAD benchmarks. The codes are available at https://github.com/Liubinggunzu/VLAD-VSA., Comment: ACM MM 2021

In this paper, a theoretical scheme is proposed for shape-programming of thin hyperelastic plates through differential growth. First, starting from the 3D governing system of a hyperelastic (neo-Hookean) plate, a consistent finite-strain plate equation system is formulated through a series-expansion and truncation approach. Based on the plate equation system, the problem of shape-programming is studied under the stress-free assumption. By equating the stress components in the plate equations to be zero, the explicit relations between growth functions and geometrical quantities of the target shape of the plate are derived. Then, a theoretical scheme of shape-programming is proposed, which can be used to identify the growth fields corresponding to arbitrary 3D shapes of the plate. To demonstrate the efficiency of the scheme, some typical examples are studied. The predicted growth functions in these examples are adopted in the numerical simulations, from which the target shapes of the plate can be recovered completely. The scheme of shape-programming proposed in the current work is applicable for manufacture of intelligent soft devices.

Non-Hermitian materials can not only exhibit exotic energy band structures but also an anomalous velocity induced by non-Hermitian anomalous Berry connection as predicted by the semiclassical equations of motion for Bloch electrons. However, it is not clear how the modified semiclassical dynamics modifies transport phenomena. Here, we theoretically demonstrate the emergence of anomalous oscillations driven by either an external dc or ac electric field, which arise from non-Hermitian anomalous Berry connection. Moreover, it is a well-known fact that geometric structures of electric wave functions can only affect the Hall conductivity. However, we are surprised to find a non-Hermitian anomalous Berry connection induced anomalous linear longitudinal conductivity independent of the scattering time. We also show the emergence of a second-order nonlinear longitudinal conductivity induced by non-Hermitian anomalous Berry connection, violating a well-known fact of its absence in a Hermitian system with symmetric energy spectra. These anomalous phenomena are illustrated in a pseudo-Hermitian system with large non-Hermitian anomalous Berry connection. Finally, we propose a practical scheme to realize the anomalous oscillations in an optical system., Comment: 13 pages including the supplementary material, 4 figures