Search

High-resolution hyperspectral imaging plays a crucial role in various remote sensing applications, yet its acquisition often faces fundamental limitations due to hardware constraints. This paper introduces S$^{3}$RNet, a novel framework for hyperspectral image pansharpening that effectively combines low-resolution hyperspectral images (LRHSI) with high-resolution multispectral images (HRMSI) through sparse spatial-spectral representation. The core of S$^{3}$RNet is the Multi-Branch Fusion Network (MBFN), which employs parallel branches to capture complementary features at different spatial and spectral scales. Unlike traditional approaches that treat all features equally, our Spatial-Spectral Attention Weight Block (SSAWB) dynamically adjusts feature weights to maintain sparse representation while suppressing noise and redundancy. To enhance feature propagation, we incorporate the Dense Feature Aggregation Block (DFAB), which efficiently aggregates inputted features through dense connectivity patterns. This integrated design enables S$^{3}$RNet to selectively emphasize the most informative features from differnt scale while maintaining computational efficiency. Comprehensive experiments demonstrate that S$^{3}$RNet achieves state-of-the-art performance across multiple evaluation metrics, showing particular strength in maintaining high reconstruction quality even under challenging noise conditions. The code will be made publicly available., Comment: Submitted to IGARSS 2025

Efficient and accurate learning of constitutive laws is crucial for accurately predicting the mechanical behavior of materials under complex loading conditions. Accurate model calibration hinges on a delicate interplay between the information embedded in experimental data and the parameters that define our constitutive models. The information encoded in the parameters of the constitutive model must be complemented by the information in the data used for calibration. This interplay raises fundamental questions: How can we quantify the information content of test data? How much information does a single test convey? Also, how much information is required to accurately learn a constitutive model? To address these questions, we introduce mechanics informatics, a paradigm for efficient and accurate constitutive model learning. At its core is the stress state entropy, a metric quantifying the information content of experimental data. Using this framework, we analyzed specimen geometries with varying information content for learning an anisotropic inelastic law. Specimens with limited information enabled accurate identification of a few parameters sensitive to the information in the data. Furthermore, we optimized specimen design by incorporating stress state entropy into a Bayesian optimization scheme. This led to the design of cruciform specimens with maximized entropy for accurate parameter identification. Conversely, minimizing entropy in Peirs shear specimens yielded a uniform pure shear stress state, showcasing the framework's flexibility in tailoring designs for specific experimental goals. Finally, we addressed experimental uncertainties and demonstrated the potential of transfer learning for replacing challenging testing protocols with simpler alternatives, while preserving calibration accuracy.

Hyperspectral image (HSI) fusion addresses the challenge of reconstructing High-Resolution HSIs (HR-HSIs) from High-Resolution Multispectral images (HR-MSIs) and Low-Resolution HSIs (LR-HSIs), a critical task given the high costs and hardware limitations associated with acquiring high-quality HSIs. While existing methods leverage spatial and spectral relationships, they often suffer from limited receptive fields and insufficient feature utilization, leading to suboptimal performance. Furthermore, the scarcity of high-quality HSI data highlights the importance of efficient data utilization to maximize reconstruction quality. To address these issues, we propose HyFusion, a novel Dual-Coupled Network (DCN) framework designed to enhance cross-domain feature extraction and enable effective feature map reusing. The framework first processes HR-MSI and LR-HSI inputs through specialized subnetworks that mutually enhance each other during feature extraction, preserving complementary spatial and spectral details. At its core, HyFusion utilizes an Enhanced Reception Field Block (ERFB), which combines shifting-window attention and dense connections to expand the receptive field, effectively capturing long-range dependencies while minimizing information loss. Extensive experiments demonstrate that HyFusion achieves state-of-the-art performance in HR-MSI/LR-HSI fusion, significantly improving reconstruction quality while maintaining a compact model size and computational efficiency. By integrating enhanced receptive fields and feature map reusing into a coupled network architecture, HyFusion provides a practical and effective solution for HSI fusion in resource-constrained scenarios, setting a new benchmark in hyperspectral imaging. Our code will be publicly available., Comment: Submitted to IGARSS 2025

Research on text simplification has primarily focused on lexical and sentence-level changes. Long document-level simplification (DS) is still relatively unexplored. Large Language Models (LLMs), like ChatGPT, have excelled in many natural language processing tasks. However, their performance on DS tasks is unsatisfactory, as they often treat DS as merely document summarization. For the DS task, the generated long sequences not only must maintain consistency with the original document throughout, but complete moderate simplification operations encompassing discourses, sentences, and word-level simplifications. Human editors employ a hierarchical complexity simplification strategy to simplify documents. This study delves into simulating this strategy through the utilization of a multi-stage collaboration using LLMs. We propose a progressive simplification method (ProgDS) by hierarchically decomposing the task, including the discourse-level, topic-level, and lexical-level simplification. Experimental results demonstrate that ProgDS significantly outperforms existing smaller models or direct prompting with LLMs, advancing the state-of-the-art in the document simplification task.

With the rapid growth of the Internet of Things and Cyber-Physical Systems, widespread sensor deployment has become essential. However, the high costs of building sensor networks limit their scale and coverage, making fine-grained deployment challenging. Inductive Spatio-Temporal Kriging (ISK) addresses this issue by introducing virtual sensors. Based on graph neural networks (GNNs) extracting the relationships between physical and virtual sensors, ISK can infer the measurements of virtual sensors from physical sensors. However, current ISK methods rely on conventional message-passing mechanisms and network architectures, without effectively extracting spatio-temporal features of physical sensors and focusing on representing virtual sensors. Additionally, existing graph construction methods face issues of sparse and noisy connections, destroying ISK performance. To address these issues, we propose DarkFarseer, a novel ISK framework with three key components. First, we propose the Neighbor Hidden Style Enhancement module with a style transfer strategy to enhance the representation of virtual nodes in a temporal-then-spatial manner to better extract the spatial relationships between physical and virtual nodes. Second, we propose Virtual-Component Contrastive Learning, which aims to enrich the node representation by establishing the association between the patterns of virtual nodes and the regional patterns within graph components. Lastly, we design a Similarity-Based Graph Denoising Strategy, which reduces the connectivity strength of noisy connections around virtual nodes and their neighbors based on their temporal information and regional spatial patterns. Extensive experiments demonstrate that DarkFarseer significantly outperforms existing ISK methods., Comment: TKDE (Under Review)

Recently, Low Earth Orbit (LEO) satellite networks (i.e., non-terrestrial network (NTN)), such as Starlink, have been successfully deployed to provide broader coverage than terrestrial networks (TN). Due to limited spectrum resources, TN and NTN may soon share the same spectrum. Therefore, fine-grained spectrum monitoring is crucial for spectrum sharing and interference avoidance. To this end, constructing a 4D radio map (RM) including three spatial dimensions and signal spectra is important. However, this requires the large deployment of sensors, and high-speed analog-to-digital converters for extensive spatial signal collection and wide power spectrum acquisition, respectively. To address these challenges, we propose a deep unsupervised learning framework without ground truths labeling requirement, DeepRM, comprised of neural compressive sensing (CS) and tensor decomposition (TD) algorithms. Firstly, we map the CS process into the optimization of a neural networksassociated loss function, and design a sparsity-performance balance training algorithm to reconstruct a wide power spectrum under limited sub-Nquist samples. Secondly, according to the output of neural CS algorithm, we also utilize neural networks to perform TD, and construct the 3D RM for each frequency, even under very sparse sensor deployment. Extensive evaluations show that DeepRM achieves lower error than its corresponding state-of-the-art baselines, especially with limited samples., Comment: 11 pages, 14 figures

Current recommendation systems recommend goods by considering users' historical behaviors, social relations, ratings, and other multi-modals. Although outdated user information presents the trends of a user's interests, no recommendation system can know the users' real-time thoughts indeed. With the development of brain-computer interfaces, it is time to explore next-generation recommenders that show users' real-time thoughts without delay. Electroencephalography (EEG) is a promising method of collecting brain signals because of its convenience and mobility. Currently, there is only few research on EEG-based recommendations due to the complexity of learning human brain activity. To explore the utility of EEG-based recommendation, we propose a novel neural network model, QUARK, combining Quantum Cognition Theory and Graph Convolutional Networks for accurate item recommendations. Compared with the state-of-the-art recommendation models, the superiority of QUARK is confirmed via extensive experiments.

Micro-video recommendation is attracting global attention and becoming a popular daily service for people of all ages. Recently, Graph Neural Networks-based micro-video recommendation has displayed performance improvement for many kinds of recommendation tasks. However, the existing works fail to fully consider the characteristics of micro-videos, such as the high timeliness of news nature micro-video recommendation and sequential interactions of frequently changed interests. In this paper, a novel Multi-aggregator Time-warping Heterogeneous Graph Neural Network (MTHGNN) is proposed for personalized news nature micro-video recommendation based on sequential sessions, where characteristics of micro-videos are comprehensively studied, users' preference is mined via multi-aggregator, the temporal and dynamic changes of users' preference are captured, and timeliness is considered. Through the comparison with the state-of-the-arts, the experimental results validate the superiority of our MTHGNN model.

Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Like, SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. In practice, permission to access gradient information is not always granted (the gradient ban), such as black-box APIs, hardware limitations, and non-differentiable systems. To bridge this gap, we introduce the first benchmark ZeroFlow to evaluate gradient-free optimization algorithms for overcoming forgetting. This benchmark examines a suite of forward pass methods across multiple methods, forgetting scenarios, and datasets. We find that forward passes alone are enough to overcome forgetting. Our findings reveal new optimization principles that highlight the potential of forward-pass in mitigating forgetting, managing task conflicts, and reducing memory demands, alongside novel enhancements that further mitigate forgetting with just one forward pass. This work provides essential insights and tools for advancing forward pass methods to overcome forgetting.

Capitalizing on the complementary advantages of generative and discriminative models has always been a compelling vision in machine learning, backed by a growing body of research. This work discloses the hidden semantic structure within score-based generative models, unveiling their potential as effective discriminative priors. Inspired by our theoretical findings, we propose DUSA to exploit the structured semantic priors underlying diffusion score to facilitate the test-time adaptation of image classifiers or dense predictors. Notably, DUSA extracts knowledge from a single timestep of denoising diffusion, lifting the curse of Monte Carlo-based likelihood estimation over timesteps. We demonstrate the efficacy of our DUSA in adapting a wide variety of competitive pre-trained discriminative models on diverse test-time scenarios. Additionally, a thorough ablation study is conducted to dissect the pivotal elements in DUSA. Code is publicly available at https://github.com/BIT-DA/DUSA., Comment: Accepted by NeurIPS 2024. Project page: https://kiwixr.github.io/projects/dusa

Coherent manipulation of lattice vibrations using ultrafast light pulses enables access to nonequilibrium 'hidden' phases with designed functionalities in quantum materials. However, expanding the understanding of nonlinear light-phonon interaction mechanisms remains crucial for developing new strategies. Here, we report re-entrant ultrafast phase transitions in SrTiO3 driven by intense terahertz excitation. As the terahertz field increases, the system transitions from the quantum paraelectric (QPE) ground state to an intermediate ferroelectric phase, and then unexpectedly reverts to a QPE state above ~500 kV/cm. The latter hidden QPE phase exhibits distinct lattice dynamics compared to the initial phases, highlighting activated antiferrodistortive phonon modes. Aided by first-principles dynamical calculations, we identify the mechanism for these complex behaviors as a superposition of multiple coherently excited eigenstates of the polar soft mode. Our results reveal a previously uncharted quantum facet of SrTiO3 and open pathways for harnessing high-order excitations to engineer quantum materials in the ultrafast regime., Comment: 18 pages, 4 figures

Community detection, also known as graph partitioning, is a well-known NP-hard combinatorial optimization problem with applications in diverse fields such as complex network theory, transportation, and smart power grids. The problem's solution space grows drastically with the number of vertices and subgroups, making efficient algorithms crucial. In recent years, quantum computing has emerged as a promising approach to tackling NP-hard problems. This study explores the use of a quantum-inspired algorithm, Simulated Bifurcation (SB), for community detection. Modularity is employed as both the objective function and a metric to evaluate the solutions. The community detection problem is formulated as a Quadratic Unconstrained Binary Optimization (QUBO) problem, enabling seamless integration with the SB algorithm. Experimental results demonstrate that SB effectively identifies community structures in benchmark networks such as Zachary's Karate Club and the IEEE 33-bus system. Remarkably, SB achieved the highest modularity, matching the performance of Fujitsu's Digital Annealer, while surpassing results obtained from two quantum machines, D-Wave and IBM. These findings highlight the potential of Simulated Bifurcation as a powerful tool for solving community detection problems., Comment: 7 pages, 4 figures, 2 tables

Open-vocabulary panoptic segmentation remains a challenging problem. One of the biggest difficulties lies in training models to generalize to an unlimited number of classes using limited categorized training data. Recent popular methods involve large-scale vision-language pre-trained foundation models, such as CLIP. In this paper, we propose OMTSeg for open-vocabulary segmentation using another large-scale vision-language pre-trained model called BEiT-3 and leveraging the cross-modal attention between visual and linguistic features in BEiT-3 to achieve better performance. Experiments result demonstrates that OMTSeg performs favorably against state-of-the-art models., Comment: ICIP 2024

The growing field of remote sensing faces a challenge: the ever-increasing size and volume of imagery data are exceeding the storage and transmission capabilities of satellite platforms. Efficient compression of remote sensing imagery is a critical solution to alleviate these burdens on satellites. However, existing compression methods are often too computationally expensive for satellites. With the continued advancement of compressed sensing theory, single-pixel imaging emerges as a powerful tool that brings new possibilities for on-orbit image compression. However, it still suffers from prolonged imaging times and the inability to perform high-resolution imaging, hindering its practical application. This paper advances the study of compressed sensing in remote sensing image compression, proposing Block Modulated Imaging (BMI). By requiring only a single exposure, BMI significantly enhances imaging acquisition speeds. Additionally, BMI obviates the need for digital micromirror devices and surpasses limitations in image resolution. Furthermore, we propose a novel decoding network specifically designed to reconstruct images compressed under the BMI framework. Leveraging the gated 3D convolutions and promoting efficient information flow across stages through a Two-Way Cross-Attention module, our decoding network exhibits demonstrably superior reconstruction performance. Extensive experiments conducted on multiple renowned remote sensing datasets unequivocally demonstrate the efficacy of our proposed method. To further validate its practical applicability, we developed and tested a prototype of the BMI-based camera, which has shown promising potential for on-orbit image compression. The code is available at https://github.com/Johnathan218/BMNet.

AI agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in LLM have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompts. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, discrepancies between developer-implemented logic and the dynamically generated content of LLMs in terms of behavior and expected outcomes can lead to defects, such as tool invocation failures and task execution errors. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, such as service interruptions. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. In this paper, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 6,854 relevant posts from StackOverflow to define 8 types of agent defects. For each type, we provided detailed descriptions with an example. Then, we designed a static analysis tool, named Agentable, to detect the defects. Agentable leverages Code Property Graphs and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, consists of 84 real-world Agents, and AgentTest, which contains 78 Agents specifically designed to include various types of defects. Our results show that Agentable achieved an overall accuracy of 88.79% and a recall rate of 91.03%. Furthermore, our analysis reveals the 889 defects of the AgentSet, highlighting the prevalence of these defects.

Grover's search algorithm has attracted great attention due to its quadratic speedup over classical algorithms in unsorted database search problems. However, Grover's algorithm is inefficient in multi-target search problems, except in the case of 1/4 of the data in the database satisfying the search conditions. Long presented a modified version of Grover's search algorithm by introducing a phase-matching condition that can search for the target state with a zero theoretical failure rate. In this work, we present an optimized exact multi-target search algorithm based on the modified Grover's algorithm by transforming the canonical diffusion operator to a more efficient diffusion operator, which can solve the multi-target search problem with a 100% success rate while requiring fewer gate counts and shallower circuit depth. After that, the optimized multi-target algorithm for four different items, including 2-qubit with 2 targets, 5-qubit with 2 targets, 5-qubit with 4 targets, and 6-qubit with 3 targets, are implemented on MindSpore framework. The experimental results show that, compared with Grover's algorithm and the modified Grover's algorithm, the proposed algorithm can reduce the quantum gate count by at least 21.1% and the depth of the quantum circuit by at least 11.4% and maintain a 100% success probability. Our code are available at https://github.com/mindsporelab/models/tree/master/research/arxiv_papers/GROVER-OP. Thanks for the support provided by MindSpore Community.

Recently, Visual Foundation Models (VFMs) have shown a remarkable generalization performance in 3D perception tasks. However, their effectiveness in large-scale outdoor datasets remains constrained by the scarcity of accurate supervision signals, the extensive noise caused by variable outdoor conditions, and the abundance of unknown objects. In this work, we propose a novel label-free learning method, Adaptive Label Correction (AdaCo), for 3D semantic segmentation. AdaCo first introduces the Cross-modal Label Generation Module (CLGM), providing cross-modal supervision with the formidable interpretive capabilities of the VFMs. Subsequently, AdaCo incorporates the Adaptive Noise Corrector (ANC), updating and adjusting the noisy samples within this supervision iteratively during training. Moreover, we develop an Adaptive Robust Loss (ARL) function to modulate each sample's sensitivity to noisy supervision, preventing potential underfitting issues associated with robust loss. Our proposed AdaCo can effectively mitigate the performance limitations of label-free learning networks in 3D semantic segmentation tasks. Extensive experiments on two outdoor benchmark datasets highlight the superior performance of our method., Comment: 2025 AAAI

A natural extension of the non-Hermitian qubit is to place it in a single-mode cavity. This setup corresponds to the quantum Rabi model (QRM) with a purely imaginary bias on the qubit, exhibiting the parity-time ($\mathcal{P}\mathcal{T}$) symmetry. In this work, we first solve the $\mathcal{P} \mathcal{T}$ symmetric QRM using the Bogoliubov operator approach. We derive the transcendental function responsible for the exact solution, which can also be used to precisely identify exceptional points. The adiabatic approximation previously used can be easily formulated within this approach by considering transitions between the same manifolds in the space of Bogoliubov operators. By further considering transitions between the nearest-neighboring manifolds, we can analytically obtain more accurate eigen-solutions. Moreover, these simple corrections can capture the main features of the dynamics, where the adiabatic approximation fails. Furthermore, the rich characteristics of the vacuum Rabi splitting in the emission spectrum is predicted. The width of the peaks increases with the coupling strength and the imaginary biases, reflecting the nature of open quantum systems. Additionally, we identify a quantum criticality-enhanced effect by calculating the quantum Fisher information. Near the exceptional points, the quantum Fisher information in the $\mathcal{P} \mathcal{T}$ symmetric QRM is significantly higher than that of the non-Hermitian qubit component. This may open a new avenue for enhancing quantum sensitivity in non-Hermitian systems by incorporating coupling with an additional degree of freedom, enabling more precise parameter estimation., Comment: 11 pages, 9 figures

We combine experiments and theoretical derivations to study the evolution of a stretched soap bubble and compare it with an open film to highlight the effect of volume conservation. We identify a critical length for both surfaces, beyond which a bottleneck develops in the middle and begins to shrink irreversibly, ultimately pinching off into multiple compartments. Before leaving the equilibrium regime, surface energy minimization governs the shape, which can be addressed theoretically via the variational method. In contrast to open films, soap bubble volume conservation introduces a Lagrange multiplier, analogous to a pressure difference, mediating long-range shape evolution. By examining how boundary constraints influence deformation, we contrast the bubble's convex-to-concave transition with the behavior of soap films under similar conditions. Our analysis of equilibrium and breakup regimes reveals critical differences between bubble and film stability profiles, shedding light on universal behaviors in non-equilibrium fluid mechanics, with implications for biological and material sciences., Comment: 9 pages

Fuel efficiency is a crucial aspect of long-distance cargo transportation by oil-powered trucks that economize on costs and decrease carbon emissions. Current predictive control methods depend on an accurate model of vehicle dynamics and engine, including weight, drag coefficient, and the Brake-specific Fuel Consumption (BSFC) map of the engine. We propose a pure data-driven method, Neural Predictive Control (NPC), which does not use any physical model for the vehicle. After training with over 20,000 km of historical data, the novel proposed NVFormer implicitly models the relationship between vehicle dynamics, road slope, fuel consumption, and control commands using the attention mechanism. Based on the online sampled primitives from the past of the current freight trip and anchor-based future data synthesis, the NVFormer can infer optimal control command for reasonable fuel consumption. The physical model-free NPC outperforms the base PCC method with 2.41% and 3.45% more significant fuel saving in simulation and open-road highway testing, respectively., Comment: 7 pages, 6 figures, for associated mpeg file, see https://www.youtube.com/watch?v=hqgpj7LhiL4

Advances in data assimilation (DA) methods have greatly improved the accuracy of Earth system predictions. To fuse multi-source data and reconstruct the nonlinear evolution missing from observations, geoscientists are developing future-oriented DA methods. In this paper, we redesign a purely data-driven latent space DA framework (DeepDA) that employs a generative artificial intelligence model to capture the nonlinear evolution in sea surface temperature. Under variational constraints, DeepDA embedded with nonlinear features can effectively fuse heterogeneous data. The results show that DeepDA remains highly stable in capturing and generating nonlinear evolutions even when a large amount of observational information is missing. It can be found that when only 10% of the observation information is available, the error increase of DeepDA does not exceed 40%. Furthermore, DeepDA has been shown to be robust in the fusion of real observations and ensemble simulations. In particular, this paper provides a mechanism analysis of the nonlinear evolution generated by DeepDA from the perspective of physical patterns, which reveals the inherent explainability of our DL model in capturing multi-scale ocean signals., Comment: 31 pages, 14 figures

In the relentless pursuit of quantum computational advantage, we present a significant advancement with the development of Zuchongzhi 3.0. This superconducting quantum computer prototype, comprising 105 qubits, achieves high operational fidelities, with single-qubit gates, two-qubit gates, and readout fidelity at 99.90%, 99.62% and 99.18%, respectively. Our experiments with an 83-qubit, 32-cycle random circuit sampling on Zuchongzhi 3.0 highlight its superior performance, achieving one million samples in just a few hundred seconds. This task is estimated to be infeasible on the most powerful classical supercomputers, Frontier, which would require approximately $6.4\times 10^9$ years to replicate the task. This leap in processing power places the classical simulation cost six orders of magnitude beyond Google's SYC-67 and SYC-70 experiments [Nature 634, 328(2024)], firmly establishing a new benchmark in quantum computational advantage. Our work not only advances the frontiers of quantum computing but also lays the groundwork for a new era where quantum processors play an essential role in tackling sophisticated real-world challenges.

In this paper, we solve the complete Salpeter equation and use the obtained relativistic wave function to calculate the strong and radiative electromagnetic decays of ${D^*}$ meson. The focus of this study is on the relativistic corrections. In our method, the wave function of $D$ meson is not pure $S$-wave, but includes both nonrelativistic $S$-wave and relativistic $P$-wave, while the wave function of $D^*$ meson includes nonrelativistic $S$-wave, as well as relativistic $P$-wave and $D$-wave. So in this case, the decay ${D^{*}\rightarrow{D}\gamma}$ is not a nonrelativistic $M1$ transition, but rather an $M1+E2+M3+E4$ decay in our method. We find that in strong decay $D^{*}\rightarrow{D}{\pi}$, the nonrelativistic contribution is dominant, while in electromagnetic decay ${D^{*}\rightarrow{D}\gamma}$, the relativistic correction is dominant., Comment: 12 pages, 2 figures

Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time., Comment: Github Repo: https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-OmniLive

Audio-driven talking head generation necessitates seamless integration of audio and visual data amidst the challenges posed by diverse input portraits and intricate correlations between audio and facial motions. In response, we propose a robust framework GoHD designed to produce highly realistic, expressive, and controllable portrait videos from any reference identity with any motion. GoHD innovates with three key modules: Firstly, an animation module utilizing latent navigation is introduced to improve the generalization ability across unseen input styles. This module achieves high disentanglement of motion and identity, and it also incorporates gaze orientation to rectify unnatural eye movements that were previously overlooked. Secondly, a conformer-structured conditional diffusion model is designed to guarantee head poses that are aware of prosody. Thirdly, to estimate lip-synchronized and realistic expressions from the input audio within limited training data, a two-stage training strategy is devised to decouple frequent and frame-wise lip motion distillation from the generation of other more temporally dependent but less audio-related motions, e.g., blinks and frowns. Extensive experiments validate GoHD's advanced generalization capabilities, demonstrating its effectiveness in generating realistic talking face results on arbitrary subjects., Comment: Accepted by AAAI 2025

Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student's learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones.

Tensor network machine learning models have shown remarkable versatility in tackling complex data-driven tasks, ranging from quantum many-body problems to classical pattern recognitions. Despite their promising performance, a comprehensive understanding of the underlying assumptions and limitations of these models is still lacking. In this work, we focus on the rigorous formulation of their no-free-lunch theorem -- essential yet notoriously challenging to formalize for specific tensor network machine learning models. In particular, we rigorously analyze the generalization risks of learning target output functions from input data encoded in tensor network states. We first prove a no-free-lunch theorem for machine learning models based on matrix product states, i.e., the one-dimensional tensor network states. Furthermore, we circumvent the challenging issue of calculating the partition function for two-dimensional Ising model, and prove the no-free-lunch theorem for the case of two-dimensional projected entangled-pair state, by introducing the combinatorial method associated to the "puzzle of polyominoes". Our findings reveal the intrinsic limitations of tensor network-based learning models in a rigorous fashion, and open up an avenue for future analytical exploration of both the strengths and limitations of quantum-inspired machine learning frameworks., Comment: 7+23 pages, comments welcome

Motivated by the recent observations of various exotic quantum states in the equilateral triangular-lattice phosphates Na$_2$BaCo(PO$_4$)$_2$ with $J\rm_{eff}$ = 1/2 and Na$_2$BaNi(PO$_4$)$_2$ with $S$ = 1, the magnetic properties of spin-5/2 antiferromagnet Na$_2$BaMn(PO$_4$)$_2$, their classical counterpart, are comprehensively investigated experimentally. DC magnetization and specific heat measurements on polycrystalline samples indicate two successive magnetic transitions at $T\rm_{N1}$ $\approx$ 1.13 K and $T\rm_{N2}$ $\approx$ 1.28 K, respectively. Zero-field neutron powder diffraction measurement at 67 mK reveals a Y-like spin configuration as its ground-state magnetic structure, with both the $ab$-plane and $c$-axis components of the Mn$^{2+}$ moments long-range ordered. The incommensurate magnetic propagation vector $k$ shows a dramatic change for the intermediate phase between $T\rm_{N1}$ and $T\rm_{N2}$, in which the spin state is speculated to change into a collinear structure with only the $c$-axis moments ordered, as stabilized by thermal fluctuations. The successive magnetic transitions observed in Na$_2$BaMn(PO$_4$)$_2$ are in line with the expectation for a triangle-lattice antiferromagnet with an easy-axis magnetic anisotropy., Comment: 8 Pages, 6 figures

Large language models (LLMs) are effective at capturing complex, valuable conceptual representations from textual data for a wide range of real-world applications. However, in fields like Intelligent Fault Diagnosis (IFD), incorporating additional sensor data-such as vibration signals, temperature readings, and operational metrics-is essential but it is challenging to capture such sensor data information within traditional text corpora. This study introduces a novel IFD approach by effectively adapting LLMs to numerical data inputs for identifying various machine faults from time-series sensor data. We propose FD-LLM, an LLM framework specifically designed for fault diagnosis by formulating the training of the LLM as a multi-class classification problem. We explore two methods for encoding vibration signals: the first method uses a string-based tokenization technique to encode vibration signals into text representations, while the second extracts statistical features from both the time and frequency domains as statistical summaries of each signal. We assess the fault diagnosis capabilities of four open-sourced LLMs based on the FD-LLM framework, and evaluate the models' adaptability and generalizability under various operational conditions and machine components, namely for traditional fault diagnosis, cross-operational conditions, and cross-machine component settings. Our results show that LLMs such as Llama3 and Llama3-instruct demonstrate strong fault detection capabilities and significant adaptability across different operational conditions, outperforming state-of-the-art deep learning (DL) approaches in many cases., Comment: 20 pages, 2 figures, 16 tables, including the tables in the appendix

The interplay between topology and magnetism often triggers the exotic quantum phases. Here, we report an accessible scheme to engineer the robust $\mathbb{Z}_{2}$ topology by intrinsic magnetism, originating from the zigzag segment connecting two armchair segments with different width, in one-dimensional graphene nanoribbon heterojunctions. Our first-principle and model simulations reveal that the emergent spin polarization substantially modifies the dimerization between junction states, forming the special SSH mechanism depending on the magnetic configurations. Interestingly, the topological phase in magnetic state is only determined by the width of the narrow armchair segment, in sharp contrast with that in the normal state. In addition, the emergent magnetism increases the bulk energy band gap by an order of magnitude than that in the nonmagnetic state. We also discuss the $\mathbb{Z}$ topology of the junction states and the termination-dependent of topological end states. Our results bring new way to tune the topology in graphene nanoribbon heterostructure, providing a new platform for future one-dimensional topological devices and molecular-scale spintronics., Comment: 5 pages, 5 figures

Common imaging techniques for detecting structural defects typically require sampling at more than twice the spatial frequency to achieve a target resolution. This study introduces a novel framework for imaging structural defects using significantly fewer samples. In this framework, defects are modeled as regions where physical properties shift from their nominal values to resemble those of air, and a linear approximation is formulated to relate these binary shifts in physical properties with corresponding changes in measurements. Recovering a binary vector from linear measurements is generally an NP-hard problem. To address this challenge, this study proposes two algorithmic approaches. The first approach relaxes the binary constraint, using convex optimization to find a solution. The second approach incorporates a binary-inducing prior and employs approximate Bayesian inference to estimate the posterior probability of the binary vector given the measurements. Both algorithmic approaches demonstrate better performance compared to existing compressive sampling methods for binary vector recovery. The framework's effectiveness is illustrated through examples of eddy current sensing to image defects in metal structures.

Face swapping aims to generate results that combine the identity from the source with attributes from the target. Existing methods primarily focus on image-based face swapping. When processing videos, each frame is handled independently, making it difficult to ensure temporal stability. From a model perspective, face swapping is gradually shifting from generative adversarial networks (GANs) to diffusion models (DMs), as DMs have been shown to possess stronger generative capabilities. Current diffusion-based approaches often employ inpainting techniques, which struggle to preserve fine-grained attributes like lighting and makeup. To address these challenges, we propose a high fidelity video face swapping (HiFiVFS) framework, which leverages the strong generative capability and temporal prior of Stable Video Diffusion (SVD). We build a fine-grained attribute module to extract identity-disentangled and fine-grained attribute features through identity desensitization and adversarial learning. Additionally, We introduce detailed identity injection to further enhance identity similarity. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) in video face swapping, both qualitatively and quantitatively.

Relic landslide, formed over a long period, possess the potential for reactivation, making them a hazardous geological phenomenon. While reliable relic landslide detection benefits the effective monitoring and prevention of landslide disaster, semantic segmentation using high-resolution remote sensing images for relic landslides faces many challenges, including the object visual blur problem, due to the changes of appearance caused by prolonged natural evolution and human activities, and the small-sized dataset problem, due to difficulty in recognizing and labelling the samples. To address these challenges, a semantic segmentation model, termed mask-recovering and interactive-feature-enhancing (MRIFE), is proposed for more efficient feature extraction and separation. Specifically, a contrastive learning and mask reconstruction method with locally significant feature enhancement is proposed to improve the ability to distinguish between the target and background and represent landslide semantic features. Meanwhile, a dual-branch interactive feature enhancement architecture is used to enrich the extracted features and address the issue of visual ambiguity. Self-distillation learning is introduced to leverage the feature diversity both within and between samples for contrastive learning, improving sample utilization, accelerating model convergence, and effectively addressing the problem of the small-sized dataset. The proposed MRIFE is evaluated on a real relic landslide dataset, and experimental results show that it greatly improves the performance of relic landslide detection. For the semantic segmentation task, compared to the baseline, the precision increases from 0.4226 to 0.5347, the mean intersection over union (IoU) increases from 0.6405 to 0.6680, the landslide IoU increases from 0.3381 to 0.3934, and the F1-score increases from 0.5054 to 0.5646.

Accurate and consistent evaluation is crucial for decision-making across numerous fields, yet it remains a challenging task due to inherent subjectivity, variability, and scale. Large Language Models (LLMs) have achieved remarkable success across diverse domains, leading to the emergence of "LLM-as-a-Judge," where LLMs are employed as evaluators for complex tasks. With their ability to process diverse data types and provide scalable, cost-effective, and consistent assessments, LLMs present a compelling alternative to traditional expert-driven evaluations. However, ensuring the reliability of LLM-as-a-Judge systems remains a significant challenge that requires careful design and standardization. This paper provides a comprehensive survey of LLM-as-a-Judge, addressing the core question: How can reliable LLM-as-a-Judge systems be built? We explore strategies to enhance reliability, including improving consistency, mitigating biases, and adapting to diverse assessment scenarios. Additionally, we propose methodologies for evaluating the reliability of LLM-as-a-Judge systems, supported by a novel benchmark designed for this purpose. To advance the development and real-world deployment of LLM-as-a-Judge systems, we also discussed practical applications, challenges, and future directions. This survey serves as a foundational reference for researchers and practitioners in this rapidly evolving field., Comment: Corrected typos & more discussion on reasoning models 33 pages, 9 figures. arXiv admin note: text overlap with arXiv:2310.05470 by other authors

Quantification of network capacity and throughput is crucial for performance evaluation of integrated space and terrestrial network (ISTN).However, existing studies mainly consider the maximum throughput as the network capacity, but such a definition would make it unreasonable that the value of the network capacity would change with different employed routing algorithms and congestion control policy, instead of being a constant quantity. In this paper, we argue that the capacity of an ISTN is solely dependent on the characteristics of the network infrastructure,and the throughput of an ISTN is the aggregate traffic transported by the network under a given traffic scenario. Then, we present a quantitative approach to assessing network capacity in relation to an unreliable ISL model (cap-uISL), and a Constrained Path Expansion throughput calculation method (THP-CPE) based on a set of known traffic paths. This method allows us to obtain the current throughput value of the network based on any given traffic paths and load demand matrix. As the traffic load increases, the throughput approaches its maximum value, which is notably smaller than the network's capacity. We experimentally determine the network capacity of CAP-uISL under various link parameters and compare our throughput quantization method, THP-CPE, with other state-of-the-art methods under four emerging ISTNs. We find that, compared with the THP-CPE, existing throughput calculation methods tend to be overestimated, while our proposed throughput calculation method maintains reasonable intervals in terms of path utilization ($<1$) under all load cases.

Surgical practice involves complex visual interpretation, procedural skills, and advanced medical knowledge, making surgical vision-language pretraining (VLP) particularly challenging due to this complexity and the limited availability of annotated data. To address the gap, we propose OphCLIP, a hierarchical retrieval-augmented vision-language pretraining framework specifically designed for ophthalmic surgical workflow understanding. OphCLIP leverages the OphVL dataset we constructed, a large-scale and comprehensive collection of over 375K hierarchically structured video-text pairs with tens of thousands of different combinations of attributes (surgeries, phases/operations/actions, instruments, medications, as well as more advanced aspects like the causes of eye diseases, surgical objectives, and postoperative recovery recommendations, etc). These hierarchical video-text correspondences enable OphCLIP to learn both fine-grained and long-term visual representations by aligning short video clips with detailed narrative descriptions and full videos with structured titles, capturing intricate surgical details and high-level procedural insights, respectively. Our OphCLIP also designs a retrieval-augmented pretraining framework to leverage the underexplored large-scale silent surgical procedure videos, automatically retrieving semantically relevant content to enhance the representation learning of narrative videos. Evaluation across 11 datasets for phase recognition and multi-instrument identification shows OphCLIP's robust generalization and superior performance.

Despite significant advancements in general artificial intelligence, such as GPT-4, their effectiveness in the medical domain (general medical AI, GMAI) remains constrained due to the absence of specialized medical knowledge. To address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal medical dataset created by converting hundreds of specialized medical datasets into meticulously constructed image-text pairs. This dataset features comprehensive task coverage, diverse modalities, and high-quality image-text data. Building upon this multimodal dataset, we propose GMAI-VL, a general medical vision-language model with a progressively three-stage training strategy. This approach significantly enhances the model's ability by integrating visual and textual information, thereby improving its ability to process multimodal data and support accurate diagnosis and clinical decision-making. Experimental evaluations demonstrate that GMAI-VL achieves state-of-the-art results across a wide range of multimodal medical tasks, such as visual question answering and medical image diagnosis. Our contributions include the development of the GMAI-VL-5.5M dataset, the introduction of the GMAI-VL model, and the establishment of new benchmarks in multiple medical domains. Code and dataset will be released at https://github.com/uni-medical/GMAI-VL.

Confidence calibration in LLMs, i.e., aligning their self-assessed confidence with the actual accuracy of their responses, enabling them to self-evaluate the correctness of their outputs. However, current calibration methods for LLMs typically estimate two scalars to represent overall response confidence and correctness, which is inadequate for long-form generation where the response includes multiple atomic facts and may be partially confident and correct. These methods also overlook the relevance of each fact to the query. To address these challenges, we propose a Fact-Level Calibration framework that operates at a finer granularity, calibrating confidence to relevance-weighted correctness at the fact level. Furthermore, comprehensive analysis under the framework inspired the development of Confidence-Guided Fact-level Self-Correction ($\textbf{ConFix}$), which uses high-confidence facts within a response as additional knowledge to improve low-confidence ones. Extensive experiments across four datasets and six models demonstrate that ConFix effectively mitigates hallucinations without requiring external knowledge sources such as retrieval systems., Comment: Code is available at https://github.com/yuanyige/fact-calibration

We show the existence of infinitely many Bethe expansions for general four dimensional superconformal theories. We then propose an analytic method to systematically obtain all the Bethe solutions, both isolated and continuous, for general superconformal theories. In particular, we show that the contribution from the continuous manifold of solutions to the index comes from isolated points on this manifold. We check our proposals on $\mathcal{N}=4$ SYM. For $U(3)$ or $SU(3)$, which are the first examples with continuous solutions, we demonstrate the non-trivial cancellation between the tachyon contributions from the previously known isolated solutions and those from the continuous solutions., Comment: 56pp

To tackle the challenges of large language model performance in natural language to SQL tasks, we introduce XiYan-SQL, an innovative framework that employs a multi-generator ensemble strategy to improve candidate generation. We introduce M-Schema, a semi-structured schema representation method designed to enhance the understanding of database structures. To enhance the quality and diversity of generated candidate SQL queries, XiYan-SQL integrates the significant potential of in-context learning (ICL) with the precise control of supervised fine-tuning. On one hand, we propose a series of training strategies to fine-tune models to generate high-quality candidates with diverse preferences. On the other hand, we implement the ICL approach with an example selection method based on named entity recognition to prevent overemphasis on entities. The refiner optimizes each candidate by correcting logical or syntactical errors. To address the challenge of identifying the best candidate, we fine-tune a selection model to distinguish nuances of candidate SQL queries. The experimental results on multiple dialect datasets demonstrate the robustness of XiYan-SQL in addressing challenges across different scenarios. Overall, our proposed XiYan-SQL achieves the state-of-the-art execution accuracy of 75.63% on Bird benchmark, 89.65% on the Spider test set, 69.86% on SQL-Eval, 41.20% on NL2GQL. The proposed framework not only enhances the quality and diversity of SQL queries but also outperforms previous methods.

Total hip arthroplasty (THA) relies on accurate landmark detection from radiographic images, but unstructured data caused by irregular patient postures or occluded anatomical markers pose significant challenges for existing methods. To address this, we propose UNSCT-HRNet (Unstructured CT - High-Resolution Net), a deep learning-based framework that integrates a Spatial Relationship Fusion (SRF) module and an Uncertainty Estimation (UE) module. The SRF module, utilizing coordinate convolution and polarized attention, enhances the model's ability to capture complex spatial relationships. Meanwhile, the UE module which based on entropy ensures predictions are anatomically relevant. For unstructured data, the proposed method can predict landmarks without relying on the fixed number of points, which shows higher accuracy and better robustness comparing with the existing methods. Our UNSCT-HRNet demonstrates over a 60% improvement across multiple metrics in unstructured data. The experimental results also reveal that our approach maintains good performance on the structured dataset. Overall, the proposed UNSCT-HRNet has the potential to be used as a new reliable, automated solution for THA surgical planning and postoperative monitoring.

In this work, we study the resonant tunneling (RT) of electrons and H atoms in double-barrier (DB) systems. Our numerical calculations directly verify the correspondence between the resonant tunneling energies and the energy levels of quasi-bound states (QBS) within the double barriers. Based on this, in-depth analyses are carried out on the modulation of QBS energy levels and numbers which show step variation with the inter-barrier spacing. The mathematical criterion for the existence of QBS is derived, and the impacts of the barrier width and barrier height on QBS levels are investigated. Taking the rectangular double-barrier as an example, we have studied the manipulation of electronic structures and optical properties of the inter-barrier region (quasi-potential well) by tuning the inter-barrier spacing (width of quasi-potential well). Atom-like optical absorption features are found in the range of infrared to visible spectrum, which can be continuously tuned by the variation of quasi-potential well width. The potential application of double-barrier nanostructures in ultrahigh-precision detection of electromagnetic radiations is demonstrated.

Soft- and hard-constrained Physics Informed Neural Networks (PINNs) have achieved great success in solving partial differential equations (PDEs). However, these methods still face great challenges when solving the Navier-Stokes equations (NSEs) with complex boundary conditions. To address these challenges, this paper introduces a novel complementary scheme combining soft and hard constraint PINN methods. The soft-constrained part is thus formulated to obtain the preliminary results with a lighter training burden, upon which refined results are then achieved using a more sophisticated hard-constrained mechanism with a primary network and a distance metric network. Specifically, the soft-constrained part focuses on boundary points, while the primary network emphasizes inner domain points, primarily through PDE loss. Additionally, the novel distance metric network is proposed to predict the power function of the distance from a point to the boundaries, which serves as the weighting factor for the first two components. This approach ensures accurate predictions for both boundary and inner domain areas. The effectiveness of the proposed method on the NSEs problem with complex boundary conditions is demonstrated by solving a 2D cylinder wake problem and a 2D blocked cavity flow with a segmented inlet problem, achieving significantly higher accuracy compared to traditional soft- and hard-constrained PINN approaches. Given PINN's inherent advantages in solving the inverse and the large-scale problems, which are challenging for traditional computational fluid dynamics (CFD) methods, this approach holds promise for the inverse design of required flow fields by specifically-designed boundary conditions and the reconstruction of large-scale flow fields by adding a limited number of training input points. The code for our approach will be made publicly available.

Non-Hermitian topology provides a research frontier for exploring topological phenomena, revealing novel topological effects and driving the development of emergent materials and platforms. Here, we explore the non-Hermitian Chern insulator phases and the hybrid skin-topological effects in checkerboard lattices with synthetic gauge fluxes. Such lattices can be realized in integrated silicon photonic nanocircuits and microresonators as well as in arrays of evanescently coupled helical optical waveguides. With a simple and tunable design, the system is found to support non-Hermitian hybrid skin topological effects, exhibiting corner skin effects when the lattice symmetry either $C_4$ or $C_2$. An unconventional physical mechanism is revealed as the origin of such a transition which is connected to the corner-induced scattering between the multiple chiral edge channels. These properties are enabled by the large Chern number and the rich non-Hermitian topological edge states in our system, revealing the diverse non-Hermitian topological bulk-boundary correspondence. Our design offers excellent controllability and experimental feasibility, making it appealing for studying non-Hermitian topological phenomena.

We extend the finite element interpolated neural network (FEINN) framework from partial differential equations (PDEs) with weak solutions in $H^1$ to PDEs with weak solutions in $H(\textbf{curl})$ or $H(\textbf{div})$. To this end, we consider interpolation trial spaces that satisfy the de Rham Hilbert subcomplex, providing stable and structure-preserving neural network discretisations for a wide variety of PDEs. This approach, coined compatible FEINNs, has been used to accurately approximate the $H(\textbf{curl})$ inner product. We numerically observe that the trained network outperforms finite element solutions by several orders of magnitude for smooth analytical solutions. Furthermore, to showcase the versatility of the method, we demonstrate that compatible FEINNs achieve high accuracy in solving surface PDEs such as the Darcy equation on a sphere. Additionally, the framework can integrate adaptive mesh refinements to effectively solve problems with localised features. We use an adaptive training strategy to train the network on a sequence of progressively adapted meshes. Finally, we compare compatible FEINNs with the adjoint neural network method for solving inverse problems. We consider a one-loop algorithm that trains the neural networks for unknowns and missing parameters using a loss function that includes PDE residual and data misfit terms. The algorithm is applied to identify space-varying physical parameters for the $H(\textbf{curl})$ model problem from partial or noisy observations. We find that compatible FEINNs achieve accuracy and robustness comparable to, if not exceeding, the adjoint method in these scenarios.

In recent years, neural networks have increasingly been employed to identify critical points of phase transitions. For the tricritical directed percolation model, its steady-state configurations encompass both first-order and second-order phase transitions. Due to the presence of crossover effects, identifying the critical points of phase transitions becomes challenging. This study utilizes Monte Carlo simulations to obtain steady-state configurations under different probabilities $p$ and $q$, and by calculating the increments in average particle density, we observe first-order transitions, second-order transitions, and regions where both types of transitions interact.These Monte Carlo-generated steady-state configurations are used as input to construct and train a convolutional neural network, from which we determine the critical points $p_{c}$ for different probabilities $q$. Furthermore, by learning the steady-state configurations associated with the superheated point $p=p_u$, we locate the tricritical point at $q_{t}=0.893$. Simultaneously, we employed a three-output CNN model to obtain the phase transition boundaries and the range of the crossover regions. Our method offers a neural network-based approach to capture critical points and distinguish phase transition boundaries, providing a novel solution to this problem.