Search

Existing Image Quality Assessment (IQA) methods achieve remarkable success in analyzing quality for overall image, but few works explore quality analysis for Regions of Interest (ROIs). The quality analysis of ROIs can provide fine-grained guidance for image quality improvement and is crucial for scenarios focusing on region-level quality. This paper proposes a novel network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a Large vision-Language model. SEAGULL incorporates a vision-language model (VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a meticulously designed Mask-based Feature Extractor (MFE) to extract global and local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs. Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises about 100w synthetic distortion images with 33 million ROIs for pre-training to improve the model's ability of regional quality perception, and SEAGULL-3k contains about 3k authentic distortion ROIs to enhance the model's ability to perceive real world distortions. After pre-training on SEAGULL-100w and fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained ROI quality assessment. Code and datasets are publicly available at the https://github.com/chencn2020/Seagull.

Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing system that enables swift actualization of creative ideas. Our system features a streamlined yet functionally robust interface, allowing for the articulation of editing operations (e.g., inserting elements, erasing objects, altering color) with minimal input. These interactions are monitored by a multimodal large language model (MLLM) to anticipate editing intentions in real time, bypassing the need for explicit prompt entry. Finally, we apply a powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process editing requests with precise control. Experimental results demonstrate the effectiveness of MagicQuill in achieving high-quality image edits. Please visit https://magic-quill.github.io to try out our system., Comment: Code and demo available at https://magic-quill.github.io

The MUltiplexed Survey Telescope (MUST) is a 6.5-meter telescope under development. Dedicated to highly-multiplexed, wide-field spectroscopic surveys, MUST observes over 20,000 targets simultaneously using 6.2-mm pitch positioning robots within a ~5 deg2 field of view. MUST aims to carry out the first Stage-V spectroscopic survey in the 2030s to map the 3D Universe with over 100 million galaxies and quasars, spanning from the nearby Universe to redshift z~5.5, corresponding to around 1 billion years after the Big Bang. To cover this extensive redshift range, we present an initial conceptual target selection algorithm for different types of galaxies, from local bright galaxies, luminous red galaxies, and emission line galaxies to high-redshift (2 < z < 5.5) Lyman-break galaxies. Using Fisher forecasts, we demonstrate that MUST can address fundamental questions in cosmology, including the nature of dark energy, test of gravity theories, and investigations into primordial physics. This is the first paper in the series of science white papers for MUST, with subsequent developments focusing on additional scientific cases such as galaxy and quasar evolution, Milky Way physics, and dynamic phenomena in the time-domain Universe., Comment: To be submitted to SCPMA

Retrieval-Augmented Generation (RAG) has been shown to improve knowledge capabilities and alleviate the hallucination problem of LLMs. The Web is a major source of external knowledge used in RAG systems, and many commercial systems such as ChatGPT and Perplexity have used Web search engines as their major retrieval systems. Typically, such RAG systems retrieve search results, download HTML sources of the results, and then extract plain texts from the HTML sources. Plain text documents or chunks are fed into the LLMs to augment the generation. However, much of the structural and semantic information inherent in HTML, such as headings and table structures, is lost during this plain-text-based RAG process. To alleviate this problem, we propose HtmlRAG, which uses HTML instead of plain text as the format of retrieved knowledge in RAG. We believe HTML is better than plain text in modeling knowledge in external documents, and most LLMs possess robust capacities to understand HTML. However, utilizing HTML presents new challenges. HTML contains additional content such as tags, JavaScript, and CSS specifications, which bring extra input tokens and noise to the RAG system. To address this issue, we propose HTML cleaning, compression, and pruning strategies, to shorten the HTML while minimizing the loss of information. Specifically, we design a two-step block-tree-based pruning method that prunes useless HTML blocks and keeps only the relevant part of the HTML. Experiments on six QA datasets confirm the superiority of using HTML in RAG systems.

We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such a design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases (e.g., objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, the model, and the interface will be released to facilitate further research., Comment: Project page: https://aim-uofa.github.io/Framer/

Full-duplex spoken dialogue systems significantly advance over traditional turn-based dialogue systems, as they allow simultaneous bidirectional communication, closely mirroring human-human interactions. However, achieving low latency and natural interactions in full-duplex dialogue systems remains a significant challenge, especially considering human conversation dynamics such as interruptions, backchannels, and overlapping speech. In this paper, we introduce a novel End-to-End GPT-based model OmniFlatten for full-duplex conversation, capable of effectively modeling the complex behaviors inherent to natural conversations with low latency. To achieve full-duplex communication capabilities, we propose a multi-stage post-training scheme that progressively adapts a text-based large language model (LLM) backbone into a speech-text dialogue LLM, capable of generating text and speech in real time, without modifying the architecture of the backbone LLM. The training process comprises three stages: modality alignment, half-duplex dialogue learning, and full-duplex dialogue learning. Throughout all training stages, we standardize the data using a flattening operation, which allows us to unify the training methods and the model architecture across different modalities and tasks. Our approach offers a straightforward modeling technique and a promising research direction for developing efficient and natural end-to-end full-duplex spoken dialogue systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site (https://omniflatten.github.io/)., Comment: Work in progress

In light of recent experimental data indicating a substantial thermal Hall effect in square lattice antiferromagnetic Mott insulators, we investigate whether a simple Mott insulator can sustain a finite thermal Hall effect. We verify that the answer is "no" if one performs calculations within a spin-only low-energy effective spin model with non-interacting magnons. However, by performing determinant quantum Monte Carlo simulations, we show the single-band $t$-$t'$-$U$ Hubbard model coupled to an orbital magnetic field does support a finite thermal Hall effect when $t' \neq 0$ and $B \neq 0$ in the Mott insulating phase. We argue that the (carrier agnostic) necessary conditions for observing a finite thermal Hall effect are time-reversal and particle-hole symmetry breaking. By considering magnon-magnon scattering using a semi-classical Boltzmann analysis, we illustrate a physical mechanism by which finite transverse thermal conductivity may arise, consistent with our symmetry argument and numerical results. Our results contradict the conventional wisdom that square and triangular lattices with SU(2) symmetry do not support a finite thermal Hall effect and call for a critical re-examination of thermal Hall effect data in insulating magnets, as the magnon contribution should not be excluded a priori., Comment: 6+23 pages, 3+11 figures

In this article, we propose new network architectures that integrate multi-functional reconfigurable intelligent surfaces (MF-RISs) into 6G networks to enhance both communication and sensing capabilities. Firstly, we elaborate how to leverage MF-RISs for improving communication performance in different communication modes including unicast, mulitcast, and broadcast and for different multi-access schemes. Next, we emphasize synergistic benefits of integrating MF-RISs with wireless sensing, enabling more accurate and efficient target detection in 6G networks. Furthermore, we present two schemes that utilize MF-RISs to enhance the performance of integrated sensing and communication (ISAC). We also study multi-objective optimization to achieve the optimal trade-off between communication and sensing performance. Finally, we present numerical results to show the performance improvements offered by MF-RISs compared to conventional RISs in ISAC. We also outline key research directions for MF-RIS under the ambition of 6G., Comment: 8 pages, 5 figures, submitted to IEEE

Employing aberration-corrected scanning transmission electron microscopy (STEM), we meticulously investigated the intrinsic chemical heterogeneity of Fe1+yTe, Fe1+yTe0.8Se0.2, and Fe1+yTe0.5Se0.5. Comprehensive analysis reveals the presence of interstitial iron atoms (Feint) across all samples, pre-dominantly occupying the 2c site of the P4/nmm space group. Moreover, a superstructure phase characterized by a wave vector q = 2/5a + 1/2c, originating from the ordering of Feint, is distinctly observable in the parent compound Fe1+yTe. In this scenario, the Feint atoms interact with adjacent Fe atoms, forming iron polycomplex and leading to an evident distortion of the FeTe4 tetrahedral. Experimental results further demonstrate effective suppression of Feint concentration and ordering through appropriate Se substitution; notably, Fe1+yTe0.5Se0.5 manifests the lowest concentration of Feint atoms. Our findings additionally indicate that Se substitution is random, and nanoscale phase separation induced by Te/Se chemical heterogeneity is commonly observed within Fe1+yTe1-xSex crystals., Comment: 14 pages, 5 figures

Sound Event Detection (SED) detects regions of sound events, while Speaker Diarization (SD) segments speech conversations attributed to individual speakers. In SED, all speaker segments are classified as a single speech event, while in SD, non-speech sounds are treated merely as background noise. Thus, both tasks provide only partial analysis in complex audio scenarios involving both speech conversation and non-speech sounds. In this paper, we introduce a novel task called Unified Audio Event Detection (UAED) for comprehensive audio analysis. UAED explores the synergy between SED and SD tasks, simultaneously detecting non-speech sound events and fine-grained speech events based on speaker identities. To tackle this task, we propose a Transformer-based UAED (T-UAED) framework and construct the UAED Data derived from the Librispeech dataset and DESED soundbank. Experiments demonstrate that the proposed framework effectively exploits task interactions and substantially outperforms the baseline that simply combines the outputs of SED and SD models. T-UAED also shows its versatility by performing comparably to specialized models for individual SED and SD tasks on DESED and CALLHOME datasets., Comment: submitted to ICASSP 2025

The echo signals of frequency diverse array multiple-input multiple-output (FDA-MIMO) feature angle-range coupling, enabling simultaneous discrimination and estimation of multiple targets at different locations. In light of this, based on FDA-MIMO, this paper explores an sensing-centric integrated sensing and communication (ISAC) system for multi-target sensing. On the transmitter side, the complex coefficients index modulation (CCIM) scheme is designed, which carries extra bits by selecting complex coefficients from the coefficient vector. At the sensing receiver, we propose the FDA-MIMO-based spatial spectrum multi-target estimation (SSMTE) method, which first jointly estimates the angle and distance of targets and then estimates the velocities. To reduce the sensing computational complexity, the low-complexity spatial spectrum estimation (LCSSE) algorithm is proposed. LCSSE reduces the complexity without degrading the sensing performance by converting the joint angle-range search into two one-dimensional searches. To address the range ambiguity caused by frequency offset, a frequency offset design criterion (FODC) is proposed. It designs the integer and fractional components of the frequency offset to ensure the ambiguity distance exceeds the maximum sensing range, thereby alleviating parameters pairing errors. Moreover, the closed-form expressions for the bit error rate (BER) tight upper bound and the Cram\'er-Rao bound (CRB) are derived. Simulation results show that the proposed system excels in multi-target sensing and communications.

Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1)extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The related code, demos, and pre-trained models are available at https://github.com/jishengpeng/WavTokenizer., Comment: Working in progress

Automatic Speech Recognition (ASR) transcripts exhibit recognition errors and various spoken language phenomena such as disfluencies, ungrammatical sentences, and incomplete sentences, hence suffering from poor readability. To improve readability, we propose a Contextualized Spoken-to-Written conversion (CoS2W) task to address ASR and grammar errors and also transfer the informal text into the formal style with content preserved, utilizing contexts and auxiliary information. This task naturally matches the in-context learning capabilities of Large Language Models (LLMs). To facilitate comprehensive comparisons of various LLMs, we construct a document-level Spoken-to-Written conversion of ASR Transcripts Benchmark (SWAB) dataset. Using SWAB, we study the impact of different granularity levels on the CoS2W performance, and propose methods to exploit contexts and auxiliary information to enhance the outputs. Experimental results reveal that LLMs have the potential to excel in the CoS2W task, particularly in grammaticality and formality, our methods achieve effective understanding of contexts and auxiliary information by LLMs. We further investigate the effectiveness of using LLMs as evaluators and find that LLM evaluators show strong correlations with human evaluations on rankings of faithfulness and formality, which validates the reliability of LLM evaluators for the CoS2W task., Comment: 7 pages, 3 figures

The first part of this series introduced the effectiveness of frequency diverse array (FDA) jamming through direct wave propagation in countering airborne phased multiple-input multiple-output (Phased-MIMO) radar. This part focuses on the effectiveness of FDA scattered wave (FDA-SW) jamming on the space-time adaptive processing (STAP) for airborne phased-MIMO radar. Distinguished from the clutter signals, the ground equidistant scatterers of FDA-SW jamming constitute an elliptical ring, whose trajectory equations are mathematically derived to further determine the spatial frequency and Doppler frequency. For the phased-MIMO radar with different transmitting partitions, the effects of jamming frequency offset of FDA-SW on the clutter rank and STAP performance are discussed. Theoretical analysis provides the variation interval of clutter rank and the relationship between the jamming frequency offset and the improvement factor (IF) notch of phased-MIMO-STAP. Importantly, the requirements of jamming frequency offset for both two-part applications are discussed in this part. Numerical results verify these mathematical findings and validate the effectiveness of the proposed FDA jamming in countering the phased-MIMO radar.

Phased multiple-input multiple-output (Phased-MIMO) radar has received increasing attention for enjoying the advantages of waveform diversity and range-dependency from frequency diverse array MIMO (FDA-MIMO) radar without sacrificing coherent processing gain through partitioning transmit subarray. This two-part series proposes a framework of electronic countermeasures (ECM) inspired by frequency diverse array (FDA) radar, called FDA jamming, evaluating its effectiveness for countering airborne phased-MIMO radar. This part introduces the principles and categories of FDA jammer and proposes the FDA jamming signal model based on the two cases of phased-MIMO radar, phased-array (PA) radar and FDA-MIMO radar. Moreover, the effects of FDA jamming on matched filtering and spatial filtering of PA and FDA-MIMO radar are analyzed. Numerical results verify the theoretical analysis and validate the effectiveness of the proposed FDA jamming in countering phased-MIMO radar.

The combination of frequency diverse array (FDA) radar technology with the multiple input multiple output (MIMO) radar architecture and waveform diversity techniques potentially promises a high integration gain with respect to conventional phased array (PA) radars. In this paper, we propose an approach to the design of the transmitter and the receiver of a coherent FDA (C-FDA) radar, that enables it to perform the demodulation with spectral overlapping, due to the small frequency offset. To this purpose, we derive the generalized space-time-range signal model and we prove that the proposed C-FDA radar has a higher coherent array gain than a PA radar, and at the same time, it effectively resolves the secondary range-ambiguous (SRA) problem of FDA-MIMO radar, allowing for mainlobe interference suppression and range-ambiguous clutter suppression. Numerical analysis results prove the effectiveness of the proposed C-FDA radar in terms on anti-interference and anti-clutter capabilities over conventional radars.

The video topic segmentation (VTS) task segments videos into intelligible, non-overlapping topics, facilitating efficient comprehension of video content and quick access to specific content. VTS is also critical to various downstream video understanding tasks. Traditional VTS methods using shallow features or unsupervised approaches struggle to accurately discern the nuances of topical transitions. Recently, supervised approaches have achieved superior performance on video action or scene segmentation over unsupervised approaches. In this work, we improve supervised VTS by thoroughly exploring multimodal fusion and multimodal coherence modeling. Specifically, (1) we enhance multimodal fusion by exploring different architectures using cross-attention and mixture of experts. (2) To generally strengthen multimodality alignment and fusion, we pre-train and fine-tune the model with multimodal contrastive learning. (3) We propose a new pre-training task tailored for the VTS task, and a novel fine-tuning task for enhancing multimodal coherence modeling for VTS. We evaluate the proposed approaches on educational videos, in the form of lectures, due to the vital role of topic segmentation of educational videos in boosting learning experiences. Additionally, we introduce a large-scale Chinese lecture video dataset to augment the existing English corpus, promoting further research in VTS. Experiments on both English and Chinese lecture datasets demonstrate that our model achieves superior VTS performance compared to competitive unsupervised and supervised baselines.

In smart city development, the automatic detection of structures and vehicles within urban or suburban areas via array radar (airborne or vehicle platforms) becomes crucial. However, the inescapable multipath effect adversely affects the radar's capability to detect and track targets. Frequency Diversity Array (FDA)-MIMO radar offers innovative solutions in mitigating multipath due to its frequency flexibility and waveform diversity traits amongst array elements. Hence, utilizing FDA-MIMO radar, this research proposes a multipath discrimination and suppression strategy to augment target detection and suppress false alarms. The primary advancement is the transformation of conventional multipath suppression into a multipath recognition issue, thereby enabling multipath components from single-frame echo data to be separated without prior knowledge. By offsetting the distance steering vectors of different objects to be detected, the accurate spectral information corresponding to the current distance unit can be extracted during spatial spectrum estimation. The direct and multipath components are differentiated depending on whether the transmitting and receiving angles match. Additionally, to mitigate high-order multipath, the echo intensity of multipath components is reduced via joint optimization of array transmit weighting and frequency increment. The numerical results show that the proposed algorithm can identify multipath at different distances in both single-target and multi-target scenarios, which is superior to the general MIMO radar., Comment: 14 pages

Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: https://aim-uofa.github.io/MovieDreamer/., Comment: 30 pages, 22 figures

We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. Rather than concentrating on specific use cases such as appearance editing (image harmonization) or semantic editing (semantic image composition), we showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition applicable to both scenarios. We observe that the pre-trained diffusion models automatically identify simple copy-paste boundary areas as low-density regions during denoising. Building on this insight, we propose to optimize the composed image towards high-density regions guided by the diffusion prior. In addition, we introduce a novel maskguided loss to further enable flexible semantic image composition. Extensive experiments validate the superiority of our approach in achieving generic zero-shot image composition. Additionally, our approach shows promising potential in various tasks, such as object removal and multiconcept customization., Comment: Accepted to Proc. Eur. Conf. Comp. Vision 2024. Project webpage: https://github.com/aim-uofa/FreeCompose

This report introduces FunAudioLLM, a model family designed to enhance natural voice interactions between humans and large language models (LLMs). At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and SenseVoice-Large supports high-precision ASR for over 50 languages, while CosyVoice excels in multi-lingual voice generation, zero-shot in-context learning, cross-lingual voice cloning, and instruction-following capabilities. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub. By integrating these models with LLMs, FunAudioLLM enables applications such as speech-to-speech translation, emotional voice chat, interactive podcasts, and expressive audiobook narration, thereby pushing the boundaries of voice interaction technology. Demos are available at https://fun-audio-llm.github.io, and the code can be accessed at https://github.com/FunAudioLLM., Comment: Work in progress. Authors are listed in alphabetical order by family name

LLMs have achieved significant performance progress in various NLP applications. However, LLMs still struggle to meet the strict requirements for accuracy and reliability in the medical field and face many challenges in clinical applications. Existing clinical diagnostic evaluation benchmarks for evaluating medical agents powered by LLMs have severe limitations. Firstly, most existing medical evaluation benchmarks face the risk of data leakage or contamination. Secondly, existing benchmarks often neglect the characteristics of multiple departments and specializations in modern medical practice. Thirdly, existing evaluation methods are limited to multiple-choice questions, which do not align with the real-world diagnostic scenarios. Lastly, existing evaluation methods lack comprehensive evaluations of end-to-end real clinical scenarios. These limitations in benchmarks in turn obstruct advancements of LLMs and agents for medicine. To address these limitations, we introduce ClinicalLab, a comprehensive clinical diagnosis agent alignment suite. ClinicalLab includes ClinicalBench, an end-to-end multi-departmental clinical diagnostic evaluation benchmark for evaluating medical agents and LLMs. ClinicalBench is based on real cases that cover 24 departments and 150 diseases. ClinicalLab also includes four novel metrics (ClinicalMetrics) for evaluating the effectiveness of LLMs in clinical diagnostic tasks. We evaluate 17 LLMs and find that their performance varies significantly across different departments. Based on these findings, in ClinicalLab, we propose ClinicalAgent, an end-to-end clinical agent that aligns with real-world clinical diagnostic practices. We systematically investigate the performance and applicable scenarios of variants of ClinicalAgent on ClinicalBench. Our findings demonstrate the importance of aligning with modern medical practices in designing medical agents.

The Transformer architecture has significantly advanced deep learning, particularly in natural language processing, by effectively managing long-range dependencies. However, as the demand for understanding complex relationships grows, refining the Transformer's architecture becomes critical. This paper introduces Skip-Layer Attention (SLA) to enhance Transformer models by enabling direct attention between non-adjacent layers. This method improves the model's ability to capture dependencies between high-level abstract features and low-level details. By facilitating direct attention between these diverse feature levels, our approach overcomes the limitations of current Transformers, which often rely on suboptimal intra-layer attention. Our implementation extends the Transformer's functionality by enabling queries in a given layer to interact with keys and values from both the current layer and one preceding layer, thus enhancing the diversity of multi-head attention without additional computational burden. Extensive experiments demonstrate that our enhanced Transformer model achieves superior performance in language modeling tasks, highlighting the effectiveness of our skip-layer attention mechanism., Comment: 7 pages, 1 figure

Training speaker-discriminative and robust speaker verification systems without explicit speaker labels remains a persisting challenge. In this paper, we propose a new self-supervised speaker verification approach, Self-Distillation Prototypes Network (SDPN), which effectively facilitates self-supervised speaker representation learning. SDPN assigns the representation of the augmented views of an utterance to the same prototypes as the representation of the original view, thereby enabling effective knowledge transfer between the views. Originally, due to the lack of negative pairs in the SDPN training process, the network tends to align positive pairs very closely in the embedding space, a phenomenon known as model collapse. To alleviate this problem, we introduce a diversity regularization term to embeddings in SDPN. Comprehensive experiments on the VoxCeleb datasets demonstrate the superiority of SDPN in self-supervised speaker verification. SDPN sets a new state-of-the-art on the VoxCeleb1 speaker verification evaluation benchmark, achieving Equal Error Rate 1.80%, 1.99%, and 3.62% for trial VoxCeleb1-O, VoxCeleb1-E and VoxCeleb1-H respectively, without using any speaker labels in training., Comment: We update this paper to an earlier paper

In this paper, we present ControlSpeech, a text-to-speech (TTS) system capable of fully cloning the speaker's voice and enabling arbitrary control and adjustment of speaking style, merely based on a few seconds of audio prompt and a simple textual style description prompt. Prior zero-shot TTS models and controllable TTS models either could only mimic the speaker's voice without further control and adjustment capabilities or were unrelated to speaker-specific voice generation. Therefore, ControlSpeech focuses on a more challenging new task-a TTS system with controllable timbre, content, and style at the same time. ControlSpeech takes speech prompts, content prompts, and style prompts as inputs and utilizes bidirectional attention and mask-based parallel decoding to capture corresponding codec representations in a discrete decoupling codec space. Moreover, we discovered the issue of text style controllability in a many-to-many mapping fashion and proposed the Style Mixture Semantic Density (SMSD) model to resolve this problem. SMSD module which is based on Gaussian mixture density networks, is designed to enhance the fine-grained partitioning and sampling capabilities of style semantic information and generate speech with more diverse styles. In terms of experiments, we make available a controllable model toolkit called ControlToolkit with a new style controllable dataset, some replicated baseline models and propose new metrics to evaluate both the control capability and the quality of generated audio in ControlSpeech. The relevant ablation studies validate the necessity of each component in ControlSpeech is necessary. We hope that ControlSpeech can establish the next foundation paradigm of controllable speech synthesis. The relevant code and demo are available at https://github.com/jishengpeng/ControlSpeech .

Although reconfigurable intelligent surfaces (RISs) have demonstrated the potential to boost network capacity and expand coverage by adjusting their electromagnetic properties, existing RIS architectures have certain limitations, such as double-fading attenuation and restricted half-space coverage. In this article, we delve into the progressive development from single to multi-functional RIS (MF-RIS) that enables simultaneous signal amplification, reflection, and refraction. We begin by detailing the hardware design and signal model that distinguish MF-RIS from traditional RISs. Subsequently, we introduce the key technologies underpinning MF-RIS-aided communications, along with the fundamental issues and challenges inherent to its deployment. We then outline the promising applications of MFRIS in the realm of communication, sensing, and computation systems, highlighting its transformative impact on these domains. Lastly, we present simulation results to demonstrate the superiority of MF-RIS in enhancing network performance in terms of spectral efficiency., Comment: 9 pages, 6 figures, submitted to IEEE magazines

Complex spatiotemporal patterns in nature significantly challenge reductionism-based modern science. The lack of a paradigm beyond reductionism hinders our understanding of the emergence of complexity. The diversity of countless patterns undermines any notion of universal mechanisms. Here, however, we show that breaking the symmetry of three simple and self-organization rules give rise to nearly all patterns in nature, such as a wide variety of Turing patterns, fractals, spiral, target and plane waves, as well as chaotic patterns. The symmetry breaking is rooted in basic physical quantities, such as positive and negative forces, space, time and bounds. Besides reproducing the hallmarks of complexity, we discover some novel phenomena, such as abrupt percolation of Turing patterns, phase transition between fractals and chaos, chaotic edge in travelling waves, etc. Our asymmetric self-organization theory established a simple and unified framework for the origin of complexity in all fields, and unveiled a deep relationship between the first principles of physics and the complex world, Comment: 97 pages, 60 figures, for full article, see https://doi.org/10.1142/S021812742430012X

Benefiting from large-scale pre-trained text-to-image (T2I) generative models, impressive progress has been achieved in customized image generation, which aims to generate user-specified concepts. Existing approaches have extensively focused on single-concept customization and still encounter challenges when it comes to complex scenarios that involve combining multiple concepts. These approaches often require retraining/fine-tuning using a few images, leading to time-consuming training processes and impeding their swift implementation. Furthermore, the reliance on multiple images to represent a singular concept increases the difficulty of customization. To this end, we propose FreeCustom, a novel tuning-free method to generate customized images of multi-concept composition based on reference concepts, using only one image per concept as input. Specifically, we introduce a new multi-reference self-attention (MRSA) mechanism and a weighted mask strategy that enables the generated image to access and focus more on the reference concepts. In addition, MRSA leverages our key finding that input concepts are better preserved when providing images with context interactions. Experiments show that our method's produced images are consistent with the given concepts and better aligned with the input text. Our method outperforms or performs on par with other training-based methods in terms of multi-concept composition and single-concept customization, but is simpler. Codes can be found at https://github.com/aim-uofa/FreeCustom., Comment: CVPR2024

Motivated by recent experimental observations of displacement-field-tuned correlated insulators at integer and half-integer fillings in chirally twisted triple bilayer graphene (CTTBG), we study the single-particle and interacting physics of CTTBG. We find that there are two inequivalent stacking orders, {\it i.e.}, ABABBC and ABABAB, and both exhibit flat bands with nontrivial topology. We then use the Hartree-Fock approximation to calculate the rich phase diagram of CTTBG at all integer and half-integer fillings in both stacking orders and under the vertical displacement field. Under a small displacement field, the groundstates are flavor polarized states for ABABBC stacking order and intervalley coherent states for ABABAB stacking order at all integer and half-integer fillings. A larger displacement field will turn them into layer-polarized states. At half-integer fillings, the groundstates also exhibit charge density wave (CDW) order. For ABABAB stacking, the groundstates are always $2\times1$ stripe state among a range of displacement fields. For ABABBC stacking, the groundstates are also $2\times1$ stripe states under a small displacement field and a larger displacement will possibly favor further translation-symmetry-breaking, depending on filling and the direction of the displacement field. We demonstrate that the CDW states observed in the experiment can originate from the strong Coulomb interaction of the flat band electrons.

Large Language Models (LLMs) have made significant progress in code generation, offering developers groundbreaking automated programming support. However, LLMs often generate code that is syntactically correct and even semantically plausible, but may not execute as expected or fulfill specified requirements. This phenomenon of hallucinations in the code domain has not been systematically explored. To advance the community's understanding and research on this issue, we introduce the concept of code hallucinations and propose a classification method for code hallucination based on execution verification. We categorize code hallucinations into four main types: mapping, naming, resource, and logic hallucinations, with each category further divided into different subcategories to understand and address the unique challenges faced by LLMs in code generation with finer granularity. Additionally, we present a dynamic detection algorithm called CodeHalu designed to detect and quantify code hallucinations. We also introduce the CodeHaluEval benchmark, which includes 8,883 samples from 699 tasks, to systematically and quantitatively evaluate code hallucinations. By evaluating 17 popular LLMs using this benchmark, we reveal significant differences in their accuracy and reliability in code generation, offering detailed insights for further improving the code generation capabilities of LLMs. The CodeHalu benchmark and code are publicly available at https://github.com/yuchen814/CodeHalu.

On-device intelligence (ODI) enables artificial intelligence (AI) applications to run on end devices, providing real-time and customized AI inference without relying on remote servers. However, training models for on-device deployment face significant challenges due to the decentralized and privacy-sensitive nature of users' data, along with end-side constraints related to network connectivity, computation efficiency, etc. Existing training paradigms, such as cloud-based training, federated learning, and transfer learning, fail to sufficiently address these practical constraints that are prevalent for devices. To overcome these challenges, we propose Privacy-Enhanced Training-as-a-Service (PTaaS), a novel service computing paradigm that provides privacy-friendly, customized AI model training for end devices. PTaaS outsources the core training process to remote and powerful cloud or edge servers, efficiently developing customized on-device models based on uploaded anonymous queries, enhancing data privacy while reducing the computation load on individual devices. We explore the definition, goals, and design principles of PTaaS, alongside emerging technologies that support the PTaaS paradigm. An architectural scheme for PTaaS is also presented, followed by a series of open problems that set the stage for future research directions in the field of PTaaS., Comment: 7 pages, 3 figures

The emerging field of topology has brought device effects to a new level. Higher-order topological insulators (HOTIs) go beyond traditional descriptions of bulk-edge correspondence, broadening the understanding of topologically insulating phases. In this paper, a second-order split-ring photonic crystal (SSPC) with zero-dimensional (0D) corner states and one-dimensional (1D) edge states is proposed. Based on the coupling strength determined by the opening direction between the split-rings, the electronic transition strength of the electronic system is imitated, and the topological trivial and non-trivial transformation of the topological two-dimensional (2D) SSH model are realized by using the rotating split-ring lattice. Theory and simulation find that SSPC has non-trivial topological edge states that can be quantified by bulk polarization. As the opening direction of the split-rings gradually changes within one period, there will be transitions between four different topological polarizations of the lowest energy bands, which can be conveniently used to achieve transitions between different topological phases. Our research can be extended to higher dimensions and broaden research paths for higher-order photonic topological insulators and semimetals., Comment: 7 pages, 4 figures

We introduce 3D-Speaker-Toolkit, an open-source toolkit for multimodal speaker verification and diarization, designed for meeting the needs of academic researchers and industrial practitioners. The 3D-Speaker-Toolkit adeptly leverages the combined strengths of acoustic, semantic, and visual data, seamlessly fusing these modalities to offer robust speaker recognition capabilities. The acoustic module extracts speaker embeddings from acoustic features, employing both fully-supervised and self-supervised learning approaches. The semantic module leverages advanced language models to comprehend the substance and context of spoken language, thereby augmenting the system's proficiency in distinguishing speakers through linguistic patterns. The visual module applies image processing technologies to scrutinize facial features, which bolsters the precision of speaker diarization in multi-speaker environments. Collectively, these modules empower the 3D-Speaker-Toolkit to achieve substantially improved accuracy and reliability in speaker-related tasks. With 3D-Speaker-Toolkit, we establish a new benchmark for multimodal speaker analysis. The toolkit also includes a handful of open-source state-of-the-art models and a large-scale dataset containing over 10,000 speakers. The toolkit is publicly available at https://github.com/modelscope/3D-Speaker.

In this work, we systematically calculate the spectrum of hidden-charm pentaquark states $qqqc\bar{c}$ $(q = u,d)$ in the chiral SU(3) quark model, which has been quite successful in reproducing consistently the energies of octet and decuplet baryon ground states, the binding energy of deuteron, and the nucleon-nucleon ($NN$) scattering phase shifts and mixing parameters for partial waves with total angular momentum up to $J=6$. The Hamiltonian contains the one-gluon-exchange (OGE) potential, the Goldstone-boson-exchange (GBE) potential, the confinement potential, and the kinetic energy of the system. We solve the Schr\"odinger equation by use of the variational method. It is found that the masses of all the experimentally observed $P_c(4312)$, $P_c(4380)$, $P_c(4440)$, and $P_c(4457)$ states are much overestimated, indicating that these states are not compact pentaquark states in the chiral SU(3) quark model. All other $qqqc\bar{c}$ $(q = u,d)$ states are found to lie much above the corresponding baryon-meson thresholds, and thus are not suggested as stable pentaquark states due to their fall-apart decays. A detailed comparison of the results with those obtained in the OGE model and the chromomagnetic interaction (CMI) model is further given., Comment: 9 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:2403.15187

Inspired by the resonance $T_{cc}^+(3875)$ recently observed by the LHCb Collaboration, we systematically explore the $S$- and $P$-wave $cc\bar{q}\bar{q}'$ $(\bar{q},\bar{q}' = \bar{u}, \bar{d}, \bar{s})$ systems in a chiral SU(3) quark model. The Hamiltonian contains the kinetic energy, the one-gluon-exchange (OGE) potential, the confinement potential, and the one-boson-exchange (OBE) potential stemming from the coupling of quark and chiral fields. The Schr\"odinger equation is solved by use of the variational method with the spacial trial wave functions chosen as Gaussian functions. It is found that the lowest state has a mass $3879$ MeV, isospin and spin-parity $IJ^P=01^+$, and quark constituent $cc\bar{u}\bar{d}$, in agreement with the experimentally observed $T_{cc}^+(3875)$. This state is approximately at the calculated $DD^\ast$ threshold, and has a root-mean-square radius about $0.48$ fm. These demonstrates that the $T_{cc}^+(3875)$ can be accommodated as a stable and compact tetraquark sate in the chiral SU(3) quark model. All the other $S$- and $P$-wave $cc\bar{q}\bar{q}'$ $(\bar{q},\bar{q}' = \bar{u}, \bar{d}, \bar{s})$ states lie about one hundred to few hundreds MeV higher than the corresponding meson-meson thresholds, and thus are not suggested to be candidates of stable and compact tetraquark states due to their fall-apart decays to two mesons., Comment: 9 pages, 2 figures

This paper addresses the problem of detecting a moving target embedded in Gaussian noise with an unknown covariance matrix for frequency diverse array multiple-input multiple-output (FDA-MIMO) radar. To end it, assume that obtaining a set of training data is available. Moreover, we propose three adaptive detectors in accordance with the one-step generalized likelihood ratio test (GLRT), two-step GLRT, and Rao criteria, namely OGLRT, TGLRT, and Rao. The LH adaptive matched filter (LHAMF) detector is also introduced when decomposing the Rao test. Next, all provided detectors have constant false alarm rate (CFAR) properties against the covariance matrix. Besides, the closed-form expressions for false alarm probability (PFA) and detection probability (PD) are derived. Finally, this paper substantiates the correctness of the aforementioned algorithms through numerical simulations.

Customization generation techniques have significantly advanced the synthesis of specific concepts across varied contexts. Multi-concept customization emerges as the challenging task within this domain. Existing approaches often rely on training a fusion matrix of multiple Low-Rank Adaptations (LoRAs) to merge various concepts into a single image. However, we identify this straightforward method faces two major challenges: 1) concept confusion, where the model struggles to preserve distinct individual characteristics, and 2) concept vanishing, where the model fails to generate the intended subjects. To address these issues, we introduce LoRA-Composer, a training-free framework designed for seamlessly integrating multiple LoRAs, thereby enhancing the harmony among different concepts within generated images. LoRA-Composer addresses concept vanishing through concept injection constraints, enhancing concept visibility via an expanded cross-attention mechanism. To combat concept confusion, concept isolation constraints are introduced, refining the self-attention computation. Furthermore, latent re-initialization is proposed to effectively stimulate concept-specific latent within designated regions. Our extensive testing showcases a notable enhancement in LoRA-Composer's performance compared to standard baselines, especially when eliminating the image-based conditions like canny edge or pose estimations. Code is released at \url{https://github.com/Young98CN/LoRA_Composer}, Comment: project page: https://github.com/Young98CN/LoRA_Composer

The quantum Rabi model (QRM) is a fundamental model for light-matter interactions. A fascinating feature of the QRM is that it manifests a quantum phase transition which is applicable for critical quantum metrology (CQM). Effective application for CQM needs the exact location of the transition point, however the conventional expression for the transition coupling is only valid in the extreme limit of low frequency, while apart from zero frequency an accurate location is still lacking. In the present work we conversely use the quantum Fisher information (QFI) in the CQM to identify the transition coupling, which finds out that transition coupling indeed much deviates from the conventional one once a finite frequency is turned on. Polaron picture is applied to analytically reproduce the numeric QFI. An accurate expression for the transition coupling is obtained by the inspiration from the fractional-power-law effect of polaron frequency renormalization. From the QFI in the polaron picture we find that the transition occurs around a point where the effective velocity and the susceptibility of the single-photon absorption rate reach maximum. Our result provides an accurate reference of transition couplings for quantum metrology at non-zero frequencies. The formulation of the QFI in the polaron picture also prepares an analytic method with an accurate compensation for the parameter regime difficult to access for the numerics. Besides the integer/fractional power law analysis to extract the underlying physics of transition, the QFI/velocity relation may also add some insight in bridging the QFI and transition observables., Comment: 14 pages, 8 figures

Very recently, the Belle~II Collaboration presented a measurement for the decays $B^+\to\bar{D}^{(*)0} K^+\bar{K}^0$ and $B^0\to D^{(*)-}K^+\bar{K}^0$, the bulk of observed $m(K^+ K_S^0)$ distributions showing low-mass structures in all four channels. In this work, we study the contributions of $\rho(770,1450)^+$, $a_2(1320)^+$ and $a_0(980,1450)^+$ resonances to these decay processes. The intermediate states $\rho(770,1450)^+$ are found to dominate the low-mass distribution of kaon pairs roughly contributing to half of the total branching fraction in each of the four decay channels. The contribution of the tensor $a_2(1320)^+$ meson is found to be negligible. Near the threshold of the kaon pair, the state $a_0(980)^+$ turns out to be much less important than expected, not being able to account for the enhancement of events in that energy region observed in the $B^+\to\bar{D}^{(*)0} K^+\bar{K}^0$ decays. Further studies both from the theoretical and experimental sides are needed to elucidate the role of the non-resonant contributions governing the formation of $K^+\bar{K}^0$ pairs near their threshold in these decay processes., Comment: 18 pages, 4 figures

Lack of nonreciprocity is one of the major drawbacks of solid-state acoustic devices, which has hindered the development of microwave-frequency acoustic isolators and circulators. Here we report giant nonreciprocal transmission of shear-horizontal surface acoustic waves (SH-SAWs) on a LiTaO3 substrate coated with a negative-positive magnetostrictive bilayer structure of Ni/Ti/FeCoSiB. Although the static magnetic moments of two layers are parallel, SH-SAWs can excite optical-mode spin waves much stronger than acoustic-mode ones at relatively low frequencies via magnetoelastic coupling. The measured magnitude nonreciprocity exceeds 40 dB (or 80 dB/mm) at 2.333 GHz. In addition, maximum nonreciprocal phase accumulation reaches 188{\deg} (376{\deg}/mm), which is desired for an effective SAW circulator. Our theoretical model and calculations provide an insight into the observed phenomena and demonstrate a pathway for further improvement of nonreciprocal acoustic devices.