Search

Large Language Models (LLMs) have demonstrated remarkable capabilities in language understanding and generation. Advanced utilization of the knowledge embedded in LLMs for automated annotation has consistently been explored. This study proposed to develop an emotion lexicon for Cantonese, a low-resource language, through collaborative efforts between LLM and human annotators. By integrating emotion labels provided by LLM and human annotators, the study leveraged existing linguistic resources including lexicons in other languages and local forums to construct a Cantonese emotion lexicon enriched with colloquial expressions. The consistency of the proposed emotion lexicon in emotion extraction was assessed through modification and utilization of three distinct emotion text datasets. This study not only validates the efficacy of the constructed lexicon but also emphasizes that collaborative annotation between human and artificial intelligence can significantly enhance the quality of emotion labels, highlighting the potential of such partnerships in facilitating natural language processing tasks for low-resource languages., Comment: 13 pages

Millimeter-wave radar is promising to provide robust and accurate vital sign monitoring in an unobtrusive manner. However, the radar signal might be distorted in propagation by ambient noise or random body movement, ruining the subtle cardiac activities and destroying the vital sign recovery. In particular, the recovery of electrocardiogram (ECG) signal heavily relies on the deep-learning model and is sensitive to noise. Therefore, this work creatively deconstructs the radar-based ECG recovery into three individual tasks and proposes a multi-task learning (MTL) framework, radarODE-MTL, to increase the robustness against consistent and abrupt noises. In addition, to alleviate the potential conflicts in optimizing individual tasks, a novel multi-task optimization strategy, eccentric gradient alignment (EGA), is proposed to dynamically trim the task-specific gradients based on task difficulties in orthogonal space. The proposed radarODE-MTL with EGA is evaluated on the public dataset with prominent improvements in accuracy, and the performance remains consistent under noises. The experimental results indicate that radarODE-MTL could reconstruct accurate ECG signals robustly from radar signals and imply the application prospect in real-life situations. The code is available at: http://github.com/ZYY0844/radarODE-MTL.

Open Radio Access Networks (RAN) aim to bring a paradigm shift to telecommunications industry, by enabling an open, intelligent, virtualized, and multi-vendor interoperable RAN ecosystem. At the center of this movement, O-RAN ALLIANCE defines the O-RAN architecture and standards, so that companies around the globe can use these specifications to create innovative and interoperable solutions. To accelerate the adoption of O-RAN products, rigorous testing of O-RAN Radio Unit (O-RU) and other O-RAN products plays a key role. O-RAN ALLIANCE has approved around 20 Open Testing and Integration Centres (OTICs) globally. OTICs serve as vendor-neutral platforms for providing the testing and integration services, with the vision that an O-RAN product certified in any OTIC is accepted in other parts of the world. To demonstrate the viability of such a certified-once-and-use-everywhere approach, one theme in the O-RAN Global PlugFest Spring 2024 is to demonstrate consistent and repeatable testing for the open fronthaul interface across multiple labs. Towards this, Japan OTIC and Asia Pacific OTIC in Singapore have teamed up together with an O-RU vendor and Keysight Technology. Our international team successfully completed all test cases defined by O-RAN ALLIANCE for O-RU conformance testing. In this paper, we share our journey in achieving this outcome, focusing on the challenges we have overcome and the lessons we have learned through this process., Comment: Published version at RitiRAN Workshop - co-located with IEEE VTC Fall 2024

Early detection of chronic diseases is beneficial to healthcare by providing a golden opportunity for timely interventions. Although numerous prior studies have successfully used machine learning (ML) models for disease diagnoses, they highly rely on medical data, which are scarce for most patients in the early stage of the chronic diseases. In this paper, we aim to diagnose hyperglycemia (diabetes), hyperlipidemia, and hypertension (collectively known as 3H) using own collected behavioral data, thus, enable the early detection of 3H without using medical data collected in clinical settings. Specifically, we collected daily behavioral data from 629 participants over a 3-month study period, and trained various ML models after data preprocessing. Experimental results show that only using the participants' uploaded behavioral data, we can achieve accurate 3H diagnoses: 80.2\%, 71.3\%, and 81.2\% for diabetes, hyperlipidemia, and hypertension, respectively. Furthermore, we conduct Shapley analysis on the trained models to identify the most influential features for each type of diseases. The identified influential features are consistent with those reported in the literature.

In this work, we describe our submissions for the Voice Privacy Challenge 2024. Rather than proposing a novel speech anonymization system, we enhance the provided baselines to meet all required conditions and improve evaluated metrics. Specifically, we implement emotion embedding and experiment with WavLM and ECAPA2 speaker embedders for the B3 baseline. Additionally, we compare different speaker and prosody anonymization techniques. Furthermore, we introduce Mean Reversion F0 for B5, which helps to enhance privacy without a loss in utility. Finally, we explore disentanglement models, namely $\beta$-VAE and NaturalSpeech3 FACodec., Comment: System description for VPC 2024

Accurate real-time tracking of dexterous hand movements and interactions has numerous applications in human-computer interaction, metaverse, robotics, and tele-health. Capturing realistic hand movements is challenging because of the large number of articulations and degrees of freedom. Here, we report accurate and dynamic tracking of articulated hand and finger movements using stretchable, washable smart gloves with embedded helical sensor yarns and inertial measurement units. The sensor yarns have a high dynamic range, responding to low 0.005 % to high 155 % strains, and show stability during extensive use and washing cycles. We use multi-stage machine learning to report average joint angle estimation root mean square errors of 1.21 and 1.45 degrees for intra- and inter-subjects cross-validation, respectively, matching accuracy of costly motion capture cameras without occlusion or field of view limitations. We report a data augmentation technique that enhances robustness to noise and variations of sensors. We demonstrate accurate tracking of dexterous hand movements during object interactions, opening new avenues of applications including accurate typing on a mock paper keyboard, recognition of complex dynamic and static gestures adapted from American Sign Language and object identification.

Single-photon sources (SPS) hold the potential to enhance the performance of quantum key distribution (QKD). QKD systems using SPS often require cryogenic cooling, while recent QKD attempts using SPS operating at room-temperature have failed to achieve long-distance transmission due to the SPS not operating at telecommunication wavelength. In this work, we have successfully demonstrated QKD using a room-temperature SPS at telecommunication wavelength. The SPS used in this work is based on point defects hosted by gallium nitride (GaN) thin films grown on sapphire substrates. We employed a time-bin and phase encoding scheme to perform the BB84 and reference-frame-independent QKD protocols over a 33 km fiber spool, achieving a secure key rate of $7.58\times 10^{-7}$ per pulse. Moreover, we also implemented a metropolitan QKD experiment over a 30 km deployed fiber, achieving a secure key rate of $6.06\times 10^{-8}$ per pulse. These results broaden the prospects for future use of SPS in commercial QKD applications.

The scarcity of domain-specific dialogue datasets limits the development of dialogue systems across applications. Existing research is constrained by general or niche datasets that lack sufficient scale for training dialogue systems. To address this gap, we introduce DiaSynth - a synthetic dialogue generation framework capable of generating high-quality, contextually rich dialogues across a wide range of domains. Unlike existing frameworks, DiaSynth uses Large Language Models (LLMs) and Chain of Thought (CoT) reasoning to generate dynamic, domain-specific dialogues with simulated personas and diverse conversational features. We perform our experiments by generating synthetic data using different LLMs and few-shot examples from DialogSum and SAMSum. The pretrained language models fine-tuned on the synthetic data outperform the base models by 16.47% on dialogue summarization, while the comparison between models fine-tuned on in-domain data and synthetic data shows that the synthetic data is able to capture 90.48% of the performance distribution of the in-domain data on dialogue summarization. The quality of the data generated also increases as we increase the size of LLM from 3B to 8B. These results validate DiaSynth's potential as a robust alternative to traditional data collection methods. We open source the code and data generated for future research., Comment: 13 pages, 1 figure

Single photon sources (SPSs) are directly applicable in quantum key distribution (QKD) because they allow the implementation of the canonical BB84 protocol. To date, QKD implementations using SPS are not widespread because of the need for cryogenic operation, or frequency conversion to a wavelength efficiently transmitted over telecommunication fibers. We report an observation of polarization-encoded QKD using a room-temperature telecom SPS based on a GaN defect. A field test over 3.5 km of deployed fiber with 4.0 dB loss yielded a secure key rate of 585.9~bps. Further testing in a 32.5 km fiber spool (attenuation of 11.2 dB), which exhibited substantially lower polarization mode dispersion, yielded a secure key rate of 50.4 bps. Both results exhibited a quantum bit error rate (QBER) of approximately 5%. These results illustrate the potential of the GaN defects for supporting polarization-encoded quantum communication.

The integration of large language models (LLMs) with pre-trained speech models has opened up new avenues in automatic speech recognition (ASR). While LLMs excel in multimodal understanding tasks, effectively leveraging their capabilities for ASR remains a significant challenge. This paper presents a novel training approach to enhance LLM performance in ASR tasks. We propose pre-training LLMs on Pinyin embedding sequences, which represent pronunciation features, to generate corresponding Chinese characters. This step enables the LLM to adapt to generating text from pronunciation features before encountering real speech data. Furthermore, we fine-tune the LoRA parameters to enhance the LLM's understanding of speech modality information. In AISHELL-1 corpus, our approach yields a 9.5% relative improvement in ASR tasks compared to the baseline without Pinyi-to-Character pre-training. Additionally, incorporating auxiliary text data for Pinyi-to-Character pre-training further boosts performance, achieving a 19.0% relative improvement., Comment: Accepted by ISCSLP2024-Special session-Speech Processing in LLM Era

Previous fake speech datasets were constructed from a defender's perspective to develop countermeasure (CM) systems without considering diverse motivations of attackers. To better align with real-life scenarios, we created LlamaPartialSpoof, a 130-hour dataset contains both fully and partially fake speech, using a large language model (LLM) and voice cloning technologies to evaluate the robustness of CMs. By examining information valuable to both attackers and defenders, we identify several key vulnerabilities in current CM systems, which can be exploited to enhance attack success rates, including biases toward certain text-to-speech models or concatenation methods. Our experimental results indicate that current fake speech detection system struggle to generalize to unseen scenarios, achieving a best performance of 24.44% equal error rate., Comment: 5 pages, submitted to ICASSP 2025

The ASVspoof 2021 benchmark, a widely-used evaluation framework for anti-spoofing, consists of two subsets: Logical Access (LA) and Deepfake (DF), featuring samples with varied coding characteristics and compression artifacts. Notably, the current state-of-the-art (SOTA) system boasts impressive performance, achieving an Equal Error Rate (EER) of 0.87% on the LA subset and 2.58% on the DF. However, benchmark accuracy is no guarantee of robustness in real-world scenarios. This paper investigates the effectiveness of utilizing room impulse responses (RIRs) to enhance fake speech and increase their likelihood of evading fake speech detection systems. Our findings reveal that this simple approach significantly improves the evasion rate, doubling the SOTA system's EER. To counter this type of attack, We augmented training data with a large-scale synthetic/simulated RIR dataset. The results demonstrate significant improvement on both reverberated fake speech and original samples, reducing DF task EER to 2.13%., Comment: 7 pages, to be presented at SLT 2024

Anisotropic pair breaking close to surfaces favors chiral superfluid $^3$He-A over time-reversal invariant $^3$He-B. Confining superfluid $^3$He into a cavity of height $D$ of the order of the Cooper pair size characterized by the coherence length $\xi_0$ -- ranging between 16 nm (34 bar) and 77 nm (0 bar) -- extends the surface effects over the whole sample volume, thus allowing stabilization of the A phase at pressures $P$ and temperatures $T$ where otherwise the B phase would be stable. In this work the surfaces of such a confined sample are covered with a superfluid $^4$He film to create specular quasiparticle scattering boundary conditions, preventing the suppression of the superfluid order parameter. We show that the chiral A phase is the stable superfluid phase under strong confinement over the full $P-T$ phase diagram down to a quasi-two-dimensional limit $D/\xi_0 = 1$. The planar phase, which is degenerate with the chiral A phase in the weak-coupling limit, is not observed. The gap inferred from measurements over the wide pressure range from 0.2 to 21.0 bar leads to an empirical ansatz for temperature-dependent strong-coupling effects. We discuss how these results pave the way for the realization of the fully-gapped two-dimensional $p_x + ip_y$ superfluid under more extreme confinement., Comment: 11 pages, 8 figures, including the Supplemental Material

Inertial microfluidics have been limited to dilute particle concentrations due to defocusing at high particle concentrations. However, we observed a counterintuitive shift of focusing to the outer wall at high concentrations, which contradicts the existing particle focusing theory based on Navier-Stokes equation. We developed a multiphase model incorporating lift forces and particle-particle interactions to explain this behaviour. Numerical simulations validated by experimental data reveal the shift is governed by the ratio of the lift force strength to that of particle interaction frequencies.

Two-dimensional van der Waals semiconductors are promising for future nanoelectronics. However, integrating high-k gate dielectrics for device applications is challenging as the inert van der Waals material surfaces hinder uniform dielectric growth. Here, we report a liquid metal oxide-assisted approach to integrate ultrathin, high-k HfO2 dielectric on 2D semiconductors with atomically smooth interfaces. Using this approach, we fabricated 2D WS2 top-gated transistors with subthreshold swings down to 74.5 mV/dec, gate leakage current density below 10-6 A/cm2, and negligible hysteresis. We further demonstrate a one-step van der Waals integration of contacts and dielectrics on graphene. This can offer a scalable approach toward integrating entire prefabricated device stack arrays with 2D materials. Our work provides a scalable solution to address the crucial dielectric engineering challenge for 2D semiconductors, paving the way for high-performance 2D electronics.

Electric-dipole-forbidden transitions play an important role as in quantum sensing, quantum information, and fundamental test in physics. As such, the development of novel and sensitive spectroscopic methods is of major interest. Here, we present a Doppler-free selective reflection experiment on the 6S1/2 --> 5D5/2 electric-quadrupole transition of cesium vapor at the vicinity of a sapphire window. This is achieved by a precision experiment overcoming limitations due to the small signal amplitude of forbidden transitions. Narrow sub-Doppler lines allow for a collisional broadening measurement on the electric-quadrupole line. The interaction of cesium atoms with the sapphire surface of the cell is evidenced, but, due to its weak contribution, a quantitative analysis remains challenging. Nevertheless, our experiment paves the way for further studies of the Casimir-Polder interaction between exotic excited-state atoms and dielectric surfaces., Comment: submitted

Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model. To explore new capabilities in language modeling for speech processing, we introduce the generative speech transcription error correction (GenSEC) challenge. This challenge comprises three post-ASR language modeling tasks: (i) post-ASR transcription correction, (ii) speaker tagging, and (iii) emotion recognition. These tasks aim to emulate future LLM-based agents handling voice-based interfaces while remaining accessible to a broad audience by utilizing open pretrained language models or agent-based APIs. We also discuss insights from baseline evaluations, as well as lessons learned for designing future evaluations., Comment: IEEE SLT 2024. The initial draft version has been done in December 2023. Post-ASR Text Processing and Understanding Community and LlaMA-7B pre-training correction model: https://huggingface.co/GenSEC-LLM/SLT-Task1-Llama2-7b-HyPo-baseline

Future electronics require aggressive scaling of channel material thickness while maintaining device performance. Two-dimensional (2D) semiconductors are promising candidates, but despite over two decades of research, experimental performance still lags theoretical expectations. Here, we develop an oxygen-free approach to push the electrical transport of 2D field-effect transistors toward the theoretical phonon-limited intrinsic mobility. We achieve record carrier mobilities of 91 (132) cm2V-1s-1 for mono- (bi-) layer MoS2 transistors on SiO2 substrate. Statistics from over 60 devices confirm that oxygen-free fabrication enhances key figures of merit by more than an order of magnitude. While previous studies suggest that 2D transition metal dichalcogenides such as MoS2 and WS2 are stable in air, we show that short-term ambient exposure can degrade their device performance through irreversible oxygen chemisorption. This study emphasizes the criticality of avoiding oxygen exposure, offering guidance for device manufacturing for fundamental research and practical applications of 2D materials.

The uptake of formalized prior elicitation from experts in Bayesian clinical trials has been limited, largely due to the challenges associated with complex statistical modeling, the lack of practical tools, and the cognitive burden on experts required to quantify their uncertainty using probabilistic language. Additionally, existing methods do not address prior-posterior coherence, i.e., does the posterior distribution, obtained mathematically from combining the estimated prior with the trial data, reflect the expert's actual posterior beliefs? We propose a new elicitation approach that seeks to ensure prior-posterior coherence and reduce the expert's cognitive burden. This is achieved by eliciting responses about the expert's envisioned posterior judgments under various potential data outcomes and inferring the prior distribution by minimizing the discrepancies between these responses and the expected responses obtained from the posterior distribution. The feasibility and potential value of the new approach are illustrated through an application to a real trial currently underway., Comment: minor update

Speaker anonymization is an effective privacy protection solution that conceals the speaker's identity while preserving the linguistic content and paralinguistic information of the original speech. To establish a fair benchmark and facilitate comparison of speaker anonymization systems, the VoicePrivacy Challenge (VPC) was held in 2020 and 2022, with a new edition planned for 2024. In this paper, we describe our proposed speaker anonymization system for VPC 2024. Our system employs a disentangled neural codec architecture and a serial disentanglement strategy to gradually disentangle the global speaker identity and time-variant linguistic content and paralinguistic information. We introduce multiple distillation methods to disentangle linguistic content, speaker identity, and emotion. These methods include semantic distillation, supervised speaker distillation, and frame-level emotion distillation. Based on these distillations, we anonymize the original speaker identity using a weighted sum of a set of candidate speaker identities and a randomly generated speaker identity. Our system achieves the best trade-off of privacy protection and emotion preservation in VPC 2024., Comment: System description for VPC 2024

Recently, visual grounding and multi-sensors setting have been incorporated into perception system for terrestrial autonomous driving systems and Unmanned Surface Vehicles (USVs), yet the high complexity of modern learning-based visual grounding model using multi-sensors prevents such model to be deployed on USVs in the real-life. To this end, we design a low-power multi-task model named NanoMVG for waterway embodied perception, guiding both camera and 4D millimeter-wave radar to locate specific object(s) through natural language. NanoMVG can perform both box-level and mask-level visual grounding tasks simultaneously. Compared to other visual grounding models, NanoMVG achieves highly competitive performance on the WaterVG dataset, particularly in harsh environments and boasts ultra-low power consumption for long endurance., Comment: 8 pages, 6 figures

Solid-state defects susceptible of spin manipulation hold great promise for scalable quantum technology. To broaden their utility, operating at room temperature and emitting in the telecom wavelength range are desired, eliminating cryogenic requirements and leveraging existing optical fiber infrastructure for transmitting the quantum information. To that end, we report that telecom single photon emitters (SPEs) in gallium nitride (GaN) exhibit optically detected magnetic resonance (ODMR) at room temperature. The analysis of ODMR as a function of magnetic field orientation enables the determination of the orientation of the spin quantization axis with respect to the GaN crystalline lattice. The optical transitions dynamics are analyzed to gain further insight into the transition rates dominating ODMR. Our findings, coupled with GaN's mature fabrication technology, could facilitate the realization of scalable quantum technology.

Large Language Models (LLMs) are known for their remarkable ability to generate synthesized 'knowledge', such as text documents, music, images, etc. However, there is a huge gap between LLM's and human capabilities for understanding abstract concepts and reasoning. We discuss these issues in a larger philosophical context of human knowledge acquisition and the Turing test. In addition, we illustrate the limitations of LLMs by analyzing GPT-4 responses to questions ranging from science and math to common sense reasoning. These examples show that GPT-4 can often imitate human reasoning, even though it lacks understanding. However, LLM responses are synthesized from a large LLM model trained on all available data. In contrast, human understanding is based on a small number of abstract concepts. Based on this distinction, we discuss the impact of LLMs on acquisition of human knowledge and education.

In their study of the giant component in inhomogeneous random graphs, Bollob\'as, Janson, and Riordan introduced a class of branching processes parametrized by a possibly unbounded graphon. We prove that two such branching processes have the same distribution if and only if the corresponding graphons are fractionally isomorphic, a notion introduced by Greb\'ik and Rocha. A different class of branching processes was introduced by Hladk\'y, Nachmias, and Tran in relation to uniform spanning trees in finite graphs approximating a given connected graphon. We prove that two such branching processes have the same distribution if and only if the corresponding graphons are fractionally isomorphic up to scalar multiple. Combined with a recent result of Archer and Shalev, this implies that if uniform spanning trees of two dense graphs have a similar local structure, they have a similar scaling limit. As a side result we give a characterization of fractional isomorphism for graphs as well as graphons in terms of their connected components., Comment: 44 pages, 1 figure; minor changes compared to the previous version

Radar-based contactless cardiac monitoring has become a popular research direction recently, but the fine-grained electrocardiogram (ECG) signal is still hard to reconstruct from millimeter-wave radar signal. The key obstacle is to decouple the cardiac activities in the electrical domain (i.e., ECG) from that in the mechanical domain (i.e., heartbeat), and most existing research only uses pure data-driven methods to map such domain transformation as a black box. Therefore, this work first proposes a signal model for domain transformation, and then a novel deep learning framework called radarODE is designed to fuse the temporal and morphological features extracted from radar signals and generate ECG. In addition, ordinary differential equations are embedded in radarODE as a decoder to provide morphological prior, helping the convergence of the model training and improving the robustness under body movements. After being validated on the dataset, the proposed radarODE achieves better performance compared with the benchmark in terms of missed detection rate, root mean square error, Pearson correlation coefficient with the improvement of 9%, 16% and 19%, respectively. The validation results imply that radarODE is capable of recovering ECG signals from radar signals with high fidelity and can be potentially implemented in real-life scenarios.

Random forests are considered a cornerstone in machine learning for their robustness and versatility. Despite these strengths, their conventional centralized training is ill-suited for the modern landscape of data that is often distributed, sensitive, and subject to privacy concerns. Federated learning (FL) provides a compelling solution to this problem, enabling models to be trained across a group of clients while maintaining the privacy of each client's data. However, adapting tree-based methods like random forests to federated settings introduces significant challenges, particularly when it comes to non-identically distributed (non-IID) data across clients, which is a common scenario in real-world applications. This paper presents a federated random forest approach that employs a novel ensemble construction method aimed at improving performance under non-IID data. Instead of growing trees independently in each client, our approach ensures each decision tree in the ensemble is iteratively and collectively grown across clients. To preserve the privacy of the client's data, we confine the information stored in the leaf nodes to the majority class label identified from the samples of the client's local data that reach each node. This limited disclosure preserves the confidentiality of the underlying data distribution of clients, thereby enhancing the privacy of the federated learning process. Furthermore, our collaborative ensemble construction strategy allows the ensemble to better reflect the data's heterogeneity across different clients, enhancing its performance on non-IID data, as our experimental results confirm., Comment: This is the authors' accepted manuscript of an article published in the journal Expert Systems With Applications. Published version available at: https://www.sciencedirect.com/science/article/pii/S0957417424016099. 22 pages, 3 figures

The human brain has the capability to associate the unknown person's voice and face by leveraging their general relationship, referred to as ``cross-modal speaker verification''. This task poses significant challenges due to the complex relationship between the modalities. In this paper, we propose a ``Multi-stage Face-voice Association Learning with Keynote Speaker Diarization''~(MFV-KSD) framework. MFV-KSD contains a keynote speaker diarization front-end to effectively address the noisy speech inputs issue. To balance and enhance the intra-modal feature learning and inter-modal correlation understanding, MFV-KSD utilizes a novel three-stage training strategy. Our experimental results demonstrated robust performance, achieving the first rank in the 2024 Face-voice Association in Multilingual Environments (FAME) challenge with an overall Equal Error Rate (EER) of 19.9%. Details can be found in https://github.com/TaoRuijie/MFV-KSD.

Text-based talking-head video editing aims to efficiently insert, delete, and substitute segments of talking videos through a user-friendly text editing approach. It is challenging because of \textbf{1)} generalizable talking-face representation, \textbf{2)} seamless audio-visual transitions, and \textbf{3)} identity-preserved talking faces. Previous works either require minutes of talking-face video training data and expensive test-time optimization for customized talking video editing or directly generate a video sequence without considering in-context information, leading to a poor generalizable representation, or incoherent transitions, or even inconsistent identity. In this paper, we propose an efficient cascaded conditional diffusion-based framework, which consists of two stages: audio to dense-landmark motion and motion to video. \textit{\textbf{In the first stage}}, we first propose a dynamic weighted in-context diffusion module to synthesize dense-landmark motions given an edited audio. \textit{\textbf{In the second stage}}, we introduce a warping-guided conditional diffusion module. The module first interpolates between the start and end frames of the editing interval to generate smooth intermediate frames. Then, with the help of the audio-to-dense motion images, these intermediate frames are warped to obtain coarse intermediate frames. Conditioned on the warped intermedia frames, a diffusion model is adopted to generate detailed and high-resolution target frames, which guarantees coherent and identity-preserved transitions. The cascaded conditional diffusion model decomposes the complex talking editing task into two flexible generation tasks, which provides a generalizable talking-face representation, seamless audio-visual transitions, and identity-preserved faces on a small dataset. Experiments show the effectiveness and superiority of the proposed method.

In the rapidly evolving field of 3D reconstruction, 3D Gaussian Splatting (3DGS) and 2D Gaussian Splatting (2DGS) represent significant advancements. Although 2DGS compresses 3D Gaussian primitives into 2D Gaussian surfels to effectively enhance mesh extraction quality, this compression can potentially lead to a decrease in rendering quality. Additionally, unreliable densification processes and the calculation of depth through the accumulation of opacity can compromise the detail of mesh extraction. To address this issue, we introduce MVG-Splatting, a solution guided by Multi-View considerations. Specifically, we integrate an optimized method for calculating normals, which, combined with image gradients, helps rectify inconsistencies in the original depth computations. Additionally, utilizing projection strategies akin to those in Multi-View Stereo (MVS), we propose an adaptive quantile-based method that dynamically determines the level of additional densification guided by depth maps, from coarse to fine detail. Experimental evidence demonstrates that our method not only resolves the issues of rendering quality degradation caused by depth discrepancies but also facilitates direct mesh extraction from dense Gaussian point clouds using the Marching Cubes algorithm. This approach significantly enhances the overall fidelity and accuracy of the 3D reconstruction process, ensuring that both the geometric details and visual quality., Comment: https://mvgsplatting.github.io

Continual Learning (CL) involves fine-tuning pre-trained models with new data while maintaining the performance on the pre-trained data. This is particularly relevant for expanding multilingual ASR (MASR) capabilities. However, existing CL methods, mainly designed for computer vision and reinforcement learning tasks, often yield sub-optimal results when directly applied to MASR. We hypothesise that this is because CL of the auto-regressive decoder in the MASR model is difficult. To verify this, we propose four optimizations on the decoder. They include decoder-layer gradient surgery, freezing unused token embeddings, suppressing output of newly added tokens, and learning rate re-scaling. Our experiments on adapting Whisper to 10 unseen languages from the Common Voice dataset demonstrate that these optimizations reduce the Average Word Error Rate (AWER) of pretrained languages from 14.2% to 12.4% compared with Experience Replay, without compromising the AWER of new languages., Comment: Proceedings of Interspeech

In this paper, we propose reverse inference optimization (RIO), a simple and effective method designed to enhance the robustness of autoregressive-model-based zero-shot text-to-speech (TTS) systems using reinforcement learning from human feedback (RLHF). To assess the quality of speech produced by the TTS system without human annotations, RIO introduces a novel concept termed as reverse inference based on the Bayesian principle, which suggests that a high-quality generated speech should be able to be used as a prompt for subsequent generation using the same TTS model. By leveraging reverse inference as the standard to select exemplars used in RLHF from the speech samples generated by the TTS system itself, RIO steers the subsequent optimization towards a direction of enhancing the TTS robustness. The RIO framework, comprising sampling, automatic annotating, and learning, obviates the need for a reward model or pairwise preference data, and significantly improves the stability of zero-shot TTS performance by reducing the discrepancies between training and inference conditions. Our experimental results verify that RIO can effectively improve both subjective and objective metrics, including mean opinion scores, word error rates, and speaker similarity. Remarkably, RIO can also diminish the incidence of bad outputs to nearly zero percent, rivalling the robustness when using ground-truth speech as the prompt., Comment: 12 pages, Work in progress

Recent synthetic speech detectors leveraging the Transformer model have superior performance compared to the convolutional neural network counterparts. This improvement could be due to the powerful modeling ability of the multi-head self-attention (MHSA) in the Transformer model, which learns the temporal relationship of each input token. However, artifacts of synthetic speech can be located in specific regions of both frequency channels and temporal segments, while MHSA neglects this temporal-channel dependency of the input sequence. In this work, we proposed a Temporal-Channel Modeling (TCM) module to enhance MHSA's capability for capturing temporal-channel dependencies. Experimental results on the ASVspoof 2021 show that with only 0.03M additional parameters, the TCM module can outperform the state-of-the-art system by 9.25% in EER. Further ablation study reveals that utilizing both temporal and channel information yields the most improvement for detecting synthetic speech., Comment: Accepted by INTERSPEECH 2024

Recent improvements in neural audio codec (NAC) models have generated interest in adopting pre-trained codecs for a variety of speech processing applications to take advantage of the efficiencies gained from high compression, but these have yet been applied to the speech separation (SS) task. SS can benefit from high compression because the compute required for traditional SS models makes them impractical for many edge computing use cases. However, SS is a waveform-masking task where compression tends to introduce distortions that severely impact performance. Here we propose a novel task of Audio Codec-based SS, where SS is performed within the embedding space of a NAC, and propose a new model, Codecformer, to address this task. At inference, Codecformer achieves a 52x reduction in MAC while producing separation performance comparable to a cloud deployment of Sepformer. This method charts a new direction for performing efficient SS in practical scenarios., Comment: This paper was accepted by Interspeech 2024

Ferroelectric materials play a crucial role in a broad range of technologies due to their unique properties that are deeply connected to the pattern and behavior of their ferroelectric (FE) domains. Chief among them, barium titanate (BaTiO$_3$; BTO) sees widespread applications such as in electronics but equally is a ferroelectric model system for fundamental research, e.g., to study the interplay of such FE domains, the domain walls (DWs), and their macroscopic properties, owed to BTO's multiple and experimentally accessible phase transitions. Here, we employ Second Harmonic Generation Microscopy (SHGM) to $\it{in-situ}$ investigate the cubic-to-tetragonal (at $\sim$126$^\circ$C) and the tetragonal-to-orthorhombic (at $\sim$5$^\circ$C) phase transition in single-crystalline BTO via 3-dimensional (3D) DW mapping. We demonstrate that SHGM imaging provides the direct visualization of FE domain switching as well as the domain dynamics in 3D, shedding light on the interplay of the domain structure and the phase transition. These results allow us to extract the different transition temperatures locally, to unveil the hysteresis behavior, and to determine the type of phase transition at play (1st/2nd order) from the recorded SHGM data. The capabilities of SHGM in uncovering these crucial phenomena can easily be applied to other ferroelectrics to provide new possibilities for $\it{in-situ}$ engineering of advanced ferroic devices., Comment: 8 pages, 3 figures, with appended supplementary material (2 pages, 2 figures)

Deep learning models for speech rely on large datasets, presenting computational challenges. Yet, performance hinges on training data size. Dataset Distillation (DD) aims to learn a smaller dataset without much performance degradation when training with it. DD has been investigated in computer vision but not yet in speech. This paper presents the first approach for DD to speech targeting Speech Emotion Recognition on IEMOCAP. We employ Generative Adversarial Networks (GANs) not to mimic real data but to distil key discriminative information of IEMOCAP that is useful for downstream training. The GAN then replaces the original dataset and can sample custom synthetic dataset sizes. It performs comparably when following the original class imbalance but improves performance by 0.3% absolute UAR with balanced classes. It also reduces dataset storage and accelerates downstream training by 95% in both cases and reduces speaker information which could help for a privacy application., Comment: Accepted at Interspeech 2024

In recent years, text-to-speech (TTS) technology has witnessed impressive advancements, particularly with large-scale training datasets, showcasing human-level speech quality and impressive zero-shot capabilities on unseen speakers. However, despite human subjective evaluations, such as the mean opinion score (MOS), remaining the gold standard for assessing the quality of synthetic speech, even state-of-the-art TTS approaches have kept human feedback isolated from training that resulted in mismatched training objectives and evaluation metrics. In this work, we investigate a novel topic of integrating subjective human evaluation into the TTS training loop. Inspired by the recent success of reinforcement learning from human feedback, we propose a comprehensive sampling-annotating-learning framework tailored to TTS optimization, namely uncertainty-aware optimization (UNO). Specifically, UNO eliminates the need for a reward model or preference data by directly maximizing the utility of speech generations while considering the uncertainty that lies in the inherent variability in subjective human speech perception and evaluations. Experimental results of both subjective and objective evaluations demonstrate that UNO considerably improves the zero-shot performance of TTS models in terms of MOS, word error rate, and speaker similarity. Additionally, we present a remarkable ability of UNO that it can adapt to the desired speaking style in emotional TTS seamlessly and flexibly., Comment: 19 pages, Preprint

The ethical integration of Artificial Intelligence (AI) in healthcare necessitates addressing fairness-a concept that is highly context-specific across medical fields. Extensive studies have been conducted to expand the technical components of AI fairness, while tremendous calls for AI fairness have been raised from healthcare. Despite this, a significant disconnect persists between technical advancements and their practical clinical applications, resulting in a lack of contextualized discussion of AI fairness in clinical settings. Through a detailed evidence gap analysis, our review systematically pinpoints several deficiencies concerning both healthcare data and the provided AI fairness solutions. We highlight the scarcity of research on AI fairness in many medical domains where AI technology is increasingly utilized. Additionally, our analysis highlights a substantial reliance on group fairness, aiming to ensure equality among demographic groups from a macro healthcare system perspective; in contrast, individual fairness, focusing on equity at a more granular level, is frequently overlooked. To bridge these gaps, our review advances actionable strategies for both the healthcare and AI research communities. Beyond applying existing AI fairness methods in healthcare, we further emphasize the importance of involving healthcare professionals to refine AI fairness concepts and methods to ensure contextually relevant and ethically sound AI applications in healthcare.

We propose an unsupervised adaptation framework, Self-TAught Recognizer (STAR), which leverages unlabeled data to enhance the robustness of automatic speech recognition (ASR) systems in diverse target domains, such as noise and accents. STAR is developed for prevalent speech foundation models based on Transformer-related architecture with auto-regressive decoding (e.g., Whisper, Canary). Specifically, we propose a novel indicator that empirically integrates step-wise information during decoding to assess the token-level quality of pseudo labels without ground truth, thereby guiding model updates for effective unsupervised adaptation. Experimental results show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains, and it sometimes even approaches the upper-bound performance of supervised adaptation. Surprisingly, we also observe that STAR prevents the adapted model from the common catastrophic forgetting problem without recalling source-domain data. Furthermore, STAR exhibits high data efficiency that only requires less than one-hour unlabeled data, and seamless generality to alternative large speech models and speech translation tasks. Our code aims to open source to the research communities., Comment: 23 pages, Preprint

Embodied perception is essential for intelligent vehicles and robots in interactive environmental understanding. However, these advancements primarily focus on vision, with limited attention given to using 3D modeling sensors, restricting a comprehensive understanding of objects in response to prompts containing qualitative and quantitative queries. Recently, as a promising automotive sensor with affordable cost, 4D millimeter-wave radars provide denser point clouds than conventional radars and perceive both semantic and physical characteristics of objects, thereby enhancing the reliability of perception systems. To foster the development of natural language-driven context understanding in radar scenes for 3D visual grounding, we construct the first dataset, Talk2Radar, which bridges these two modalities for 3D Referring Expression Comprehension (REC). Talk2Radar contains 8,682 referring prompt samples with 20,558 referred objects. Moreover, we propose a novel model, T-RadarNet, for 3D REC on point clouds, achieving State-Of-The-Art (SOTA) performance on the Talk2Radar dataset compared to counterparts. Deformable-FPN and Gated Graph Fusion are meticulously designed for efficient point cloud feature modeling and cross-modal fusion between radar and text features, respectively. Comprehensive experiments provide deep insights into radar-based 3D REC. We release our project at https://github.com/GuanRunwei/Talk2Radar., Comment: 8 pages, 5 figures

Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR), which aims to predict the ground-truth transcription from the decoded N-best hypotheses. Thanks to the strong language generation ability of LLMs and rich information in the N-best list, GER shows great effectiveness in enhancing ASR results. However, it still suffers from two limitations: 1) LLMs are unaware of the source speech during GER, which may lead to results that are grammatically correct but violate the source speech content, 2) N-best hypotheses usually only vary in a few tokens, making it redundant to send all of them for GER, which could confuse LLM about which tokens to focus on and thus lead to increased miscorrection. In this paper, we propose ClozeGER, a new paradigm for ASR generative error correction. First, we introduce a multimodal LLM (i.e., SpeechGPT) to receive source speech as extra input to improve the fidelity of correction output. Then, we reformat GER as a cloze test with logits calibration to remove the input information redundancy and simplify GER with clear instructions. Experiments show that ClozeGER achieves a new breakthrough over vanilla GER on 9 popular ASR datasets., Comment: 14 pages, Accepted by ACL 2024

Ferroelectric domain walls (DWs) are promising structures for assembling future nano-electronic circuit elements on a larger scale, since reporting domain wall currents of up to 1 mA per single DW. One key requirement hereto is their reproducible manufacturing by gaining preparative control over domain size and domain wall conductivity (DWC). To date, most works on DWC have focused on exploring the fundamental electrical properties of individual DWs within single shot experiments, with emphasis on quantifying the origins for DWC. Very few reports exist when it comes to compare the DWC properties between two separate DWs, and literally nothing exists where issues of reproducibility in DWC devices have been addressed. To fill this gap while facing the challenge of finding guidelines achieving predictable DWC performance, we report on a procedure that allows us to reproducibly prepare single hexagonal domains of a predefined diameter into uniaxial ferroelectric (FE) lithium niobate (LN) single crystals of 200 and 300 micrometers thickness, respectively. We show that the domain diameter can be controlled with an error of a few percent. As-grown DWs are then subjected to a standard procedure of current-controlled high-voltage DWC enhancement, repetitively reaching a DWC increase of 6 orders of magnitude. While all resulting DWs show significantly enhanced DWC values, subtle features in their individual current-voltage (I-V) characteristics hint towards different 3D shapes into the bulk, with variations probably reflecting local heterogeneities by defects, DW pinning, and surface-near DW inclination, which seem to have a larger impact than expected., Comment: 12 pages, 7 figures, with appended supplement (9 pages, 7 figures)

A permanently available molecular-beam injection setup for controlled molecules (COMO) was installed and commissioned at the small quantum systems (SQS) instrument at the European x-ray free-electron laser (EuXFEL). A $b$-type electrostatic deflector allows for pure state-, size-, and isomer-selected samples of polar molecules and clusters. The source provides a rotationally cold ($T\approx1$~K) and dense ($\rho\approx10^8$~cm$^{-3}$) molecular beam with pulse durations up to 100~\us generated by a new version of the Even-Lavie valve. Here, a performance overview of the COMO setup is presented along with characterization experiments performed both, with an optical laser at the Center for Free-Electron-Laser Science and with x-rays at EuXFEL under burst-mode operation. COMO was designed to be attached to different instruments at the EuXFEL, in particular at the small quantum systems (SQS) and single particles, clusters, and biomolecules (SPB) instruments. This advanced controlled-molecules injection setup enables XFEL studies using highly defined samples with soft and hard x-ray FEL radiation for applications ranging from atomic, molecular, and cluster physics to elementary processes in chemistry and biology.

We study high quality-factor (high Q) resonances supported by periodic arrays of Mie resonators from the perspectives of both Bloch wave theory and multiple scattering theory. We reveal that, unlike a common belief, the bound states in the continuum (BICs) derived by the Bloch-wave theory do not directly determine the resonance with the highest Q value in large but finite arrays. Higher Q factors appear to be associated with collective resonances formed by nominally guided modes below the light line associated with strong effect of both electric and magnetic multipoles. Our findings offer valuable insights into accessing the modes with higher Q resonances via bonding modes within finite metastructures. Our results underpin the pivotal significance of magnetic and electric multipoles in the design of resonant metadevices and nonlocal flat-band optics. Moreover, our demonstrations reveal that coupled arrays of high-Q microcavities do not inherently result in a stronger light-matter interaction when compared to coupled low-Q nanoresonators. This result emphasizes the critical importance of the study of multiple light-scattering effects in cavity-based systems., Comment: 13 pages; 5 figures; An unified picture of BICs