Search

Recent advances in speech spoofing necessitate stronger verification mechanisms in neural speech codecs to ensure authenticity. Current methods embed numerical watermarks before compression and extract them from reconstructed speech for verification, but face limitations such as separate training processes for the watermark and codec, and insufficient cross-modal information integration, leading to reduced watermark imperceptibility, extraction accuracy, and capacity. To address these issues, we propose WMCodec, the first neural speech codec to jointly train compression-reconstruction and watermark embedding-extraction in an end-to-end manner, optimizing both imperceptibility and extractability of the watermark. Furthermore, We design an iterative Attention Imprint Unit (AIU) for deeper feature integration of watermark and speech, reducing the impact of quantization noise on the watermark. Experimental results show WMCodec outperforms AudioSeal with Encodec in most quality metrics for watermark imperceptibility and consistently exceeds both AudioSeal with Encodec and reinforced TraceableSpeech in extraction accuracy of watermark. At bandwidth of 6 kbps with a watermark capacity of 16 bps, WMCodec maintains over 99% extraction accuracy under common attacks, demonstrating strong robustness.

The growing prominence of the field of audio deepfake detection is driven by its wide range of applications, notably in protecting the public from potential fraud and other malicious activities, prompting the need for greater attention and research in this area. The ADD 2023 challenge goes beyond binary real/fake classification by emulating real-world scenarios, such as the identification of manipulated intervals in partially fake audio and determining the source responsible for generating any fake audio, both with real-life implications, notably in audio forensics, law enforcement, and construction of reliable and trustworthy evidence. To further foster research in this area, in this article, we describe the dataset that was used in the fake game, manipulation region location and deepfake algorithm recognition tracks of the challenge. We also focus on the analysis of the technical methodologies by the top-performing participants in each task and note the commonalities and differences in their approaches. Finally, we discuss the current technical limitations as identified through the technical analysis, and provide a roadmap for future research directions. The dataset is available for download at http://addchallenge.cn/downloadADD2023., Comment: This work has been submitted to the IEEE for possible publication

Various threats posed by the progress in text-to-speech (TTS) have prompted the need to reliably trace synthesized speech. However, contemporary approaches to this task involve adding watermarks to the audio separately after generation, a process that hurts both speech quality and watermark imperceptibility. In addition, these approaches are limited in robustness and flexibility. To address these problems, we propose TraceableSpeech, a novel TTS model that directly generates watermarked speech, improving watermark imperceptibility and speech quality. Furthermore, We design the frame-wise imprinting and extraction of watermarks, achieving higher robustness against resplicing attacks and temporal flexibility in operation. Experimental results show that TraceableSpeech outperforms the strong baseline where VALL-E or HiFicodec individually uses WavMark in watermark imperceptibility, speech quality and resilience against resplicing attacks. It also can apply to speech of various durations. The code is avaliable at https://github.com/zjzser/TraceableSpeech, Comment: acceped by interspeech 2024

Recent strides in neural speech synthesis technologies, while enjoying widespread applications, have nonetheless introduced a series of challenges, spurring interest in the defence against the threat of misuse and abuse. Notably, source attribution of synthesized speech has value in forensics and intellectual property protection, but prior work in this area has certain limitations in scope. To address the gaps, we present our findings concerning the identification of the sources of synthesized speech in this paper. We investigate the existence of speech synthesis model fingerprints in the generated speech waveforms, with a focus on the acoustic model and the vocoder, and study the influence of each component on the fingerprint in the overall speech waveforms. Our research, conducted using the multi-speaker LibriTTS dataset, demonstrates two key insights: (1) vocoders and acoustic models impart distinct, model-specific fingerprints on the waveforms they generate, and (2) vocoder fingerprints are the more dominant of the two, and may mask the fingerprints from the acoustic model. These findings strongly suggest the existence of model-specific fingerprints for both the acoustic model and the vocoder, highlighting their potential utility in source identification applications., Comment: Accepted by CCL 2024

Audio deepfake detection is an emerging active topic. A growing number of literatures have aimed to study deepfake detection algorithms and achieved effective performance, the problem of which is far from being solved. Although there are some review literatures, there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences across various types of deepfake audio, then outline and analyse competitions, datasets, features, classifications, and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are discussed. In addition, we perform a unified comparison of representative features and classifiers on ASVspoof 2021, ADD 2023 and In-the-Wild datasets for audio deepfake detection, respectively. The survey shows that future research should address the lack of large scale datasets in the wild, poor generalization of existing detection methods to unknown fake attacks, as well as interpretability of detection results.

Audio deepfake detection is an emerging topic in the artificial intelligence community. The second Audio Deepfake Detection Challenge (ADD 2023) aims to spur researchers around the world to build new innovative technologies that can further accelerate and foster research on detecting and analyzing deepfake speech utterances. Different from previous challenges (e.g. ADD 2022), ADD 2023 focuses on surpassing the constraints of binary real/fake classification, and actually localizing the manipulated intervals in a partially fake speech as well as pinpointing the source responsible for generating any fake audio. Furthermore, ADD 2023 includes more rounds of evaluation for the fake audio game sub-challenge. The ADD 2023 challenge includes three subchallenges: audio fake game (FG), manipulation region location (RL) and deepfake algorithm recognition (AR). This paper describes the datasets, evaluation metrics, and protocols. Some findings are also reported in audio deepfake detection tasks.

Text-based speech editing allows users to edit speech by intuitively cutting, copying, and pasting text to speed up the process of editing speech. In the previous work, CampNet (context-aware mask prediction network) is proposed to realize text-based speech editing, significantly improving the quality of edited speech. This paper aims at a new task: adding emotional effect to the editing speech during the text-based speech editing to make the generated speech more expressive. To achieve this task, we propose Emo-CampNet (emotion CampNet), which can provide the option of emotional attributes for the generated speech in text-based speech editing and has the one-shot ability to edit unseen speakers' speech. Firstly, we propose an end-to-end emotion-selectable text-based speech editing model. The key idea of the model is to control the emotion of generated speech by introducing additional emotion attributes based on the context-aware mask prediction network. Secondly, to prevent the emotion of the generated speech from being interfered by the emotional components in the original speech, a neutral content generator is proposed to remove the emotion from the original speech, which is optimized by the generative adversarial framework. Thirdly, two data augmentation methods are proposed to enrich the emotional and pronunciation information in the training set, which can enable the model to edit the unseen speaker's speech. The experimental results that 1) Emo-CampNet can effectively control the emotion of the generated speech in the process of text-based speech editing; And can edit unseen speakers' speech. 2) Detailed ablation experiments further prove the effectiveness of emotional selectivity and data augmentation methods. The demo page is available at https://hairuo55.github.io/Emo-CampNet/, Comment: Under review, 12 pages, 11 figures, demo page is available at https://hairuo55.github.io/Emo-CampNet/

Many datasets have been designed to further the development of fake audio detection. However, fake utterances in previous datasets are mostly generated by altering timbre, prosody, linguistic content or channel noise of original audio. These datasets leave out a scenario, in which the acoustic scene of an original audio is manipulated with a forged one. It will pose a major threat to our society if some people misuse the manipulated audio with malicious purpose. Therefore, this motivates us to fill in the gap. This paper proposes such a dataset for scene fake audio detection named SceneFake, where a manipulated audio is generated by only tampering with the acoustic scene of an real utterance by using speech enhancement technologies. Some scene fake audio detection benchmark results on the SceneFake dataset are reported in this paper. In addition, an analysis of fake attacks with different speech enhancement technologies and signal-to-noise ratios are presented in this paper. The results indicate that scene fake utterances cannot be reliably detected by baseline models trained on the ASVspoof 2019 dataset. Although these models perform well on the SceneFake training set and seen testing set, their performance is poor on the unseen test set. The dataset (https://zenodo.org/record/7663324#.Y_XKMuPYuUk) and benchmark source codes (https://github.com/ADDchallenge/SceneFake) are publicly available., Comment: Accepted by Pattern Recognition, 1 April 2024

Energetic X-ray radiations emitted from various accretion systems are widely considered to be produced by Comptonization in the hot corona. The corona and its interaction with the disc play an essential role in the evolution of the system and are potentially responsible for many observed features. However many intrinsic properties of the corona are still poorly understood, especially for the geometrical configurations. The traditional spectral fitting method is not powerful enough to distinguish various configurations. In this paper we intent to investigate the possible configurations by modeling the polarization properties of X-ray radiations. The geometries of the corona include the slab, sphere and cylinder. The simulations are implemented through the publicly available code, LEMON, which can deal with the polarized radiative transfer and different electron distributions readily. The results demonstrate clearly that the observed polarizations are dependent on the geometry of the corona heavily. The slab-like corona produces the highest polarization degrees, the following are the cylinder and sphere. One of the interesting things is that the polarization degrees first increase gradually and then decrease with the increase of photon energy. For slab geometry there exists a zero point where the polarization vanishes and the polarization angle rotates for $90^\circ$. These results may potentially be verified by the upcoming missions for polarized X-ray observations, such as $IXPE$ and $eXTP$., Comment: 29 pages, 12 figures, RAA accepted. The source code can be downloaded from \url{https://bitbucket.org/yangxiaolinsc/corona_geometry/src/main/}

In this paper, we proposed a new Monte Carlo radiative transport (MCRT) scheme, which is based completely on the Neumann series solution of Fredholm integral equation. This scheme indicates that the essence of MCRT is the calculation of infinite terms of multiple integrals in Neumann solution simultaneously. Under this perspective we redescribed MCRT procedure systematically, in which the main work amounts to choose an associated probability distribution function (PDF) for a set of random variables and the corresponding unbiased estimation functions. We can select a relatively optimal estimation procedure that has a lower variance from an infinite possible choices, such as the term by term estimation. In this scheme, MCRT can be regarded as a pure problem of integral evaluation, rather than as the tracing of random walking photons. Keeping this in mind, one can avert some subtle intuitive mistakes. In addition the $\delta$-functions in these integrals can be eliminated in advance by integrating them out directly. This fact together with the optimal chosen random variables can remarkably improve the Monte Carlo (MC) computational efficiency and accuracy, especially in systems with axial or spherical symmetry. An MCRT code, Lemon (Linear Integral Equations' Monte Carlo Solver Based on the Neumann solution), has been developed completely based on this scheme. Finally, we intend to verify the validation of Lemon, a suite of test problems mainly restricted to flat spacetime have been reproduced and the corresponding results are illustrated in detail., Comment: 37 pages, 28 figures. The code can be download from: https://github.com/yangxiaolinyn/Lemon (or https://bitbucket.org/yangxiaolinsc/lemonsourcecode/src/main/) and https://doi.org/10.5281/zenodo.4686355. Comments are welcome

Bound states in the continuum (BICs) have attracted much attention in recent years due to the infinite quality factor (Q-factor) resonance and extremely localized field. In this study, BICs have been demonstrated by dielectric metasurfaces with hybrid surface lattice resonance (SLR) in the experiment. By breaking the symmetry of geometry, SLR can be easily switched between BICs and quasi-BICs. Comparing with literature, switching between BICs and quasi-BICs is usually accompanied by wavelength shift. Here, a design rule is proposed to prevent the wavelength shift when the Q-factor is changing. Also, such a design also makes subsequent identification of the laser threshold more credible. Due to the high Q-factor, low threshold laser is one of the intuitive applications of BICs. Utilize the high localized ability of BICs, low threshold BICs laser can be achieved by the dielectric metasurface immersed with Rhodamine 6G. Interestingly, due to the high Q-factor resonance of BICs, the laser signals and images can be observed in almost transparent samples. Not only the BICs laser is demonstrated in the experiment, but also the mechanism of BICs is deeply analyzed. This study can help readers better understand this novel feature of BICs, and provide the way for engineer BICs metasurfaces. The device can provide various applications, including laser, optical sensing, non-linear optics enhancement, and single-photon source., Comment: 18 pages, 4 figures