Search

We propose FlowDec, a neural full-band audio codec for general audio sampled at 48 kHz that combines non-adversarial codec training with a stochastic postfilter based on a novel conditional flow matching method. Compared to the prior work ScoreDec which is based on score matching, we generalize from speech to general audio and move from 24 kbit/s to as low as 4 kbit/s, while improving output quality and reducing the required postfilter DNN evaluations from 60 to 6 without any fine-tuning or distillation techniques. We provide theoretical insights and geometric intuitions for our approach in comparison to ScoreDec as well as another recent work that uses flow matching, and conduct ablation studies on our proposed components. We show that FlowDec is a competitive alternative to the recent GAN-dominated stream of neural codecs, achieving FAD scores better than those of the established GAN-based codec DAC and listening test scores that are on par, and producing qualitatively more natural reconstructions for speech and harmonic structures in music., Comment: Accepted at ICLR 2025

Finding Lyapunov functions to certify the stability of control systems has been an important topic for verifying safety-critical systems. Most existing methods on finding Lyapunov functions require access to the dynamics of the system. Accurately describing the complete dynamics of a control system however remains highly challenging in practice. Latest trend of using learning-enabled control systems further reduces the transparency. Hence, a method for black-box systems would have much wider applications. Our work stems from the recent idea of sampling and exploiting Lipschitz continuity to approximate the unknown dynamics. Given Lipschitz constants, one can derive a non-statistical upper bounds on approximation errors; hence a strong certification on this approximation can certify the unknown dynamics. We significantly improve this idea by directly approximating the Lie derivative of Lyapunov functions instead of the dynamics. We propose a framework based on the learner-verifier architecture from Counterexample-Guided Inductive Synthesis (CEGIS). Our insight of combining regional verification conditions and counterexample-guided sampling enables a guided search for samples to prove stability region-by-region. Our CEGIS algorithm further ensures termination. Our numerical experiments suggest that it is possible to prove the stability of 2D and 3D systems with a few thousands of samples. Our visualization also reveals the regions where the stability is difficult to prove. In comparison with the existing black-box approach, our approach at the best case requires less than 0.01% of samples., Comment: 30 pages, 3 figures. This is the extended version of the same paper accepted by the International Conference on Hybrid Systems: Computation and Control (HSCC) 2025

On-demand entangled photon pairs at telecom wavelengths are crucial for quantum communication, distributed quantum computing, and quantum-enhanced sensing and metrology. The O-band is particularly advantageous because of its minimal chromatic dispersion and transmission loss in optical fibers, making it well-suited for long-distance quantum networks. Site-controlled nanowire quantum dots have emerged as a promising platform for the on-demand generation of single and entangled photons, offering high extraction efficiency and the potential for scalable fabrication in large uniform arrays. However, their operation has been largely restricted to the visible and first near-infrared (NIR-I) windows. Here, we demonstrate an on-demand bright source of entangled photon pairs with high fidelity in the telecom O-band based on site-controlled nanowire quantum dots. We measure a fine-structure splitting of 4.6 $\mu$eV, verifying the suitability of the quantum dot for generating high-fidelity polarization-entangled photon pairs. Full quantum state tomography of the two-photon state generated by the biexciton\hyph exciton cascade reveals a maximum fidelity of $85.8\% \pm 1.1\%$ to the $\Phi^+$ Bell state, and a maximum concurrence of $75.1\% \pm 2.1\%$. We estimate the source efficiency at the first lens to be 12.5$\%$. This bright, scalable, and deterministic source of entangled photons in the telecom range represents a valuable step forward in advancing practical quantum applications at telecom wavelengths.

The quantification of audio aesthetics remains a complex challenge in audio processing, primarily due to its subjective nature, which is influenced by human perception and cultural context. Traditional methods often depend on human listeners for evaluation, leading to inconsistencies and high resource demands. This paper addresses the growing need for automated systems capable of predicting audio aesthetics without human intervention. Such systems are crucial for applications like data filtering, pseudo-labeling large datasets, and evaluating generative audio models, especially as these models become more sophisticated. In this work, we introduce a novel approach to audio aesthetic evaluation by proposing new annotation guidelines that decompose human listening perspectives into four distinct axes. We develop and train no-reference, per-item prediction models that offer a more nuanced assessment of audio quality. Our models are evaluated against human mean opinion scores (MOS) and existing methods, demonstrating comparable or superior performance. This research not only advances the field of audio aesthetics but also provides open-source models and datasets to facilitate future work and benchmarking. We release our code and pre-trained model at: https://github.com/facebookresearch/audiobox-aesthetics, Comment: Repository: https://github.com/facebookresearch/audiobox-aesthetics Website: https://ai.meta.com/research/publications/meta-audiobox-aesthetics-unified-automatic-quality-assessment-for-speech-music-and-sound/

Neural audio codecs have been widely adopted in audio-generative tasks because their compact and discrete representations are suitable for both large-language-model-style and regression-based generative models. However, most neural codecs struggle to model out-of-domain audio, resulting in error propagations to downstream generative tasks. In this paper, we first argue that information loss from codec compression degrades out-of-domain robustness. Then, we propose full-band 48~kHz ComplexDec with complex spectral input and output to ease the information loss while adopting the same 24~kbps bitrate as the baseline AuidoDec and ScoreDec. Objective and subjective evaluations demonstrate the out-of-domain robustness of ComplexDec trained using only the 30-hour VCTK corpus., Comment: 5 pages, 2 figures, 2 tables. Proc. ICASSP, 2025

This study addresses the issue of leveraging federated learning to improve data privacy and performance in IVF embryo selection. The EM (Expectation-Maximization) algorithm is incorporated into deep learning models to form a federated learning framework for quality evaluation of blastomere cleavage using two-dimensional images. The framework comprises a server site and several client sites characterized in that each is locally trained with an EM algorithm. Upon the completion of the local EM training, a separate 5-mode mixture distribution is generated for each client, the clients' distribution statics are then uploaded to the server site and aggregated therein to produce a global (sharing) 5-mode distribution. During the inference phase, each client uses image classifiers and an instance segmentor, assisted by the global 5-mode distribution acting as a calibrator to (1) identify the absolute cleavage timing of blastomere, i.e., tPNa, tPNf, t2, t3, t4, t5, t6, t7, and t8, (2) track the cleavage process of blastomeres to detect the irregular cleavage patterns, and (3) assess the symmetry degree of blastomeres. Experimental results show that the proposed method outperforms commercial Time-Lapse Incubators in reducing the average error of timing prediction by twofold. The proposed facilitate frameworks the adaptability and scalability of classifiers and segmentor to data variability associated with patients in different locations or countries., Comment: 6 pages, 8 figures. Accepted by the 1st Workshop on Federated Learning for Unbounded and Intelligent Decentralization (FLUID). in conjunction with AAAI 2025 in Philadelphia, Pennsylvania, USA

Inverting visual representations within deep neural networks (DNNs) presents a challenging and important problem in the field of security and privacy for deep learning. The main goal is to invert the features of an unidentified target image generated by a pre-trained DNN, aiming to reconstruct the original image. Feature inversion holds particular significance in understanding the privacy leakage inherent in contemporary split DNN execution techniques, as well as in various applications based on the extracted DNN features. In this paper, we explore the use of diffusion models, a promising technique for image synthesis, to enhance feature inversion quality. We also investigate the potential of incorporating alternative forms of prior knowledge, such as textual prompts and cross-frame temporal correlations, to further improve the quality of inverted features. Our findings reveal that diffusion models can effectively leverage hidden information from the DNN features, resulting in superior reconstruction performance compared to previous methods. This research offers valuable insights into how diffusion models can enhance privacy and security within applications that are reliant on DNN features.

We present experimental results on tuning biphoton frequency by introducing a detuned coupling field in spontaneous four-wave mixing (SFWM), and examine its impact on the pairing ratio. This tunability is achieved by manipulating the inherent electromagnetically induced transparency (EIT) effect in the double-$\Lambda$ scheme. Introducing a detuned coupling field degrades the efficiency of EIT-based stimulated four-wave mixing, which in turn reduces the biphoton pairing ratio. However, this reduction can be mitigated by increasing the optical power of the coupling field. Additionally, we observe that blue- and red-detuning the biphoton frequency results in distinct temporal profiles of biphoton wavepackets due to phase mismatch. These findings provide insights into the mechanisms of frequency-tunable biphoton generation via SFWM, and suggest potential optimizations for applications in quantum communication and information processing.

Contacts between particles in dense, sheared suspensions are believed to underpin much of their rheology. Roughness and adhesion are known to constrain the relative motion of particles, and thus globally affect the shear response, but an experimental description of how they microscopically influence the transmission of forces and relative displacements within contacts is lacking. Here we show that an innovative colloidal-probe atomic force microscopy technique allows the simultaneous measurement of normal and tangential forces exchanged between tailored surfaces and microparticles while tracking their relative sliding and rolling, unlocking the direct measurement of coefficients of rolling friction, as well as of sliding friction. We demonstrate that, in the presence of sufficient traction, particles spontaneously roll, reducing dissipation and promoting longer-lasting contacts. Conversely, when rolling is prevented, friction is greatly enhanced for rough and adhesive surfaces, while smooth particles coated by polymer brushes maintain well-lubricated contacts. We find that surface roughness induces rolling due to load-dependent asperity interlocking, leading to large off-axis particle rotations. In contrast, smooth, adhesive surfaces promote rolling along the principal axis of motion. Our results offer direct values of friction coefficients for numerical studies and an interpretation of the onset of discontinuous shear thickening based on them, opening up new ways to tailor rheology via contact engineering., Comment: 7 Figures

We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface style changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x larger) via novel knowledge distillation and personal adaptation techniques. We integrate knowledge distillation with a masked autoencoder, developing a compact yet powerful gaze estimation model. This model is further fine-tuned with Adapters, enabling highly accurate and personalized gaze predictions with minimal user input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low latency on the edge device (Raspberry Pi 4). Furthermore, we describe applications of GazeGen, illustrating its versatility and effectiveness in various usage scenarios., Comment: 12 pages, 10 figures

Multi-object 3D Grounding involves locating 3D boxes based on a given query phrase from a point cloud. It is a challenging and significant task with numerous applications in visual understanding, human-computer interaction, and robotics. To tackle this challenge, we introduce D-LISA, a two-stage approach incorporating three innovations. First, a dynamic vision module that enables a variable and learnable number of box proposals. Second, a dynamic camera positioning that extracts features for each proposal. Third, a language-informed spatial attention module that better reasons over the proposals to output the final prediction. Empirically, experiments show that our method outperforms the state-of-the-art methods on multi-object 3D grounding by 12.8% (absolute) and is competitive in single-object 3D grounding., Comment: NeurIPS 2024

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user's image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models. All videos from this paper are available at https://go.fb.me/MovieGenResearchVideos.

Multivariate time-series data in fields like healthcare and industry are informative but challenging due to high dimensionality and lack of labels. Recent self-supervised learning methods excel in learning rich representations without labels but struggle with disentangled embeddings and inductive bias issues like transformation-invariance. To address these challenges, we introduce TimeDRL, a framework for multivariate time-series representation learning with dual-level disentangled embeddings. TimeDRL features: (i) disentangled timestamp-level and instance-level embeddings using a [CLS] token strategy; (ii) timestamp-predictive and instance-contrastive tasks for representation learning; and (iii) avoidance of augmentation methods to eliminate inductive biases. Experiments on forecasting and classification datasets show TimeDRL outperforms existing methods, with further validation in semi-supervised settings with limited labeled data., Comment: This submission has been withdrawn to avoid duplication with a full version of the paper that is already available in another arXiv entry (arXiv:2410.12606). The withdrawn version was a short format prepared for a NeurIPS workshop and is no longer necessary as a separate arXiv submission

Low-Latency and Low-Power Edge AI is essential for Virtual Reality and Augmented Reality applications. Recent advances show that hybrid models, combining convolution layers (CNN) and transformers (ViT), often achieve superior accuracy/performance tradeoff on various computer vision and machine learning (ML) tasks. However, hybrid ML models can pose system challenges for latency and energy-efficiency due to their diverse nature in dataflow and memory access patterns. In this work, we leverage the architecture heterogeneity from Neural Processing Units (NPU) and Compute-In-Memory (CIM) and perform diverse execution schemas to efficiently execute these hybrid models. We also introduce H4H-NAS, a Neural Architecture Search framework to design efficient hybrid CNN/ViT models for heterogeneous edge systems with both NPU and CIM. Our H4H-NAS approach is powered by a performance estimator built with NPU performance results measured on real silicon, and CIM performance based on industry IPs. H4H-NAS searches hybrid CNN/ViT models with fine granularity and achieves significant (up to 1.34%) top-1 accuracy improvement on ImageNet dataset. Moreover, results from our Algo/HW co-design reveal up to 56.08% overall latency and 41.72% energy improvements by introducing such heterogeneous computing over baseline solutions. The framework guides the design of hybrid network architectures and system architectures of NPU+CIM heterogeneous systems.

Electric guitar tone modeling typically focuses on the non-linear transformation from clean to amplifier-rendered audio. Traditional methods rely on one-to-one mappings, incorporating device parameters into neural models to replicate specific amplifiers. However, these methods are limited by the need for specific training data. In this paper, we adapt a model based on the previous work, which leverages a tone embedding encoder and a feature wise linear modulation (FiLM) condition method. In this work, we altered conditioning method using a hypernetwork-based gated convolutional network (GCN) to generate audio that blends clean input with the tone characteristics of reference audio. By extending the training data to cover a wider variety of amplifier tones, our model is able to capture a broader range of tones. Additionally, we developed a real-time plugin to demonstrate the system's practical application, allowing users to experience its performance interactively. Our results indicate that the proposed system achieves superior tone modeling versatility compared to traditional methods., Comment: demo of the ISMIR paper

Subsampling layers play a crucial role in deep nets by discarding a portion of an activation map to reduce its spatial dimensions. This encourages the deep net to learn higher-level representations. Contrary to this motivation, we hypothesize that the discarded activations are useful and can be incorporated on the fly to improve models' prediction. To validate our hypothesis, we propose a search and aggregate method to find useful activation maps to be used at test time. We applied our approach to the task of image classification and semantic segmentation. Extensive experiments over nine different architectures on multiple datasets show that our method consistently improves model test-time performance, complementing existing test-time augmentation techniques. Our code is available at https://github.com/ca-joe-yang/discard-in-subsampling., Comment: ECCV 2024

Advancements in open-source pre-trained backbones make it relatively easy to fine-tune a model for new tasks. However, this lowered entry barrier poses potential risks, e.g., bad actors developing models for harmful applications. A question arises: Is possible to develop a pre-trained model that is difficult to fine-tune for certain downstream tasks? To begin studying this, we focus on few-shot classification (FSC). Specifically, we investigate methods to make FSC more challenging for a set of restricted classes while maintaining the performance of other classes. We propose to meta-learn over the pre-trained backbone in a manner that renders it a ''poor initialization''. Our proposed Learning to Obstruct (LTO) algorithm successfully obstructs four FSC methods across three datasets, including ImageNet and CIFAR100 for image classification, as well as CelebA for attribute classification., Comment: ECCV 2024

Modern organizations necessitate continuous business processes improvement to maintain efficiency, adaptability, and competitiveness. In the last few years, the Internet of Things, via the deployment of sensors and actuators, has heavily been adopted in organizational and industrial settings to monitor and automatize physical processes influencing and enhancing how people and organizations work. Such advancements are now pushed forward by the rise of the Digital Twin paradigm applied to organizational processes. Advanced ways of managing and maintaining business processes come within reach as there is a Digital Twin of a business process - a virtual replica with real-time capabilities of a real process occurring in an organization. Combining business process models with real-time data and simulation capabilities promises to provide a new way to guide day-to-day organization activities. However, integrating Digital Twins and business processes is a non-trivial task, presenting numerous challenges and ambiguities. This manifesto paper aims to contribute to the current state of the art by clarifying the relationship between business processes and Digital Twins, identifying ongoing research and open challenges, thereby shedding light on and driving future exploration of this innovative interplay., Comment: 15 pages, 3 figures, 1 table

Coherent control of single photon sources is a key requirement for the advancement of photonic quantum technologies. Among them, nanowire-based quantum dot sources are popular due to their potential for on-chip hybrid integration. Here we demonstrate on-demand single-photon generation ($g^{(2)}(0)(X^{*}) =0.078$ and $g^{(2)}(0)(X)= 0.03$) from resonantly excited InAsP/InP nanowire quantum dots and observe Rabi oscillations in the dot emission, indicating successful coherent manipulation of the excitonic states in the nanowire. We also measure a low emission time jitter for resonant excitation as compared to above-band excitation. This work addresses the long-standing challenge of resonantly exciting nanowire-quantum dots. It paves the way for hybrid quantum photonic integration, enabling spin-photon entanglement and matter memories on-chip.

Neural audio codec models are becoming increasingly important as they serve as tokenizers for audio, enabling efficient transmission or facilitating speech language modeling. The ideal neural audio codec should maintain content, paralinguistics, speaker characteristics, and audio information even at low bitrates. Recently, numerous advanced neural codec models have been proposed. However, codec models are often tested under varying experimental conditions. As a result, we introduce the Codec-SUPERB challenge at SLT 2024, designed to facilitate fair and lightweight comparisons among existing codec models and inspire advancements in the field. This challenge brings together representative speech applications and objective metrics, and carefully selects license-free datasets, sampling them into small sets to reduce evaluation computation costs. This paper presents the challenge's rules, datasets, five participant systems, results, and findings.

One of the classical tests of general relativity is the precision measurements by Pound and Rebka of red-shift/blue-shift of photons in a gravitational field. In this essay, we lay out a temporal version of the Pound-Rebka experiment. The emission and absorption of photons occurs at different times, rather than at different spatial locations as in the original Pound-Rebka experiment. This temporal Pound-Rebka experiment is equivalent to a gravitational Aharonov-Bohm Effect and is testable via current or near future satellite experiments., Comment: 10 pages, 1 figure. Awarded "honorable mention" in 2024 GRF essay contest. Accepted for publication in IJMPD

Quantum reservoir engineering aims to transform typically detrimental dissipations into advantageous resources. We present a versatile method for creating a squeezed thermal reservoir for quantum systems. By coupling the system to a lossy mode within a normal thermal environment, we can emulate the effect of a squeezed reservoir characterized by reduced fluctuations in one quadrature. We demonstrate this approach through two illustrative cases: one for two-level systems, such as qubits or atoms, and another for bosonic modes, like photons or phonons. This method leverages constant linear interactions within a normal thermal environment, ensuring experimental feasibility without requiring squeezed light inputs or time-dependent modulations. This technique holds promise for various applications, including the enhancement of dissipative squeezing, stabilization of entanglement, advancement of quantum simulations, exploration of quantum thermodynamics and phase transitions, and improvement of precision measurements., Comment: 6 pages, 3 figures

Neural network models for guitar amplifier emulation, while being effective, often demand high computational cost and lack interpretability. Drawing ideas from physical amplifier design, this paper aims to address these issues with a new differentiable digital signal processing (DDSP)-based model, called ``DDSP guitar amp,'' that models the four components of a guitar amp (i.e., preamp, tone stack, power amp, and output transformer) using specific DSP-inspired designs. With a set of time- and frequency-domain metrics, we demonstrate that DDSP guitar amp achieves performance comparable with that of black-box baselines while requiring less than 10\% of the computational operations per audio sample, thereby holding greater potential for usages in real-time applications., Comment: Preprint paper

Two-dimensional (2-D) monolayer transition-metal dichalcogenides (TMDCs) are promising materials for realizing ultracompact, low-threshold semiconductor lasers. And the development of the electrical-driven TMDC devices is crucial for enhancing the integration potential of practical optoelectronic systems. However, at current stage, the electrically-driven 2-D TMDC laser has never been realized. Herein, we have developed the first electrically-driven 2-D TMDC microcavity laser.

In Business Process Management (BPM), effectively comprehending process models is crucial yet poses significant challenges, particularly as organizations scale and processes become more complex. This paper introduces a novel framework utilizing the advanced capabilities of Large Language Models (LLMs) to enhance the interpretability of complex process models. We present different methods for abstracting business process models into a format accessible to LLMs, and we implement advanced prompting strategies specifically designed to optimize LLM performance within our framework. Additionally, we present a tool, AIPA, that implements our proposed framework and allows for conversational process querying. We evaluate our framework and tool by i) an automatic evaluation comparing different LLMs, model abstractions, and prompting strategies and ii) a user study designed to assess AIPA's effectiveness comprehensively. Results demonstrate our framework's ability to improve the accessibility and interpretability of process models, pioneering new pathways for integrating AI technologies into the BPM field.

The neural codec model reduces speech data transmission delay and serves as the foundational tokenizer for speech language models (speech LMs). Preserving emotional information in codecs is crucial for effective communication and context understanding. However, there is a lack of studies on emotion loss in existing codecs. This paper evaluates neural and legacy codecs using subjective and objective methods on emotion datasets like IEMOCAP. Our study identifies which codecs best preserve emotional information under various bitrate scenarios. We found that training codec models with both English and Chinese data had limited success in retaining emotional information in Chinese. Additionally, resynthesizing speech through these codecs degrades the performance of speech emotion recognition (SER), particularly for emotions like sadness, depression, fear, and disgust. Human listening tests confirmed these findings. This work guides future speech technology developments to ensure new codecs maintain the integrity of emotional information in speech.

Replicating analog device circuits through neural audio effect modeling has garnered increasing interest in recent years. Existing work has predominantly focused on a one-to-one emulation strategy, modeling specific devices individually. In this paper, we tackle the less-explored scenario of one-to-many emulation, utilizing conditioning mechanisms to emulate multiple guitar amplifiers through a single neural model. For condition representation, we use contrastive learning to build a tone embedding encoder that extracts style-related features of various amplifiers, leveraging a dataset of comprehensive amplifier settings. Targeting zero-shot application scenarios, we also examine various strategies for tone embedding representation, evaluating referenced tone embedding against two retrieval-based embedding methods for amplifiers unseen in the training time. Our findings showcase the efficacy and potential of the proposed methods in achieving versatile one-to-many amplifier modeling, contributing a foundational step towards zero-shot audio modeling applications., Comment: ISMIR 2024

In Extended Reality (XR) applications, high data rates and low latency are crucial for immersive experiences. Uplink transmission in XR is challenging due to the limited antennas and power of lightweight XR devices. To improve data transmission rates, we investigate a relay-assisted carrier aggregation (RACA) system. The XR device simultaneously transmits data to an access point (AP) and a relay in proximity over low-frequency and high-frequency bands, respectively. Then, the relay down-converts and amplifies the signals to the AP, effectively acting as an additional transmit antenna for the XR device. In this paper, we propose two algorithms to maximize the data rate of the XR device in their respective protocols. In the centralized protocol, the rate maximization problem is equivalently transformed as a weighted mean square error minimization (WMMSE) problem which can be solved iteratively by alternative optimization. In the distributed protocol, the rate maximization problem is decomposed into two independent sub-problems where the rate of the direct link and the rate of the relay link are maximized by singular value decomposition (SVD)-based methods with water-filling (WF). Simulation results show that the rate of the RACA system is improved by $32\%$ compared to that of the conventional carrier aggregation scheme.

We release the EARS (Expressive Anechoic Recordings of Speech) dataset, a high-quality speech dataset comprising 107 speakers from diverse backgrounds, totaling in 100 hours of clean, anechoic speech data. The dataset covers a large range of different speaking styles, including emotional speech, different reading styles, non-verbal sounds, and conversational freeform speech. We benchmark various methods for speech enhancement and dereverberation on the dataset and evaluate their performance through a set of instrumental metrics. In addition, we conduct a listening test with 20 participants for the speech enhancement task, where a generative method is preferred. We introduce a blind test set that allows for automatic online evaluation of uploaded data. Dataset download links and automatic evaluation server can be found online., Comment: Accepted at Interspeech 2024

The trend of scaling up speech generation models poses a threat of biometric information leakage of the identities of the voices in the training data, raising privacy and security concerns. In this paper, we investigate training multi-speaker text-to-speech (TTS) models using data that underwent speaker anonymization (SA), a process that tends to hide the speaker identity of the input speech while maintaining other attributes. Two signal processing-based and three deep neural network-based SA methods were used to anonymize VCTK, a multi-speaker TTS dataset, which is further used to train an end-to-end TTS model, VITS, to perform unseen speaker TTS during the testing phase. We conducted extensive objective and subjective experiments to evaluate the anonymized training data, as well as the performance of the downstream TTS model trained using those data. Importantly, we found that UTMOS, a data-driven subjective rating predictor model, and GVD, a metric that measures the gain of voice distinctiveness, are good indicators of the downstream TTS performance. We summarize insights in the hope of helping future researchers determine the goodness of the SA system for multi-speaker TTS training., Comment: 5 pages. Submitted to Signal Processing Letters. Audio sample page: https://unilight.github.io/Publication-Demos/publications/sa-tts-spl/index.html