Search

This paper presents our systems (denoted as T13) for the singing voice conversion challenge (SVCC) 2023. For both in-domain and cross-domain English singing voice conversion (SVC) tasks (Task 1 and Task 2), we adopt a recognition-synthesis approach with self-supervised learning-based representation. To achieve data-efficient SVC with a limited amount of target singer/speaker's data (150 to 160 utterances for SVCC 2023), we first train a diffusion-based any-to-any voice conversion model using publicly available large-scale 750 hours of speech and singing data. Then, we finetune the model for each target singer/speaker of Task 1 and Task 2. Large-scale listening tests conducted by SVCC 2023 show that our T13 system achieves competitive naturalness and speaker similarity for the harder cross-domain SVC (Task 2), which implies the generalization ability of our proposed method. Our objective evaluation results show that using large datasets is particularly beneficial for cross-domain SVC., Comment: Accepted to ASRU 2023

Automatic speech recognition (ASR) systems often encounter difficulties in accurately recognizing rare words, leading to errors that can have a negative impact on downstream tasks such as keyword spotting, intent detection, and text summarization. To address this challenge, we present a novel ASR postprocessing method that focuses on improving the recognition of rare words through error detection and context-aware error correction. Our method optimizes the decoding process by targeting only the predicted error positions, minimizing unnecessary computations. Moreover, we leverage a rare word list to provide additional contextual knowledge, enabling the model to better correct rare words. Experimental results across five datasets demonstrate that our proposed method achieves significantly lower word error rates (WERs) than previous approaches while maintaining a reasonable inference speed. Furthermore, our approach exhibits promising robustness across different ASR systems., Comment: 6 pages, 5 figures, conference

This paper shows the usefulness of the Perov contraction theorem, which is a generalization of the classical Banach contraction theorem, for solving Markov dynamic programming problems. When the reward function is unbounded, combining an appropriate weighted supremum norm with the Perov contraction theorem yields a unique fixed point of the Bellman operator under weaker conditions than existing approaches. An application to the optimal savings problem shows that the average growth rate condition derived from the spectral radius of a certain nonnegative matrix is sufficient and almost necessary for obtaining a solution.

We present the second edition of the VoiceMOS Challenge, a scientific event that aims to promote the study of automatic prediction of the mean opinion score (MOS) of synthesized and processed speech. This year, we emphasize real-world and challenging zero-shot out-of-domain MOS prediction with three tracks for three different voice evaluation scenarios. Ten teams from industry and academia in seven different countries participated. Surprisingly, we found that the two sub-tracks of French text-to-speech synthesis had large differences in their predictability, and that singing voice-converted samples were not as difficult to predict as we had expected. Use of diverse datasets and listener information during training appeared to be successful approaches., Comment: Accepted to ASRU 2023

In healthy-to-pathological voice conversion (H2P-VC), healthy speech is converted into pathological while preserving the identity. The paper improves on previous two-stage approach to H2P-VC where (1) speech is created first with the appropriate severity, (2) then the speaker identity of the voice is converted while preserving the severity of the voice. Specifically, we propose improvements to (2) by using phonetic posteriorgrams (PPG) and global style tokens (GST). Furthermore, we present a new dataset that contains parallel recordings of pathological and healthy speakers with the same identity which allows more precise evaluation. Listening tests by expert listeners show that the framework preserves severity of the source sample, while modelling target speaker's voice. We also show that (a) pathology impacts x-vectors but not all speaker information is lost, (b) choosing source speakers based on severity labels alone is insufficient., Comment: 7 pages, 3 figures, 5 tables. Accepted to IEEE Automatic Speech Recognition and Understanding Workshop 2023

How should financial institutions hedge their balance sheets against interest rate risk when they have long-term assets and liabilities? Using the perspective of functional and numerical analysis, we propose a model-free bond portfolio selection method that generalizes classical immunization and accommodates arbitrary liability structure, portfolio constraints, and perturbations in interest rates. We prove the generic existence of an immunizing portfolio that maximizes the worst-case equity with a tight error estimate and provide a solution algorithm. Numerical evaluations using empirical and simulated yield curves from a no-arbitrage term structure model support the feasibility and accuracy of our approach relative to existing methods.

We prove that when individual firms employ constant-returns-to-scale production functions, the aggregate production function defined by the maximum achievable total output given total inputs is always linear on some part of the domain. Our result provides a microfoundation for the linear production function.

We propose a novel framework for electrolaryngeal speech intelligibility enhancement through the use of robust linguistic encoders. Pretraining and fine-tuning approaches have proven to work well in this task, but in most cases, various mismatches, such as the speech type mismatch (electrolaryngeal vs. typical) or a speaker mismatch between the datasets used in each stage, can deteriorate the conversion performance of this framework. To resolve this issue, we propose a linguistic encoder robust enough to project both EL and typical speech in the same latent space, while still being able to extract accurate linguistic information, creating a unified representation to reduce the speech type mismatch. Furthermore, we introduce HuBERT output features to the proposed framework for reducing the speaker mismatch, making it possible to effectively use a large-scale parallel dataset during pretraining. We show that compared to the conventional framework using mel-spectrogram input and output features, using the proposed framework enables the model to synthesize more intelligible and naturally sounding speech, as shown by a significant 16% improvement in character error rate and 0.83 improvement in naturalness score., Comment: Accepted to ICASSP 2024. Demo page: lesterphillip.github.io/icassp2024_el_sie

This study introduces a novel training paradigm, audio difference learning, for improving audio captioning. The fundamental concept of the proposed learning method is to create a feature representation space that preserves the relationship between audio, enabling the generation of captions that detail intricate audio information. This method employs a reference audio along with the input audio, both of which are transformed into feature representations via a shared encoder. Captions are then generated from these differential features to describe their differences. Furthermore, a unique technique is proposed that involves mixing the input audio with additional audio, and using the additional audio as a reference. This results in the difference between the mixed audio and the reference audio reverting back to the original input audio. This allows the original input's caption to be used as the caption for their difference, eliminating the need for additional annotations for the differences. In the experiments using the Clotho and ESC50 datasets, the proposed method demonstrated an improvement in the SPIDEr score by 7% compared to conventional methods., Comment: submitted to ICASSP2024

We introduce and begin the study of quasi-BPS categories for K3 surfaces, which are a categorical version of the BPS cohomologies for K3 surfaces. We construct semiorthogonal decompositions of derived categories of coherent sheaves on moduli stacks of semistable objects on K3 surfaces, where each summand is a categorical Hall product of quasi-BPS categories. We also prove the wall-crossing equivalence of quasi-BPS categories, which generalizes Halpern-Leistner's wall-crossing equivalence of moduli spaces of stable objects for primitive Mukai vectors on K3 surfaces. We also introduce and study a reduced quasi-BPS category. When the weight is coprime to the Mukai vector, the reduced quasi-BPS category is proper, smooth, and its Serre functor is trivial \'{e}tale locally on the good moduli space. Moreover we prove that its topological K-theory recovers the BPS invariants of K3 surfaces, which are known to be equal to the Euler characteristics of Hilbert schemes of points on K3 surfaces. We regard reduced quasi-BPS categories as noncommutative hyperk\"ahler varieties which are categorical versions of crepant resolutions of singular symplectic moduli spaces of semistable objects on K3 surfaces., Comment: 66 pages

In previous works, we introduced and studied certain categories called quasi-BPS categories associated to symmetric quivers with potential, preprojective algebras, and local surfaces. They have properties reminiscent of BPS invariants/ cohomologies in enumerative geometry, for example they play important roles in categorical wall-crossing formulas. In this paper, we make the connections between quasi-BPS categories and BPS cohomologies more precise via the cycle map for topological K-theory. We show the existence of filtrations on topological K-theory of quasi-BPS categories whose associated graded are isomorphic to the monodromy invariant BPS cohomologies. Along the way, we also compute the topological K-theory of categories of matrix factorizations in terms of the monodromy invariant vanishing cycles (a version of this comparison was already known by work of Blanc-Robalo-To\"en-Vezzosi), prove a Grothendieck-Riemann-Roch theorem for matrix factorizations, and prove the compatibility between the Koszul equivalence in K-theory and dimensional reduction in cohomology. In a separate paper, we use the results from this paper to show that the quasi-BPS categories of K3 surfaces recover the BPS invariants of the corresponding local surface, which are Euler characteristics of Hilbert schemes of points on K3 surfaces., Comment: 80 pages, the main theorems apply now to all quivers

We study certain categories associated to symmetric quivers with potential, called quasi-BPS categories. We construct semiorthogonal decompositions of the categories of matrix factorizations for moduli stacks of representations of (framed or unframed) symmetric quivers with potential, where the summands are categorical Hall products of quasi-BPS categories. These results generalize our previous results about the three loop quiver. We prove several properties of quasi-BPS categories: wall-crossing equivalence, strong generation, and categorical support lemma in the case of tripled quivers with potential. We also introduce reduced quasi-BPS categories for preprojective algebras, which have trivial relative Serre functor and are indecomposable when the weight is coprime with the total dimension. In this case, we regard the reduced quasi-BPS categories as noncommutative local hyperk\"ahler varieties, and as (twisted) categorical versions of crepant resolutions of singularities of good moduli spaces of representations of preprojective algebras. The studied categories include the local models of quasi-BPS categories of K3 surfaces. In a follow-up paper, we establish analogous properties for quasi-BPS categories of K3 surfaces., Comment: 55 pages

Non-autoregressive (non-AR) sequence-to-seqeunce (seq2seq) models for voice conversion (VC) is attractive in its ability to effectively model the temporal structure while enjoying boosted intelligibility and fast inference thanks to non-AR modeling. However, the dependency of current non-AR seq2seq VC models on ground truth durations extracted from an external AR model greatly limits its generalization ability to smaller training datasets. In this paper, we first demonstrate the above-mentioned problem by varying the training data size. Then, we present AAS-VC, a non-AR seq2seq VC model based on automatic alignment search (AAS), which removes the dependency on external durations and serves as a proper inductive bias to provide the required generalization ability for small datasets. Experimental results show that AAS-VC can generalize better to a training dataset of only 5 minutes. We also conducted ablation studies to justify several model design choices. The audio samples and implementation are available online., Comment: Submitted to ICASSP 2024. Demo: https://unilight.github.io/Publication-Demos/publications/aas-vc/index.html. Code: https://github.com/unilight/seq2seq-vc

With the increasing importance of data privacy protection, various privacy-preserving machine learning methods have been proposed. In the clustering domain, various algorithms with a federated learning framework (i.e., federated clustering) have been actively studied and showed high clustering performance while preserving data privacy. However, most of the base clusterers (i.e., clustering algorithms) used in existing federated clustering algorithms need to specify the number of clusters in advance. These algorithms, therefore, are unable to deal with data whose distributions are unknown or continually changing. To tackle this problem, this paper proposes a privacy-preserving continual federated clustering algorithm. In the proposed algorithm, an adaptive resonance theory-based clustering algorithm capable of continual learning is used as a base clusterer. Therefore, the proposed algorithm inherits the ability of continual learning. Experimental results with synthetic and real-world datasets show that the proposed algorithm has superior clustering performance to state-of-the-art federated clustering algorithms while realizing data privacy protection and continual learning ability. The source code is available at \url{https://github.com/Masuyama-lab/FCAC}., Comment: This paper is currently under review. arXiv admin note: substantial text overlap with arXiv:2305.01507

Foreign accent conversion (FAC) is a special application of voice conversion (VC) which aims to convert the accented speech of a non-native speaker to a native-sounding speech with the same speaker identity. FAC is difficult since the native speech from the desired non-native speaker to be used as the training target is impossible to collect. In this work, we evaluate three recently proposed methods for ground-truth-free FAC, where all of them aim to harness the power of sequence-to-sequence (seq2seq) and non-parallel VC models to properly convert the accent and control the speaker identity. Our experimental evaluation results show that no single method was significantly better than the others in all evaluation axes, which is in contrast to conclusions drawn in previous studies. We also explain the effectiveness of these methods with the training input and output of the seq2seq model and examine the design choice of the non-parallel VC model, and show that intelligibility measures such as word error rates do not correlate well with subjective accentedness. Finally, our implementation is open-sourced to promote reproducible research and help future researchers improve upon the compared systems., Comment: Accepted to the 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). Demo page: https://unilight.github.io/Publication-Demos/publications/fac-evaluate. Code: https://github.com/unilight/seq2seq-vc

One objective of Speech Quality Assessment (SQA) is to estimate the ranks of synthetic speech systems. However, recent SQA models are typically trained using low-precision direct scores such as mean opinion scores (MOS) as the training objective, which is not straightforward to estimate ranking. Although it is effective for predicting quality scores of individual sentences, this approach does not account for speech and system preferences when ranking multiple systems. We propose a training framework of SQA models that can be trained with only preference scores derived from pairs of MOS to improve ranking prediction. Our experiment reveals conditions where our framework works the best in terms of pair generation, aggregation functions to derive system score from utterance preferences, and threshold functions to determine preference from a pair of MOS. Our results demonstrate that our proposed method significantly outperforms the baseline model in Spearman's Rank Correlation Coefficient., Comment: Accepted by Interspeech 2023, oral

Building defect prediction models based on online learning can enhance prediction accuracy. It continuously rebuilds a new prediction model when adding a new data point. However, predicting a module as "non-defective" (i.e., negative prediction) can result in fewer test cases for such modules. Therefore, defects can be overlooked during testing, even when the module is defective. The erroneous test results are used as learning data by online learning, which could negatively affect prediction accuracy. In our experiment, we demonstrate this negative influence on prediction accuracy., Comment: 2 pages