Search

We experimentally demonstrate magnon-polaritons with ultrastrong coupling and directional nonreciprocity in a metamolecule lacking time-reversal and space-inversion symmetries at room temperature. These experimental results are reproduced well via numerical simulations and theoretical consideration. Ultrastrong coupling is due to a direct interaction of magnons in the magnetic meta-atom with microwave photons confined in the chiral meta-atom as a resonator. Our results reveal a crucial step in identifying deepstrongly-coupled and optically-moving magnon-polaritons for hybrid quantum systems, synthetic gauge fields, and quasi-particle ``chemistry'' using metamaterials., Comment: 6 pages, 3 figures

Multimodal language models that process both text and speech have a potential for applications in spoken dialogue systems. However, current models face two major challenges in response generation latency: (1) generating a spoken response requires the prior generation of a written response, and (2) speech sequences are significantly longer than text sequences. This study addresses these issues by extending the input and output sequences of the language model to support the parallel generation of text and speech. Our experiments on spoken question answering tasks demonstrate that our approach improves latency while maintaining the quality of response content. Additionally, we show that latency can be further reduced by generating speech in multiple sequences. Demo samples are available at https://rinnakk.github.io/research/publications/PSLM., Comment: 9 pages, 6 figures, 4 tables, accepted for Findings of EMNLP 2024. Demo samples: https://rinnakk.github.io/research/publications/PSLM

AI democratization aims to create a world in which the average person can utilize AI techniques. To achieve this goal, numerous research institutes have attempted to make their results accessible to the public. In particular, large pre-trained models trained on large-scale data have shown unprecedented potential, and their release has had a significant impact. However, most of the released models specialize in the English language, and thus, AI democratization in non-English-speaking communities is lagging significantly. To reduce this gap in AI access, we released Generative Pre-trained Transformer (GPT), Contrastive Language and Image Pre-training (CLIP), Stable Diffusion, and Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) pre-trained in Japanese. By providing these models, users can freely interface with AI that aligns with Japanese cultural values and ensures the identity of Japanese culture, thus enhancing the democratization of AI. Additionally, experiments showed that pre-trained models specialized for Japanese can efficiently achieve high performance in Japanese tasks., Comment: 9 pages, 1 figure, 5 tables, accepted for LREC-COLING 2024. Models are publicly available at https://huggingface.co/rinna

We report numerical studies of microwave bianisotropies by magnetochiral (MCh) metasurfaces consisting of double Z-type gammadions with perpendicularly magnetized substrates. The metasurfaces' effective polarizability tensor, extracted from calculated reflection and transmission coefficients, has components of the non-reciprocal moving-type bianisotropy as well as the reciprocal chiral-type bianisotropy and non-reciprocal magneto-optical (MO) effect. Combination of the metasurfaces with contraposition MCh metasurfaces cancels both the chiral-type bianisotropy and MO effect, resulting in pure moving optical media. Achieved perfect transmission with phase difference of $\pi$ realizes an ideal gyrator for arbitrary spatial and polarization modes., Comment: 11 pages, 10 figures

Advances in machine learning have made it possible to perform various text and speech processing tasks, such as automatic speech recognition (ASR), in an end-to-end (E2E) manner. E2E approaches utilizing pre-trained models are gaining attention for conserving training data and resources. However, most of their applications in ASR involve only one of either a pre-trained speech or a language model. This paper proposes integrating a pre-trained speech representation model and a large language model (LLM) for E2E ASR. The proposed model enables the optimization of the entire ASR process, including acoustic feature extraction and acoustic and language modeling, by combining pre-trained models with a bridge network and also enables the application of remarkable developments in LLM utilization, such as parameter-efficient domain adaptation and inference optimization. Experimental results demonstrate that the proposed model achieves a performance comparable to that of modern E2E ASR models by utilizing powerful pre-training models with the proposed integrated approach., Comment: 17 pages, 4 figures, 9 tables, accepted for Findings of ACL 2024. The model is available at https://huggingface.co/rinna/nue-asr

The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents. However, generating human-like spoken dialogues from these written dialogues remains challenging. Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation. This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter. Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap. Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility., Comment: 18 pages, 8 figures, 9 tables, audio samples: https://rinnakk.github.io/research/publications/CHATS/

With the advent of quantum technology, the interest in communication tasks assisted by quantum systems has increased both in academia and industry. Nonetheless, the transmission of a quantum state in real-world scenarios is bounded by environmental noise, so that the quantum channel is an open quantum system. In this work, we study a high-dimensional open quantum system in a multicore optical fiber by characterizing the environmental interaction as quantum operations corresponding to probabilistic phase-flips. The experimental platform is currently state-of-the-art for quantum information processing with multicore fibers. At a given evolution stage we observe a non-Markovian behaviour of the system, which is demonstrated through a proof-of-principle implementation of the Quantum Vault protocol. A better understanding of phase-noise in multicore fibers will improve several real-world communication protocols, since they are a prime candidate to be adopted in future telecom networks., Comment: 11 pages, 6 figures

In a controllable text generation dataset, there exist unannotated attributes that could provide irrelevant learning signals to models that use it for training and thus degrade their performance. We propose focused prefix tuning(FPT) to mitigate the problem and to enable the control to focus on the desired attribute. Experimental results show that FPT can achieve better control accuracy and text fluency than baseline models in single-attribute control tasks. In multi-attribute control tasks, FPT achieves comparable control accuracy with the state-of-the-art approach while keeping the flexibility to control new attributes without retraining existing models., Comment: Accepted to the ACL 2023

Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data., Comment: 5 pages, 2 figures, 3 tables, accepted for INTERSPEECH 2023. Audio samples: https://rinnakk.github.io/research/publications/UniFLG

We propose a method for scene-level sketch-to-photo synthesis with text guidance. Although object-level sketch-to-photo synthesis has been widely studied, whole-scene synthesis is still challenging without reference photos that adequately reflect the target style. To this end, we leverage knowledge from recent large-scale pre-trained generative models, resulting in text-guided sketch-to-photo synthesis without the need for reference images. To train our model, we use self-supervised learning from a set of photographs. Specifically, we use a pre-trained edge detector that maps both color and sketch images into a standardized edge domain, which reduces the gap between photograph-based edge images (during training) and hand-drawn sketch images (during inference). We implement our method by fine-tuning a latent diffusion model (i.e., Stable Diffusion) with sketch and text conditions. Experiments show that the proposed method translates original sketch images that are not extracted from color images into photos with compelling visual quality.

The recent text-to-speech (TTS) has achieved quality comparable to that of humans; however, its application in spoken dialogue has not been widely studied. This study aims to realize a TTS that closely resembles human dialogue. First, we record and transcribe actual spontaneous dialogues. Then, the proposed dialogue TTS is trained in two stages: first stage, variational autoencoder (VAE)-VITS or Gaussian mixture variational autoencoder (GMVAE)-VITS is trained, which introduces an utterance-level latent variable into variational inference with adversarial learning for end-to-end text-to-speech (VITS), a recently proposed end-to-end TTS model. A style encoder that extracts a latent speaking style representation from speech is trained jointly with TTS. In the second stage, a style predictor is trained to predict the speaking style to be synthesized from dialogue history. During inference, by passing the speaking style representation predicted by the style predictor to VAE/GMVAE-VITS, speech can be synthesized in a style appropriate to the context of the dialogue. Subjective evaluation results demonstrate that the proposed method outperforms the original VITS in terms of dialogue-level naturalness., Comment: 5 pages, 3 figures, accepted for INTERSPEECH 2022. Audio samples: https://rinnakk.github.io/research/publications/DialogueTTS/

The development of neural vocoders (NVs) has resulted in the high-quality and fast generation of waveforms. However, conventional NVs target a single sampling rate and require re-training when applied to different sampling rates. A suitable sampling rate varies from application to application due to the trade-off between speech quality and generation speed. In this study, we propose a method to handle multiple sampling rates in a single NV, called the MSR-NV. By generating waveforms step-by-step starting from a low sampling rate, MSR-NV can efficiently learn the characteristics of each frequency band and synthesize high-quality speech at multiple sampling rates. It can be regarded as an extension of the previously proposed NVs, and in this study, we extend the structure of Parallel WaveGAN (PWG). Experimental evaluation results demonstrate that the proposed method achieves remarkably higher subjective quality than the original PWG trained separately at 16, 24, and 48 kHz, without increasing the inference time. We also show that MSR-NV can leverage speech with lower sampling rates to further improve the quality of the synthetic speech., Comment: 6 pages including supplement, 3 figures, accepted for INTERSPEECH 2022. Audio samples: https://rinnakk.github.io/research/publications/MSR-NV/

This paper proposes a hierarchical generative model with a multi-grained latent variable to synthesize expressive speech. In recent years, fine-grained latent variables are introduced into the text-to-speech synthesis that enable the fine control of the prosody and speaking styles of synthesized speech. However, the naturalness of speech degrades when these latent variables are obtained by sampling from the standard Gaussian prior. To solve this problem, we propose a novel framework for modeling the fine-grained latent variables, considering the dependence on an input text, a hierarchical linguistic structure, and a temporal structure of latent variables. This framework consists of a multi-grained variational autoencoder, a conditional prior, and a multi-level auto-regressive latent converter to obtain the different time-resolution latent variables and sample the finer-level latent variables from the coarser-level ones by taking into account the input text. Experimental results indicate an appropriate method of sampling fine-grained latent variables without the reference signal at the synthesis stage. Our proposed framework also provides the controllability of speaking style in an entire utterance., Comment: 5 pages, accepted to INTERSPEECH 2020, demo page: https://www.rinna.jp/research/interspeech2020/

Dancing to music is one of human's innate abilities since ancient times. In machine learning research, however, synthesizing dance movements from music is a challenging problem. Recently, researchers synthesize human motion sequences through autoregressive models like recurrent neural network (RNN). Such an approach often generates short sequences due to an accumulation of prediction errors that are fed back into the neural network. This problem becomes even more severe in the long motion sequence generation. Besides, the consistency between dance and music in terms of style, rhythm and beat is yet to be taken into account during modeling. In this paper, we formalize the music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture to efficiently process long sequences of music features and capture the fine-grained correspondence between music and dance. Furthermore, we propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation, which gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead. Extensive experiments show that our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation. We also make a demo video to demonstrate the superior performance of our proposed approach at https://www.youtube.com/watch?v=lmE20MEheZ8., Comment: This paper includes the unrigorous quantitative experimental results and has been withdrawn from the conference

Orbital angular momentum (OAM) of photons is carried upon the wave front of an optical vortex and is important in physics research due to its fundamental degree of freedom. As for the interaction with materials, the optical OAM was shown to be transferred to the valence electron based on modified selection rules where a single ion is carefully aligned to the center of a vortex [Schmiegelow et al.(2016)]. We here demonstrate an elaborate way of extracting the distributed OAMs in the two-dimensional x-ray wave field at the exit-face of the specimen by a vorticity-sensitive microscopy, which detects vorticity in the reciprocal-space wave field downstream of the optical vortices. This method enables us to measure the distributed topological charges and OAMs by a single image with the wide field of view over 300 $\mu$m. Among various wavelength, the research of x-ray OAM is especially interesting since it could be closely related to the predicted OAM-induced x-ray dichroism [Veenendaal et al.(2007)]., Comment: 4 pages, 3 figures

We develop a theory for the trajectory of an x ray in the presence of a crystal deformation. A set of equations of motion for an x-ray wave packet including the dynamical diffraction is derived, taking into account the Berry phase as a correction to geometrical optics. The trajectory of the wave packet has a shift of the center position due to a crystal deformation. Remarkably, in the vicinity of the Bragg condition, the shift is enhanced by a factor $\omega /\Delta \omega$ ($\omega$: frequency of an x ray, $\Delta\omega$: gap frequency induced by the Bragg reflection). Comparison with the conventional dynamical diffraction theory is also made., Comment: 4 pages, 2 figures. Title changed

We theoretically propose that the optical analog of a Lorentz force acting on a ray of light is realized in multiferroic materials such as GaFeO$_3$ showing the magnetoelectric effect. The toroidal moment plays the role of a ``vector potential" while its rotation corresponds to a "magnetic field" for photons. Hence, the light is subject to the Lorentz force when propagating through the domain wall region of the ferromagnetic or ferroelectric order. A realistic estimate on the magnitude of this effect is given., Comment: 4 pages, 2 figures. The title is changed

We theoretically propose a method to enhance dramatically a magneto-chiral(MC) effect by using the photonic crystals composed of a multiferroic material. The MC effect, the directional birefringence even for unpolarized light, is so small that it has been difficult to observe experimentally. Two kinds of periodic structures are investigated; (a) a multilayer and (b) a stripe composed of a magneto-chiral material and air. In both cases, the difference in reflectivity between different magnetization directions is enhanced by a factor of hundreds compared with a bulk material., Comment: 3 pages, 3 figures