Search

We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants. However, current spoken language models struggle to generate plausible speech past tens of seconds, from high temporal resolution of speech tokens causing loss of coherence, to architectural issues with long-sequence training or extrapolation, to memory costs at inference time. With these considerations we propose SpeechSSM, the first speech language model to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporaneous speech) in a single decoding session without text intermediates, based on recent advances in linear-time sequence modeling. Furthermore, to address growing challenges in spoken language evaluation, especially in this new long-form setting, we propose: new embedding-based and LLM-judged metrics; quality measurements over length and time; and a new benchmark for long-form speech processing and generation, LibriSpeech-Long. Speech samples and the dataset are released at https://google.github.io/tacotron/publications/speechssm/

Chatbot research is advancing with the growing importance of chatbots in fields that require human interactions, such as customer support and mental health care. Despite these advancements, chatbots still face significant challenges in understanding subtle nuances and managing long conversation histories. To address these issues, our study introduces a dual approach: firstly, we employ Emotional Preference Optimization (EPO) to train chatbots not only with correct responses but also with counter-emotional responses-those that are contextually similar but emotionally divergent. This training enables the model to discern fine nuance distinctions between correct and counter-emotional responses, thereby enhancing the quality of its responses. Secondly, we introduce MambaCompressor to effectively compress and manage extensive conversation histories, significantly reducing time and memory complexities while improving the chatbot's contextual understanding. Our comprehensive experiments across multiple datasets demonstrate that our model significantly outperforms existing models in generating empathetic responses and efficiently managing lengthy dialogues.

In human communication, both verbal and non-verbal cues play a crucial role in conveying emotions, intentions, and meaning beyond words alone. These non-linguistic information, such as facial expressions, eye contact, voice tone, and pitch, are fundamental elements of effective interactions, enriching conversations by adding emotional and contextual depth. Recognizing the importance of non-linguistic content in communication, we present AV-EmoDialog, a dialogue system designed to exploit verbal and non-verbal information from users' audio-visual inputs to generate more responsive and empathetic interactions. AV-EmoDialog systematically exploits the emotional cues in audio-visual dialogues; extracting speech content and emotional tones from speech, analyzing fine-grained facial expressions from visuals, and integrating these cues to generate emotionally aware responses in an end-to-end manner. Through extensive experiments, we validate that the proposed AV-EmoDialog outperforms existing multimodal LLMs in generating not only emotionally appropriate but also contextually appropriate responses.

We present a general theoretical framework for group resetting dynamics in multi-agent systems in a drift potential. This setup contrasts with a typical resetting problem that involves a single searcher looking for a target, with resetting traditionally studied to optimize the search time to a target. More recently, resetting has also been used as a regulatory mechanism to avoid adverse outcomes, such as preventing critically high water levels in dams or deleveraging financial portfolios. Here, we extend current resetting theories to group dynamics, with applications ranging from bacterial evolution under antibiotic pressure to multiple-searcher optimization algorithms. Our framework incorporates extreme value statistics and renewal theory, from which we derive a master equation for the center of mass distribution of a group of searchers. This master equation allows us to calculate essential observables analytically. For example, how the group's average position depends on group size, resetting rates, drift potential strength, and diffusion constants. This theoretical approach offers a new perspective on optimizing group search and regulatory mechanisms through resetting., Comment: 5 pages, 3 figures

DEtection TRansformer (DETR) has emerged as a promising architecture for object detection, offering an end-to-end prediction pipeline. In practice, however, DETR generates hundreds of predictions that far outnumber the actual number of objects present in an image. This raises the question: can we trust and use all of these predictions? Addressing this concern, we present empirical evidence highlighting how different predictions within the same image play distinct roles, resulting in varying reliability levels across those predictions. More specifically, while multiple predictions are often made for a single object, our findings show that most often one such prediction is well-calibrated, and the others are poorly calibrated. Based on these insights, we demonstrate identifying a reliable subset of DETR's predictions is crucial for accurately assessing the reliability of the model at both object and image levels. Building on this viewpoint, we first tackle the shortcomings of widely used performance and calibration metrics, such as average precision and various forms of expected calibration error. Specifically, they are inadequate for determining which subset of DETR's predictions should be trusted and utilized. In response, we present Object-level Calibration Error (OCE), which is capable of assessing the calibration quality both across different models and among various configurations within a specific model. As a final contribution, we introduce a post hoc Uncertainty Quantification (UQ) framework that predicts the accuracy of the model on a per-image basis. By contrasting the average confidence scores of positive (i.e., likely to be matched) and negative predictions determined by OCE, the framework assesses the reliability of the DETR model for each test image.

Employing large language models (LLMs) to enable embodied agents has become popular, yet it presents several limitations in practice. In this work, rather than using LLMs directly as agents, we explore their use as tools for embodied agent learning. Specifically, to train separate agents via offline reinforcement learning (RL), an LLM is used to provide dense reward feedback on individual actions in training datasets. In doing so, we present a consistency-guided reward ensemble framework (CoREN), designed for tackling difficulties in grounding LLM-generated estimates to the target environment domain. The framework employs an adaptive ensemble of spatio-temporally consistent rewards to derive domain-grounded rewards in the training datasets, thus enabling effective offline learning of embodied agents in different environment domains. Experiments with the VirtualHome benchmark demonstrate that CoREN significantly outperforms other offline RL agents, and it also achieves comparable performance to state-of-the-art LLM-based agents with 8B parameters, despite CoREN having only 117M parameters for the agent policy network and using LLMs only for training., Comment: Findings of EMNLP-2024 Camera Ready Version

We propose GE2E-KWS -- a generalized end-to-end training and evaluation framework for customized keyword spotting. Specifically, enrollment utterances are separated and grouped by keywords from the training batch and their embedding centroids are compared to all other test utterance embeddings to compute the loss. This simulates runtime enrollment and verification stages, and improves convergence stability and training speed by optimizing matrix operations compared to SOTA triplet loss approaches. To benchmark different models reliably, we propose an evaluation process that mimics the production environment and compute metrics that directly measure keyword matching accuracy. Trained with GE2E loss, our 419KB quantized conformer model beats a 7.5GB ASR encoder by 23.6% relative AUC, and beats a same size triplet loss model by 60.7% AUC. Our KWS models are natively streamable with low memory footprints, and designed to continuously run on-device with no retraining needed for new keywords (zero-shot)., Comment: 8 pages, 6 figures, 2 tables The paper is accepted in IEEE Spoken Language Technology (SLT) 2024

As we enter the era of big data, collecting high-quality data is very important. However, collecting data by humans is not only very time-consuming but also expensive. Therefore, many scientists have devised various methods to collect data using computers. Among them, there is a method called web crawling, but the authors found that the crawling method has a problem in that unintended data is collected along with the user. The authors found that this can be filtered using the object recognition model YOLOv10. However, there are cases where data that is not properly filtered remains. Here, image reclassification was performed by additionally utilizing the distance output from the Siamese network, and higher performance was recorded than other classification models. (average \_f1 score YOLO+MobileNet 0.678->YOLO+SiameseNet 0.772)) The user can specify a distance threshold to adjust the balance between data deficiency and noise-robustness. The authors also found that the Siamese network can achieve higher performance with fewer resources because the cropped images are used for object recognition when processing images in the Siamese network. (Class 20 mean-based f1 score, non-crop+Siamese(MobileNetV3-Small) 80.94 -> crop preprocessing+Siamese(MobileNetV3-Small) 82.31) In this way, the image retrieval system that utilizes two consecutive models to reduce errors can save users' time and effort, and build better quality data faster and with fewer resources than before., Comment: 15 pages, 13 figures, 2 tables

We introduce ET-Former, a novel end-to-end algorithm for semantic scene completion using a single monocular camera. Our approach generates a semantic occupancy map from single RGB observation while simultaneously providing uncertainty estimates for semantic predictions. By designing a triplane-based deformable attention mechanism, our approach improves geometric understanding of the scene than other SOTA approaches and reduces noise in semantic predictions. Additionally, through the use of a Conditional Variational AutoEncoder (CVAE), we estimate the uncertainties of these predictions. The generated semantic and uncertainty maps will aid in the formulation of navigation strategies that facilitate safe and permissible decision-making in the future. Evaluated on the Semantic-KITTI dataset, ET-Former achieves the highest IoU and mIoU, surpassing other methods by 15.16% in IoU and 24.24% in mIoU, while reducing GPU memory usage of existing methods by 25%-50.5%.

Large language models (LLMs) have demonstrated remarkable performance in the legal domain, with GPT-4 even passing the Uniform Bar Exam in the U.S. However their efficacy remains limited for non-standardized tasks and tasks in languages other than English. This underscores the need for careful evaluation of LLMs within each legal system before application. Here, we introduce KBL, a benchmark for assessing the Korean legal language understanding of LLMs, consisting of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the Korean bar exam (4 domains, 53 tasks, 2,510 examples). First two datasets were developed in close collaboration with lawyers to evaluate LLMs in practical scenarios in a certified manner. Furthermore, considering legal practitioners' frequent use of extensive legal documents for research, we assess LLMs in both a closed book setting, where they rely solely on internal knowledge, and a retrieval-augmented generation (RAG) setting, using a corpus of Korean statutes and precedents. The results indicate substantial room and opportunities for improvement., Comment: EMNLP 2024 Findings

We present the spatially resolved relationship between the dust-to-gas mass ratio (DGR) and gas-phase metallicity (Zgas or 12+log(O/H)) (i.e., DGR-Zgas relation) of 11 nearby galaxies with a large metallicity range (1.5 dex of 12+log(O/H)) at (sub-)kpc scales. We used the large field-of-view (> 3') optical pseudo-Integral Field Spectroscopy data taken by the TYPHOON/PrISM survey, covering the optical size of galaxies, combining them with multi-wavelength data (far-UV to far-IR, CO, and HI 21 cm radio). A large scatter of DGR in the intermediate metallicity galaxies (8.0 < 12+log(O/H) < 8.3) is found, which is in line with dust evolution models, where grain growth begins to dominate the mechanism of dust mass accumulation. In the lowest metallicity galaxy of our sample, Sextans A (12+log(O/H) < 7.6), the star-forming regions have significantly higher DGR values (by 0.5-2 dex) than the global estimates from literature at the same metallicity but aligns with the DGR values from metal depletion method from Damped Lyman Alpha systems and high hydrogen gas density regions of Sextans A. Using dust evolution models with a Bayesian MCMC approach suggests: 1) a high SN dust yield and 2) a negligible amount of photofragmentation by UV radiation, although we note that our sample in the low-metallicity regime is limited to Sextans A. On the other hand, it is also possible that while metallicity influences DGR, gas density also plays a role, indicating an early onset of dust grain growth in the dust mass build-up process despite its low metallicity., Comment: 20 pages, 10 figures, 7 tables, accepted for publication in MNRAS

Digital pathology has made significant advances in tumor diagnosis and segmentation, but image variability due to differences in organs, tissue preparation, and acquisition - known as domain shift - limits the effectiveness of current algorithms. The COSAS (Cross-Organ and Cross-Scanner Adenocarcinoma Segmentation) challenge addresses this issue by improving the resilience of segmentation algorithms to domain shift, with Task 2 focusing on adenocarcinoma segmentation using a diverse dataset from six scanners, pushing the boundaries of clinical diagnostics. Our approach employs unsupervised learning through stain separation within a multi-task learning framework using a multi-decoder autoencoder. This model isolates stain matrix and stain density, allowing it to handle color variation and improve generalization across scanners. We further enhanced the robustness of the model with a mixture of stain augmentation techniques and used a U-net architecture for segmentation. The novelty of our method lies in the use of stain separation within a multi-task learning framework, which effectively disentangles histological structures from color variations. This approach shows promise for improving segmentation accuracy and generalization across different histopathological stains, paving the way for more reliable diagnostic tools in digital pathology.

Most star formation in the local Universe occurs in spiral galaxies, but their origin remains an unanswered question. Various theories have been proposed to explain the development of spiral arms, each predicting different spatial distributions of the interstellar medium. This study maps the star formation rate (SFR) and gas-phase metallicity of nine spiral galaxies with the TYPHOON survey to test two dominating theories: density wave theory and dynamic spiral theory. We discuss the environmental effects on our galaxies, considering reported environments and merging events. Taking advantage of the large field of view covering the entire optical disk, we quantify the fluctuation of SFR and metallicity relative to the azimuthal distance from the spiral arms. We find higher SFR and metallicity in the trailing edge of NGC~1365 (by 0.117~dex and 0.068~dex, respectively) and NGC~1566 (by 0.119~dex and 0.037~dex, respectively), which is in line with density wave theory. NGC~2442 shows a different result with higher metallicity (0.093~dex) in the leading edge, possibly attributed to an ongoing merging. The other six spiral galaxies show no statistically significant offset in SFR or metallicity, consistent with dynamic spiral theory. We also compare the behaviour of metallicity inside and outside the co-rotation radius (CR) of NGC~1365 and NGC~1566. We find comparable metallicity fluctuations near and beyond the CR of NGC~1365, indicating gravitational perturbation. NGC~1566 shows the greatest fluctuation near the CR, in line with the analytic spiral arms. Our work highlights that a combination of mechanisms explains the origin of spiral features in the local Universe., Comment: 22 pages, 10 figures, 4 tables, accepted for publication in MNRAS

The keyword spotting (KWS) problem requires large amounts of real speech training data to achieve high accuracy across diverse populations. Utilizing large amounts of text-to-speech (TTS) synthesized data can reduce the cost and time associated with KWS development. However, TTS data may contain artifacts not present in real speech, which the KWS model can exploit (overfit), leading to degraded accuracy on real speech. To address this issue, we propose applying an adversarial training method to prevent the KWS model from learning TTS-specific features when trained on large amounts of TTS data. Experimental results demonstrate that KWS model accuracy on real speech data can be improved by up to 12% when adversarial loss is used in addition to the original KWS loss. Surprisingly, we also observed that the adversarial setup improves accuracy by up to 8%, even when trained solely on TTS and real negative speech data, without any real positive examples., Comment: to be published in a Workshop at Interspeech 2024, Synthetic Data's Transformative Role in Foundational Speech Models

This paper explores the use of TTS synthesized training data for KWS (keyword spotting) task while minimizing development cost and time. Keyword spotting models require a huge amount of training data to be accurate, and obtaining such training data can be costly. In the current state of the art, TTS models can generate large amounts of natural-sounding data, which can help reducing cost and time for KWS model development. Still, TTS generated data can be lacking diversity compared to real data. To pursue maximizing KWS model accuracy under the constraint of limited resources and current TTS capability, we explored various strategies to mix TTS data and real human speech data, with a focus on minimizing real data use and maximizing diversity of TTS output. Our experimental results indicate that relatively small amounts of real audio data with speaker diversity (100 speakers, 2k utterances) and large amounts of TTS synthesized data can achieve reasonably high accuracy (within 3x error rate of baseline), compared to the baseline (trained with 3.8M real positive utterances)., Comment: to be published in a Workshop at Interspeech 2024, Synthetic Data's Transformative Role in Foundational Speech Models

One of the challenges in developing a high quality custom keyword spotting (KWS) model is the lengthy and expensive process of collecting training data covering a wide range of languages, phrases and speaking styles. We introduce Synth4Kws - a framework to leverage Text to Speech (TTS) synthesized data for custom KWS in different resource settings. With no real data, we found increasing TTS phrase diversity and utterance sampling monotonically improves model performance, as evaluated by EER and AUC metrics over 11k utterances of the speech command dataset. In low resource settings, with 50k real utterances as a baseline, we found using optimal amounts of TTS data can improve EER by 30.1% and AUC by 46.7%. Furthermore, we mix TTS data with varying amounts of real data and interpolate the real data needed to achieve various quality targets. Our experiments are based on English and single word utterances but the findings generalize to i18n languages and other keyword types., Comment: 5 pages, 5 figures, 2 tables The paper is accepted in Interspeech SynData4GenAI 2024 Workshop - https://syndata4genai.org/#call-for-papers

In this paper, we introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response, marking the initial step towards creating an avatar chatbot system without relying on intermediate text. To this end, we newly introduce MultiDialog, the first large-scale multimodal (i.e., audio and visual) spoken dialogue corpus containing 340 hours of approximately 9,000 dialogues, recorded based on the open domain dialogue dataset, TopicalChat. The MultiDialog contains parallel audio-visual recordings of conversation partners acting according to the given script with emotion annotations, which we expect to open up research opportunities in multimodal synthesis. Our Face-to-Face spoken dialogue model incorporates a textually pretrained large language model and adapts it into the audio-visual spoken dialogue domain by incorporating speech-text joint pretraining. Through extensive experiments, we validate the effectiveness of our model in facilitating a face-to-face conversation. Demo and data are available at https://multidialog.github.io and https://huggingface.co/datasets/IVLLab/MultiDialog, respectively., Comment: Accepted to ACL 2024 (Oral)

This paper explores the conditions for determining fixed nodes in structured networks, specifically focusing on directed acyclic graphs (DAGs). We introduce several necessary and sufficient conditions for determining fixed nodes in $p$-layered DAGs. This is accomplished by defining the problem of maximum disjoint stems, based on the observation that all DAGs can be represented as hierarchical structures with a unique label for each layer. For structured networks, we discuss the importance of fixed nodes by considering their controllability against the variations of network parameters. Moreover, we present an efficient algorithm that simultaneously performs labeling and fixed node search for $p$-layered DAGs with an analysis of its time complexity. The results presented in this paper have implications for the analysis of controllability at the individual node level in structured networks.

This paper presents new results and reinterpretation of existing conditions for strong structural controllability in a structured network determined by the zero/non-zero patterns of edges. For diffusively-coupled networks with self-loops, we first establish a necessary and sufficient condition for strong structural controllability, based on the concepts of dedicated and sharing nodes. Subsequently, we define several conditions for strong structural controllability across various graph types by decomposing them into disjoint path graphs. We further extend our findings by introducing a composition rule, facilitating the analysis of strong structural controllability in larger networks. This rule allows us to determine the strong structural controllability of connected graphs called pactus graphs (a generalization of the well-known cactus graph) by consideration of the strong structural controllability of its disjoint component graphs. In this process, we introduce the notion of a component input node, which is a state node that functions identically to an external input node. Based on this concept, we present an algorithm with approximate polynomial complexity to determine the minimum number of external input nodes required to maintain strong structural controllability in a diffusively-coupled network with self-loops., Comment: 11 pages, 5 figures. arXiv admin note: substantial text overlap with arXiv:2205.05275

Using adaptive optics with the Multi-Unit Spectroscopic Explorer (MUSE) on the Very Large Telescope (VLT), the Middle Ages Galaxy Properties with Integral Field Spectroscopy (MAGPI) survey allows us to study the spatially resolved Universe at a crucial time of ~4 Gyr ago ($z$ ~ 0.3) when simulations predict the greatest diversity in evolutionary pathways for galaxies. We investigate the radial trends in the star formation (SF) activity and luminosity-weighted stellar ages as a function of offset from the star-forming main sequence (SFMS) for a total of 294 galaxies. Using both H$\alpha$ emission and the 4000 Angstrom break (i.e., D4000) as star formation rate (SFR) tracers, we find overall flat radial profiles for galaxies lying on and above the SFMS, suggestive of physical processes that enhance/regulate SF throughout the entire galaxy disc. However, for galaxies lying below the SFMS, we find positive gradients in SF suggestive of inside-out quenching. Placing our results in context with results from other redshift regimes suggests an evolution in radial trends at $z$ ~ 0.3 for SF galaxies above the SFMS, from uniformly enhanced SF at $z$ ~ 1 and $z$ ~ 0.3 to centrally enhanced SF at $z$ ~ 0 (when averaged over a wide range of mass). We also capture higher local SFRs for galaxies below the SFMS compared to that of $z$ ~ 0, which can be explained by a larger population of quenched satellites in the local Universe and/or different treatments of limitations set by the D4000-sSFR relation., Comment: 19 pages, 15 figures, 4 tables, accepted for publication in MNRAS

Understanding the behaviors of ecological systems is challenging given their multi-faceted complexity. To proceed, theoretical models such as Lotka-Volterra dynamics with random interactions have been investigated by the dynamical mean-field theory to provide insights into underlying principles such as how biodiversity and stability depend on the randomness in interaction strength. Yet the fully-connected structure assumed in these previous studies is not realistic as revealed by a vast amount of empirical data. We derive a generic formula for the abundance distribution under an arbitrary distribution of degree, the number of interacting neighbors, which leads to degree-dependent abundance patterns of species. Notably, in contrast to the well-mixed system, the number of surviving species can be reduced as the community becomes cooperative in heterogeneous interaction structures. Our study, therefore, demonstrates that properly taking into account heterogeneity in the interspecific interaction structure is indispensable to understanding the diversity in large ecosystems, and our general theoretical framework can apply to a much wider range of interacting many-body systems.

Providing emotional support through dialogue systems is becoming increasingly important in today's world, as it can support both mental health and social interactions in many conversation scenarios. Previous works have shown that using persona is effective for generating empathetic and supportive responses. They have often relied on pre-provided persona rather than inferring them during conversations. However, it is not always possible to obtain a user persona before the conversation begins. To address this challenge, we propose PESS (Persona Extraction through Semantic Similarity), a novel framework that can automatically infer informative and consistent persona from dialogues. We devise completeness loss and consistency loss based on semantic similarity scores. The completeness loss encourages the model to generate missing persona information, and the consistency loss guides the model to distinguish between consistent and inconsistent persona. Our experimental results demonstrate that high-quality persona information inferred by PESS is effective in generating emotionally supportive responses., Comment: Accepted by ICASSP2024

Within the context of structured networks, this paper introduces the concept of the Fixed Strongly Structurally Controllable Subspace (FSSCS), enabling a comprehensive characterization of controllable subspaces. From a graph-theoretical viewpoint, the paper defines Fixed Strongly Structurally Controllable (FSSC) nodes based on the FSSCS concept and establishes the necessary and sufficient conditions for their identification. This paper proposes a method for determining the exact dimension of the Strongly Structurally Controllable Subspace (SSCS) in hierarchical directed acyclic graphs, employing a blend of graph-theoretical approaches and controllability matrix analyses. This approach not only facilitates the identification of FSSC nodes but also enhances our understanding of the robustness of node controllability against variations in network parameters within structured networks, marking a significant advancement in the field of strong structural controllability of individual nodes.