Search

Text-to-image (T2I) diffusion models are popular for introducing image manipulation methods, such as editing, image fusion, inpainting, etc. At the same time, image-to-video (I2V) and text-to-video (T2V) models are also built on top of T2I models. We present Kandinsky 3, a novel T2I model based on latent diffusion, achieving a high level of quality and photorealism. The key feature of the new architecture is the simplicity and efficiency of its adaptation for many types of generation tasks. We extend the base T2I model for various applications and create a multifunctional generation system that includes text-guided inpainting/outpainting, image fusion, text-image fusion, image variations generation, I2V and T2V generation. We also present a distilled version of the T2I model, evaluating inference in 4 steps of the reverse process without reducing image quality and 3 times faster than the base model. We deployed a user-friendly demo system in which all the features can be tested in the public domain. Additionally, we released the source code and checkpoints for the Kandinsky 3 and extended models. Human evaluations show that Kandinsky 3 demonstrates one of the highest quality scores among open source generation systems., Comment: Accepted for EMNLP 2024 (Demo track)

Proving formula depth lower bounds is a fundamental challenge in complexity theory, with the strongest known bound of $(3 - o(1))\log n$ established by Hastad over 25 years ago. The Karchmer--Raz--Wigderson (KRW) conjecture offers a promising approach to advance these bounds and separate P from NC$^{1}$. It suggests that the depth complexity of a function composition $f \diamond g$ approximates the sum of the depth complexities of $f$ and $g$. The Karchmer--Wigderson (KW) relation framework translates formula depth into communication complexity, restating the KRW conjecture as $\mathsf{CC}(\mathsf{KW}_f \diamond \mathsf{KW}_g) \approx \mathsf{CC}(\mathsf{KW}_f) + \mathsf{CC}(\mathsf{KW}_g)$. Prior work has confirmed the conjecture under various relaxations, often replacing one or both KW relations with the universal relation or constraining the communication game through strong composition. In this paper, we examine the strong composition $\mathsf{KW}_{\mathsf{XOR}} \circledast \mathsf{KW}_f$ of the parity function and a random Boolean function $f$. We prove that with probability $1-o(1)$, any protocol solving this composition requires at least $n^{3 - o(1)}$ leaves. This result establishes a depth lower bound of $(3 - o(1))\log n$, matching Hastad's bound, but is applicable to a broader class of inner functions, even when the outer function is simple. Though bounds for the strong composition do not translate directly to formula depth bounds, they usually help to analyze the standard composition (of the corresponding two functions) which is directly related to formula depth.

In this work the Coulomb effects (Coulomb correlations) in $\pi^+\pi^-$ pairs produced in p + Ni collisions at 24 GeV/$c$, are studied using experimental $\pi^+\pi^-$ pair distributions in $Q$, the relative momentum in the pair center of mass system (c.m.s), and its projections $Q_L$ (longitudinal component) and $Q_t$ (transverse component) relative to the pair direction in the laboratory system (l.s.). The $Q$, $Q_L$, and $Q_t$ distributions of the {\sl Coulomb pairs} in the c.m.s. have been simulated assuming they are described by the phase space modified by the known point-like Coulomb correlation function $A_C(Q)$, corrected for small effects due to the nonpoint-like pair production and the strong two-pion interaction. The same distributions of {\sl non-Coulomb pairs} have been simulated according to the phase space, but without $A_C(Q)$. It is shown that the number of {\sl Coulomb pairs} in all $Q_t$ intervals, including the small $Q_t$ (small opening angles $\theta$ in the l.s.) is calculated with the theoretical precision better than 2\%. The comparison of the simulated and experimental numbers of {\sl Coulomb pairs} at small $Q_t$ allows us to check and correct the detection efficiency for the pairs with small $\theta$ (0.06 mrad and smaller). It is shown that {\sl Coulomb pairs} can be used as a new physical tool to check and correct the quality of the simulated events. The special property of the {\sl Coulomb pairs} is the possibility of checking and correcting the detection efficiency, especially for the pairs with small opening angles.

Fortran's prominence in scientific computing requires strategies to ensure both that legacy codes are efficient on high-performance computing systems, and that the language remains attractive for the development of new high-performance codes. Coarray Fortran (CAF), part of the Fortran 2008 standard introduced for parallel programming, facilitates distributed memory parallelism with a syntax familiar to Fortran programmers, simplifying the transition from single-processor to multi-processor coding. This research focuses on innovating and refining a parallel programming methodology that fuses the strengths of Intel Coarray Fortran, Nvidia CUDA Fortran, and OpenMP for distributed memory parallelism, high-speed GPU acceleration and shared memory parallelism respectively. We consider the management of pageable and pinned memory, CPU-GPU affinity in NUMA multiprocessors, and robust compiler interfacing with speed optimisation. We demonstrate our method through its application to a parallelised Poisson solver and compare the methodology, implementation, and scaling performance to that of the Message Passing Interface (MPI), finding CAF offers similar speeds with easier implementation. For new codes, this approach offers a faster route to optimised parallel computing. For legacy codes, it eases the transition to parallel computing, allowing their transformation into scalable, high-performance computing applications without the need for extensive re-design or additional syntax., Comment: Accepted to the Journal of Parallel and Distributed Computing

Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.

Self-testing protocols enable the certification of quantum systems in a device-independent manner, i.e. without knowledge of the inner workings of the quantum devices under test. Here, we demonstrate this high standard for characterization routines with superconducting circuits, a prime platform for building large-scale quantum computing systems. We first develop the missing theory allowing for the self-testing of Pauli measurements. We then self-test Bell pair generation and measurements at the same time, performing a complete self-test in a system composed of two entangled superconducting circuits operated at a separation of 30 meters. In an experiment based on 17 million trials, we measure an average CHSH (Clauser-Horne-Shimony-Holt) S-value of 2.236. Without relying on additional assumptions on the experimental setup, we certify an average Bell state fidelity of at least 58.9% and an average measurement fidelity of at least 89.5% in a device-independent manner, both with 99% confidence. This enables applications in the field of distributed quantum computing and communication with superconducting circuits, such as delegated quantum computing.

Large language models (LLMs) can spend extra compute during inference to generate intermediate thoughts, which helps to produce better final responses. Since Chain-of-Thought (Wei et al., 2022), many such System 2 techniques have been proposed such as Rephrase and Respond (Deng et al., 2023a), System 2 Attention (Weston and Sukhbaatar, 2023) and Branch-Solve-Merge (Saha et al., 2023). In this work we investigate self-supervised methods to ``compile'' (distill) higher quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences, as this reasoning has been distilled into System 1. We show that several such techniques can be successfully distilled, resulting in improved results compared to the original System 1 performance, and with less inference cost than System 2. We posit that such System 2 distillation will be an important feature of future continually learning AI systems, enabling them to focus System 2 capabilities on the reasoning tasks that they cannot yet do well.

Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT., Comment: Accepted to Interspeech 2024

Aligned instruction following models can better fulfill user requests than their unaligned counterparts. However, it has been shown that there is a length bias in evaluation of such models, and that training algorithms tend to exploit this bias by learning longer responses. In this work we show how to train models that can be controlled at inference time with instructions containing desired length constraints. Such models are superior in length instructed evaluations, outperforming standard instruction following models such as GPT4, Llama 3 and Mixtral., Comment: 13 pages

Recent advancements in human preference optimization, initially developed for Language Models (LMs), have shown promise for text-to-image Diffusion Models, enhancing prompt alignment, visual appeal, and user preference. Unlike LMs, Diffusion Models typically optimize in pixel or VAE space, which does not align well with human perception, leading to slower and less efficient training during the preference alignment stage. We propose using a perceptual objective in the U-Net embedding space of the diffusion model to address these issues. Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and supervised fine-tuning (SFT) within this embedding space. This method significantly outperforms standard latent-space implementations across various metrics, including quality and computational cost. For SDXL, our approach provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt following against original open-sourced SDXL-DPO on the PartiPrompts dataset, while significantly reducing compute. Our approach not only improves the efficiency and quality of human preference alignment for diffusion models but is also easily integrable with other optimization techniques. The training code and LoRA weights will be available here: https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1

One of the most striking findings in modern research on large language models (LLMs) is that scaling up compute during training leads to better results. However, less attention has been given to the benefits of scaling compute during inference. This survey focuses on these inference-time approaches. We explore three areas under a unified mathematical formalism: token-level generation algorithms, meta-generation algorithms, and efficient generation. Token-level generation algorithms, often called decoding algorithms, operate by sampling a single token at a time or constructing a token-level search space and then selecting an output. These methods typically assume access to a language model's logits, next-token distributions, or probability scores. Meta-generation algorithms work on partial or full sequences, incorporating domain knowledge, enabling backtracking, and integrating external information. Efficient generation methods aim to reduce token costs and improve the speed of generation. Our survey unifies perspectives from three research communities: traditional natural language processing, modern LLMs, and machine learning systems.

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST). Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it's pretraining process. Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment. Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments., Comment: Accepted to ACL 2024 (findings)

The Spin Physics Detector collaboration proposes to install a universal detector in the second interaction point of the NICA collider under construction (JINR, Dubna) to study the spin structure of the proton and deuteron and other spin-related phenomena using a unique possibility to operate with polarized proton and deuteron beams at a collision energy up to 27 GeV and a luminosity up to $10^{32}$ cm$^{-2}$ s$^{-1}$. As the main goal, the experiment aims to provide access to the gluon TMD PDFs in the proton and deuteron, as well as the gluon transversity distribution and tensor PDFs in the deuteron, via the measurement of specific single and double spin asymmetries using different complementary probes such as charmonia, open charm, and prompt photon production processes. Other polarized and unpolarized physics is possible, especially at the first stage of NICA operation with reduced luminosity and collision energy of the proton and ion beams. This document is dedicated exclusively to technical issues of the SPD setup construction.

Text-to-image (T2I) diffusion models achieve state-of-the-art results in image synthesis and editing. However, leveraging such pretrained models for video editing is considered a major challenge. Many existing works attempt to enforce temporal consistency in the edited video through explicit correspondence mechanisms, either in pixel space or between deep features. These methods, however, struggle with strong nonrigid motion. In this paper, we introduce a fundamentally different approach, which is based on the observation that spatiotemporal slices of natural videos exhibit similar characteristics to natural images. Thus, the same T2I diffusion model that is normally used only as a prior on video frames, can also serve as a strong prior for enhancing temporal consistency by applying it on spatiotemporal slices. Based on this observation, we present Slicedit, a method for text-based video editing that utilizes a pretrained T2I diffusion model to process both spatial and spatiotemporal slices. Our method generates videos that retain the structure and motion of the original video while adhering to the target text. Through extensive experiments, we demonstrate Slicedit's ability to edit a wide range of real-world videos, confirming its clear advantages compared to existing competing methods. Webpage: https://matankleiner.github.io/slicedit/, Comment: ICML 2024. Code and examples are available at https://matankleiner.github.io/slicedit/

We prove that for the functions of the form $g(x) = h(x) + \frac{C}{x+i}$, where $h$ belongs to the continuous Wiener algebra $W_0$, the intersection of the frame set $\mathcal{F}_g$ with every hyperbola $\{\alpha, \beta > 0 \mid \alpha\beta = c\}$ is open in the relative topology. In particular, this applies to all rational functions $g$., Comment: 7 pages

The SS-HORSE-NCSM method is generalized to the case of democratic decay into an odd number of fragments. This method is applied to the search for resonances in three-neutron system (trineutron) using ab initio No-Core Shell Model calculations with realistic nucleon-nucleon potentials. The $3/2^-$ and $1/2^-$ strongly overlapping resonances are predicted when softened $NN$ interactions are used and are preferred over the case where bare $NN$ interactions of the chiral effective field theory are used with no resonance obtained., Comment: 7 pages, 2 figures

Magnetic resonance methods offer a unique chance for in-depth study of conductive organic material systems, not only accounts for number of charge carriers, but also allows manipulations of spin dynamics of particles. Here we present a study of continuous-wave electrically detected magnetic resonance on a range of organic conjugate polymers under transistor architecture, with tunability in both carrier concentration and drifting electric field. We demonstrate the general existence of bipolaron between mobile and trapped charges at the magnetic resonance condition, then estimates the wavefunction expansion of charge carriers on their hopping sites along conductive pathways. This finding allows direct observation of energetically disordered trap sites under the charge hopping picture, linking-up the microscopic structural properties to the bulk electronic performances., Comment: No ready for publishing, need more about discussion about the wavefunctions

There have been emerging research interest and advances in speech-to-speech translation (S2ST), translating utterances from one language to another. This work proposes Multitask Speech Language Model (MSLM), which is a decoder-only speech language model trained in a multitask setting. Without reliance on text training data, our model is able to support multilingual S2ST with speaker style preserved.

Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and non-autoregressive (NAR) speech LMs and provide insights into the prompt design and content semantic units. Our analysis reveals that heterogeneous and nonstationary prompts hurt the audio quality in contrast to the previous finding that longer prompts always lead to better synthesis. Moreover, we find that the speaker style of the synthesized audio is also affected by the content in addition to the prompt. We further show that semantic units carry rich acoustic information such as pitch, tempo, volume and speech emphasis, which might be leaked from the content to the synthesized audio.

This paper investigates a time-varying version of weak-form market efficiency in the BRICS countries. A moving window test for sample autocorrelations is applied alongside a Kalman filter approach to recover the hidden dynamics of the market efficiency process through appropriate time-varying autoregressive models with both homoscedastic and heteroscedastic conditional variance. Monthly data covers the period from January 1995 to December 2020, which includes the 2008-2009 global financial crisis and the recent COVID-19 recession. The results reveal that all the BRICS stock markets were affected during both periods, but generally remained weak-form efficient, with the exception of China., Comment: arXiv admin note: text overlap with arXiv:2310.04125

In this paper, we continue to study the derivative-free extended Kalman filtering (DF-EKF) framework for state estimation of continuous-discrete nonlinear stochastic systems. Having considered the Euler-Maruyama and It\^{o}-Taylor discretization schemes for solving stochastic differential equations, we derive the related filters' moment equations based on the derivative-free EKF principal. In contrast to the recently derived MATLAB-based continuous-discrete DF-EKF techniques, the novel DF-EKF methods preserve an information about the underlying stochastic process and provide the estimation procedure for a fixed number of iterates at the propagation steps. Additionally, the DF-EKF approach is particularly effective for working with stochastic systems with highly nonlinear and/or nondifferentiable drift and observation functions, but the price to be paid is its degraded numerical stability (to roundoff) compared to the standard EKF framework. To eliminate the mentioned pitfall of the derivative-free EKF methodology, we develop the conventional algorithms together with their stable square-root implementation methods. In contrast to the published DF-EKF results, the new square-root techniques are derived within both the Cholesky and singular value decompositions. A performance of the novel filters is demonstrated on a number of numerical tests including well- and ill-conditioned scenarios.

More than 40 years ago, Schroeppel and Shamir presented an algorithm that solves the Subset Sum problem for $n$ integers in time $O^*(2^{0.5n})$ and space $O^*(2^{0.25n})$. The time upper bound remains unbeaten, but the space upper bound has been improved to $O^*(2^{0.249999n})$ in a recent breakthrough paper by Nederlof and W\k{e}grzycki (STOC 2021). Their algorithm is a clever combination of a number of previously known techniques with a new reduction and a new algorithm for the Orthogonal Vectors problem. In this paper, we improve the space bound by Nederlof and W\k{e}grzycki to $O^*(2^{0.246n})$ and also simplify their algorithm and its analysis. We achieve this by using an idea, due to Howgrave-Graham and Joux, of using a random prime number to filter the family of subsets. We incorporate it into the algorithm by Schroeppel and Shamir and then use this amalgam inside the representation technique. This allows us to reduce an instance of Subset Sum to a larger number of instances of weighted orthogonal vector.

One of the modern research lines in econometrics studies focuses on translating a wide variety of structural econometric models into their state-space form, which allows for efficient unknown dynamic system state and parameter estimations by the Kalman filtering scheme. The mentioned trend yields advanced state-space model structures, which demand innovative estimation techniques driven by application requirements to be devised. This paper explores both the linear time-invariant multiple-input, multiple-output system (LTI MIMO) and the pairwise Markov model (PMM) with the related pairwise Kalman filter (PKF). In particular, we design robust gradient-based adaptive Kalman-like filtering methods for the simultaneous state and parameter estimation in the outlined model structures. Our methods are fast and accurate because their analytically computed gradient is utilized in the calculation instead of its numerical approximation. Also, these employ the numerically robust $UDU^\top$-factorization-based Kalman filter implementation, which is reliable in practice. Our novel techniques are examined on numerical examples and used for treating one stochastic model in econometrics.

In this paper, a singular value decomposition (SVD) approach is developed for implementing the cubature Kalman filter. The discussed estimator is one of the most popular and widely used method for solving nonlinear Bayesian filtering problem in practice. To improve its numerical stability (with respect to roundoff errors) and practical reliability of computations, the SVD-based methodology recently proposed for the classical Kalman filter is generalized on the nonlinear filtering problem. More precisely, we suggest the SVD-based solution for the continuous-discrete cubature Kalman filter and design two estimators: (i) the filter based on the traditionally used Euler-Maruyama discretization scheme; (ii) the estimator based on advanced It\^{o}-Taylor expansion for discretizing the underlying stochastic differential equations. Both estimators are formulated in terms of SVD factors of the filter error covariance matrix and belong to the class of stable factored-form (square-root) algorithms. The new methods are tested on a radar tracking problem.

In mathematical neuroscience, a special interest is paid to a working memory mechanism in the neural tissue modeled by the Dynamic Neural Field (DNF) in the presence of model uncertainties. The working memory facility implies that the neurons' activity remains self-sustained after the external stimulus removal due to the recurrent interactions in the networks and allows the system to cope with missing sensors' information. In our previous works, we have developed two reconstruction methods of the neural membrane potential from {\it incomplete} data available from the sensors. The methods are derived within the Extended Kalman filtering approach by using the Euler-Maruyama method and the It\^{o}-Taylor expansion of order 1.5. It was shown that the It\^{o}-Taylor EKF-based restoration process is more accurate than the Euler-Maruyama-based alternative. It improves the membrane potential reconstruction quality in case of incomplete sensors information. The aim of this paper is to investigate their pattern recognition facilities, i.e. the quality of the pattern formation reconstruction in case of model uncertainties and incomplete information. The numerical experiments are provided for an example of the stochastic DNF with multiple active zones arisen in a neural tissue.

Recent research in nonlinear filtering and signal processing has suggested an efficient derivative-free Extended Kalman filter (EKF) designed for discrete-time stochastic systems. Such approach, however, has failed to address the estimation problem for continuous-discrete models. In this paper, we develop a novel continuous-discrete derivative-free EKF methodology by deriving the related moment differential equations (MDEs) and sample point differential equations (SPDEs). Additionally, we derive their Cholesky-based square-root MDEs and SPDEs and obtain several numerically stable derivative-free EKF methods. Finally, we propose the MATLAB-oriented implementations for all continuous-discrete derivative-free EKF algorithms derived. They are easy to implement because of the built-in fashion of the MATLAB numerical integrators utilized for solving either the MDEs or SPDEs in use, which are the ordinary differential equations (ODEs). More importantly, these are accurate derivative-free EKF implementations because any built-in MATLAB ODE solver includes the discretization error control that bounds the discretization error arisen and makes the implementation methods accurate. Besides, this is done in automatic way and no extra coding is required from users. The new filters are particularly effective for working with stochastic systems with highly nonlinear and/or nondifferentiable drift and observation functions, i.e. when the calculation of Jacobian matrices are either problematical or questionable. The performance of the novel filtering methods is demonstrated on several numerical tests.

Aims. The initial stages of planet formation may start concurrently with the formation of a gas-dust protoplanetary disk. This makes the study of the earliest stages of protoplanetary disk formation crucially important. Here we focus on dust growth and pebble formation in a protoplanetary disk that is still accreting from a parental cloud core. Methods. We have developed an original three-dimensional numerical hydrodynamics code, which computes the collapse of rotating clouds and disk formation on nested meshes using a novel hybrid Coarray Fortran-OpenMP approach for distributed and shared memory parallelization. Dust dynamics and growth are also included in the simulations. Results. We found that the dust growth from $\sim 1~\mu$m to 1-10~mm already occurs in the initial few thousand years of disk evolution but the Stokes number hardly exceeds 0.1 because of higher disk densities and temperatures compared to the minimum mass Solar nebular. The ratio of the dust-to-gas vertical scale heights remains rather modest, 0.2--0.5, which may be explained by the perturbing action of spiral arms that develop in the disk soon after its formation. The dust-to-gas mass ratio in the disk midplane is highly nonhomogeneous throughout the disk extent and is in general enhanced by a factor of several compared to the fiducial 1:100 value. Low St hinders strong dust accumulation in the spiral arms compared to the rest of the disk and the nonsteady nature of the spirals is also an obstacle. The spatial distribution of pebbles in the disk midplane exhibits a highly nonhomogeneous and patchy character. The total mass of pebbles in the disk increases with time and reaches a few tens of Earth masses after a few tens of thousand years of disk evolution. Abridged., Comment: Accepted for publication by Astronomy & Astrophysics

The nuclear two-photon or double-gamma ($2\gamma$) decay is a second-order electromagnetic process whereby a nucleus in an excited state emits two gamma rays simultaneously. To be able to directly measure the $2\gamma$ decay rate in the low-energy regime below the electron-positron pair-creation threshold, we combined the isochronous mode of a storage ring with Schottky resonant cavities. The newly developed technique can be applied to isomers with excitation energies down to $\sim100$\,keV and half-lives as short as $\sim10$\,ms. The half-life for the $2\gamma$ decay of the first-excited $0^+$ state in bare $^{72}\mathrm{Ge}$ ions was determined to be $23.9\left(6\right)$\,ms, which strongly deviates from expectations., Comment: Submitted to Physical Review Letters, 8 pages, 3 figures, 2 tables, 7 equations

Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication

A stable square-root approach has been recently proposed for the unscented Kalman filter (UKF) and fifth-degree cubature Kalman filter (5D-CKF) as well as for the mixed-type methods consisting of the extended Kalman filter (EKF) time update and the UKF/5D-CKF measurement update steps. The mixed-type estimators provide a good balance in trading between estimation accuracy and computational demand because of the EKF moment differential equations involved. The key benefit is a consolidation of reliable state mean and error covariance propagation by using delicate discretization error control while solving the EKF moment differential equations and an accurate measurement update according to the advanced UKF and/or 5D-CKF filtering strategies. Meanwhile the drawback of the previously proposed estimators is an utilization of sophisticated numerical integration scheme with the built-in discretization error control that is, in fact, a complicated and computationally costly tool. In contrast, we design here the mixed-type methods that keep the same estimation quality but reduce a computational time significantly. The novel estimators elegantly utilize any MATLAB-based numerical integration scheme developed for solving ordinary differential equations (ODEs) with the required accuracy tolerance pre-defined by users. In summary, a simplicity of the suggested estimators, their numerical robustness with respect to roundoff due to the square-root form utilized as well as their estimation accuracy due to the MATLAB ODEs solvers with discretization error control involved are the attractive features of the novel estimators. The numerical experiments are provided for illustrating a performance of the suggested methods in comparison with the existing ones.