Search

Rotary Position Embedding (RoPE) encodes relative and absolute positional information in Transformer-based models through rotation matrices applied to input vectors within sequences. While RoPE has demonstrated superior performance compared to other positional embedding technologies in natural language processing tasks, its effectiveness in speech processing applications remains understudied. In this work, we conduct a comprehensive evaluation of RoPE across diverse automatic speech recognition (ASR) tasks. Our experimental results demonstrate that for ASR tasks, RoPE consistently achieves lower error rates compared to the currently widely used relative positional embedding. To facilitate further research, we release the implementation and all experimental recipes through the SpeechBrain toolkit.

In an experiment on a turbulent jet, we detect interfacial turbulent layers in a frame that moves, on average, along with the \tnti. This significantly prolongs the observation time of scalar and velocity structures and enables the measurement of two types of Lagrangian coherent structures. One structure, the finite-time Lyapunov field (FTLE), quantifies advective transport barriers of fluid parcels while the other structure highlights barriers of diffusive momentum transport. These two complementary structures depend on large-scale and small-scale motion and are therefore associated with the growth of the turbulent region through engulfment or nibbling, respectively. We detect the \tnti\ from cluster analysis, where we divide the measured scalar field into four clusters. Not only the \tnti\ can be found this way, but also the next, internal, turbulent-turbulent interface. Conditional averages show that these interfaces are correlated with barriers of advective and diffusive transport when the Lagrangian integration time is smaller than the integral time scale. Diffusive structures decorrelate faster since they have a smaller timescale. Conditional averages of these structures at internal turbulent-turbulent interfaces show the same pattern with a more pronounced jump at the interface indicative of a shear layer. This is quite an unexpected outcome, as the internal interface is now defined not by the presence or absence of vorticity, but by conditional vorticity corresponding to two uniform concentration zones. The long-time diffusive momentum flux along Lagrangian paths represents the growth of the turbulent flow into the irrotational domain, a direct demonstration of nibbling. The diffusive flux parallel to the \tnti\ appears to be concentrated in a diffusive superlayer whose width is comparable with the Taylor microscale, which is relatively invariant in time.

In this paper, we study the stability of a simple model of a Hyperloop vehicle resulting from the interaction between electromagnetic and aeroelastic forces for both constant and periodically varying coefficients (i.e., parametric excitation). For the constant coefficients, through linear stability analysis, we analytically identify three distinct regions for the physically significant equilibrium point. Further inspection reveals that the system exhibits limit-cycle vibrations in one of these regions. Using the harmonic balance method, we determine the properties of the limit cycle, thereby unravelling the frequency and amplitude that characterize the periodic oscillations of the system's variables. For the varying coefficients case, the stability is studied using Floquet analysis and Hills determinant method. The part of the stability boundary related to parametric resonance has an elliptical shape, while the remaining part remains unchanged. One of the major findings is that a linear parametric force, can suppress or amplify the parametric resonance induced by another parametric force depending on the amplitude of the former. In the context of the Hyperloop system, this means that parametric resonance caused by base excitation-in other words by the linearized parametric electromagnetic force can be suppressed by modulating the coefficient of the aeroelastic force in the same frequency. The effectiveness is highly dependent on the phase difference between the modulation and the base excitation. The origin of the suppression is attributed to the stabilizing character of the parametric aeroelastic force as revealed through energy analysis. We provide analytical expressions for the stability boundaries and for the stability's dependence on the phase shift of the modulation.

Detections of fast X-ray transients (FXTs) have been accrued over the last few decades. However, their origin has remained mysterious. There is now rapid progress thanks to timely discoveries and localisations with the Einstein Probe mission. Early results indicate that FXTs may frequently, but not always, be associated with gamma-ray bursts (GRBs). Here, we report on the multi-wavelength counterpart of FXT EP240414a, which has no reported gamma-ray counterpart. The transient is located 25.7~kpc in projection from a massive galaxy at $z=0.40$. We perform comprehensive photometric and spectroscopic follow-up. The optical light curve shows at least three distinct emission episodes with timescales of $\sim 1, 4$ and 15 days and peak absolute magnitudes of $M_R \sim -20$, $-21$, and $-19.5$, respectively. The optical spectrum at early times is extremely blue, inconsistent with afterglow emission. It may arise from the interaction of both jet and supernova shock waves with the stellar envelope and a dense circumstellar medium, as has been suggested for some Fast Blue Optical Transients (LFBOTs). At late times, the spectrum evolves to a broad-lined~Type~Ic supernova, similar to those seen in collapsar long-GRBs. This implies that the progenitor of EP240414a is a massive star creating a jet-forming supernova inside a dense envelope, resulting in an X-ray outburst with a luminosity of $\sim 10^{48}$ erg s$^{-1}$, and the complex observed optical/IR light curves. If correct, this argues for a causal link between the progenitors of long-GRBs, FXTs and LFBOTs., Comment: 36 pages, 13 figures, submitted to ApJ

Automatic speech recognition (ASR) with an encoder equipped with self-attention, whether streaming or non-streaming, takes quadratic time in the length of the speech utterance. This slows down training and decoding, increase their cost, and limit the deployment of the ASR in constrained devices. SummaryMixing is a promising linear-time complexity alternative to self-attention for non-streaming speech recognition that, for the first time, preserves or outperforms the accuracy of self-attention models. Unfortunately, the original definition of SummaryMixing is not suited to streaming speech recognition. Hence, this work extends SummaryMixing to a Conformer Transducer that works in both a streaming and an offline mode. It shows that this new linear-time complexity speech encoder outperforms self-attention in both scenarios while requiring less compute and memory during training and decoding.

Self-supervised learning (SSL) models usually require weeks of pre-training with dozens of high-end GPUs. These models typically have a multi-headed self-attention (MHSA) context encoder. However, MHSA takes quadratic time and space in the input length, contributing to the high pre-training cost. Linear-complexity alternatives to MHSA have been proposed. For instance, in supervised training, the SummaryMixing model is the first to outperform MHSA across multiple speech processing tasks. However, these cheaper alternatives have not been explored for SSL yet. This paper studies a linear-complexity context encoder for SSL for the first time. With better or equivalent performance for the downstream tasks of the MP3S benchmark, SummaryMixing reduces the pre-training time and peak VRAM of wav2vec 2.0 model by 18% and by 23%, respectively, leading to the pre-training of a 155M wav2vec 2.0 model finished within one week with 4 Tesla A100 GPUs. Code is available at https://github.com/SamsungLabs/SummaryMixing., Comment: Interspeech 2024

Federated learning (FL) allows clients to collaboratively train a global model without sharing their local data with a server. However, clients' contributions to the server can still leak sensitive information. Differential privacy (DP) addresses such leakage by providing formal privacy guarantees, with mechanisms that add randomness to the clients' contributions. The randomness makes it infeasible to train large transformer-based models, common in modern federated learning systems. In this work, we empirically evaluate the practicality of fine-tuning large scale on-device transformer-based models with differential privacy in a federated learning system. We conduct comprehensive experiments on various system properties for tasks spanning a multitude of domains: speech recognition, computer vision (CV) and natural language understanding (NLU). Our results show that full fine-tuning under differentially private federated learning (DP-FL) generally leads to huge performance degradation which can be alleviated by reducing the dimensionality of contributions through parameter-efficient fine-tuning (PEFT). Our benchmarks of existing DP-PEFT methods show that DP-Low-Rank Adaptation (DP-LoRA) consistently outperforms other methods. An even more promising approach, DyLoRA, which makes the low rank variable, when naively combined with FL would straightforwardly break differential privacy. We therefore propose an adaptation method that can be combined with differential privacy and call it DP-DyLoRA. Finally, we are able to reduce the accuracy degradation and word error rate (WER) increase due to DP to less than 2% and 7% respectively with 1 million clients and a stringent privacy budget of $\epsilon=2$., Comment: 16 pages, 10 figures, 5 tables

The nature of the minute-to-hour long Fast X-ray Transients (FXTs) localised by telescopes such as Chandra, Swift, and XMM-Newton remains mysterious, with numerous models suggested for the events. Here, we report multi-wavelength observations of EP240315a, a 1600 s long transient detected by the Einstein Probe, showing it to have a redshift of z=4.859. We measure a low column density of neutral hydrogen, indicating that the event is embedded in a low-density environment, further supported by direct detection of leaking ionising Lyman-continuum. The observed properties are consistent with EP240315a being a long-duration gamma-ray burst, and these observations support an interpretation in which a significant fraction of the FXT population are lower-luminosity examples of similar events. Such transients are detectable at high redshifts by the Einstein Probe and, in the (near) future, out to even larger distances by SVOM, THESEUS, and Athena, providing samples of events into the epoch of reionisation., Comment: 41 pages, 7 figures, submitted

Federated learning (FL) is an emerging machine learning (ML) training paradigm where clients own their data and collaborate to train a global model, without revealing any data to the server and other participants. Researchers commonly perform experiments in a simulation environment to quickly iterate on ideas. However, existing open-source tools do not offer the efficiency required to simulate FL on larger and more realistic FL datasets. We introduce pfl-research, a fast, modular, and easy-to-use Python framework for simulating FL. It supports TensorFlow, PyTorch, and non-neural network models, and is tightly integrated with state-of-the-art privacy algorithms. We study the speed of open-source FL frameworks and show that pfl-research is 7-72$\times$ faster than alternative open-source frameworks on common cross-device setups. Such speedup will significantly boost the productivity of the FL research community and enable testing hypotheses on realistic FL datasets that were previously too resource intensive. We release a suite of benchmarks that evaluates an algorithm's overall performance on a diverse set of realistic scenarios. The code is available on GitHub at https://github.com/apple/pfl-research.

Railway transition zones (RTZs) experience higher rates of degradation compared to open tracks, which leads to increased maintenance costs and reduced vailability. Despite existing literature on railway track assessment and maintenance, effective design solutions for RTZs are still limited. Therefore, a robust design criterion is required to develop effective solutions. This paper presents a two-step approach for formulation of a design criterion to delay the onset of processes leading to uneven track geometry due to operation driven permanent deformations in RTZs. Firstly, a systematic analysis of each track component in a RTZ is performed by examining spatial and temporal variations in kinematic responses, stresses and energies. Secondly, the study proposes an energy-based criterion to be assessed using a model with linear elastic material behavior, and states that an amplification in the total strain energy in the proximity of the transition interface is an indicator of increased (and thus non-uniform) degradation in RTZs compared to the open tracks. The correlation between the total strain energy (assessed in the model with linear material behaviour) and the permanent irreversible deformations is demonstrated using a model with non-linear elastoplastic material behavior of the ballast layer. In the end, it is claimed that minimising the magnitude of total strain energy will lead to reduced degradation and a uniform distribution of total strain energy in each trackbed layer along the longitudinal direction of the track will ensure uniformity in the track geometry.

The Transducer (e.g. RNN-Transducer or Conformer-Transducer) generates an output label sequence as it traverses the input sequence. It is straightforward to use in streaming mode, where it generates partial hypotheses before the complete input has been seen. This makes it popular in speech recognition. However, in streaming mode the Transducer has a mathematical flaw which, simply put, restricts the model's ability to change its mind. The fix is to replace local normalisation (e.g. a softmax) with global normalisation, but then the loss function becomes impossible to evaluate exactly. A recent paper proposes to solve this by approximating the model, severely degrading performance. Instead, this paper proposes to approximate the loss function, allowing global normalisation to apply to a state-of-the-art streaming model. Global normalisation reduces its word error rate by 9-11% relative, closing almost half the gap between streaming and lookahead mode., Comment: 9 pages plus references and appendices

Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28% faster and reducing memory use by half., Comment: Interspeech 2024

Long-distance travel has seen little attention in the past, largely due its sporadic nature. A single long-distance trip can amount to a distance equivalent to a year's worth of commute trips, resulting in a similar, if not worse, environmental footprint. Understanding travellers' behaviour is thus just as relevant for such trips. As international travel is slowly picking up from the COVID-19 pandemic, it has been marred by an abundance of national and regional pandemic-related safety measures. While their primary goal is to protect the local population from infection, these safety may also make travellers feel safer while travelling. This perceived safety can - and likely does - differ from the true efficacy of the measures. In this research, we investigate people's perception of eight COVID-19-related safety measures related to long-distance trips and how subjective perception of safety impacts their mode choice among car, train and aircraft. We employ a Hierarchical Information Integration (HII) approach to capture subjective perceptions and then model the obtained data by means of a Latent Class Choice Model, resulting in four distinct segments. To extrapolate the segments onto the rating experiment of HII, we apply a weighted least squares (WLS) regression, to obtain segment-specific safety perception. Two segments show a relatively high value-of-time (72EUR/h and 50EUR/h), tend to be more mode-agnostic and prefer determining the level of risk by themselves (relying primarily on infection and vaccination rate). The remaining two segments have a lower value-of-time (38EUR / h and 15EUR/h) and have strong mode affinity, for the train and car respectively. Future research could look into a way that segments the sample based on both the mode choice and rating experiment, providing additional insights into the heterogeneity of individuals in their perceptions.

Visual microscopic study of diseased tissue by pathologists has been the cornerstone for cancer diagnosis and prognostication for more than a century. Recently, deep learning methods have made significant advances in the analysis and classification of tissue images. However, there has been limited work on the utility of such models in generating histopathology images. These synthetic images have several applications in pathology including utilities in education, proficiency testing, privacy, and data sharing. Recently, diffusion probabilistic models were introduced to generate high quality images. Here, for the first time, we investigate the potential use of such models along with prioritized morphology weighting and color normalization to synthesize high quality histopathology images of brain cancer. Our detailed results show that diffusion probabilistic models are capable of synthesizing a wide range of histopathology images and have superior performance compared to generative adversarial networks.

Federated Learning (FL) is a technique to train models using data distributed across devices. Differential Privacy (DP) provides a formal privacy guarantee for sensitive data. Our goal is to train a large neural network language model (NNLM) on compute-constrained devices while preserving privacy using FL and DP. However, the DP-noise introduced to the model increases as the model size grows, which often prevents convergence. We propose Partial Embedding Updates (PEU), a novel technique to decrease noise by decreasing payload size. Furthermore, we adopt Low Rank Adaptation (LoRA) and Noise Contrastive Estimation (NCE) to reduce the memory demands of large models on compute-constrained devices. This combination of techniques makes it possible to train large-vocabulary language models while preserving accuracy and privacy.

Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users' devices without harming privacy. PFL is efficient for models, such as neural networks, that have a fixed number of parameters, and thus a fixed-dimensional gradient vector. Such models include neural-net language models, but not tokenizers, the topic of this work. Training a tokenizer requires frequencies of words from an unlimited vocabulary, and existing methods for finding an unlimited vocabulary need a separate privacy budget. A workaround is to train the tokenizer on publicly available data. However, in this paper we first show that a tokenizer trained on mismatched data results in worse model performance compared to a privacy-violating "oracle" tokenizer that accesses user data, with perplexity increasing by 20%. We also show that sub-word tokenizers are better suited to the federated context than word-level ones, since they can encode new words, though with more tokens per word. Second, we propose a novel method to obtain a tokenizer without using any additional privacy budget. During private federated learning of the language model, we sample from the model, train a new tokenizer on the sampled sequences, and update the model embeddings. We then continue private federated learning, and obtain performance within 1% of the "oracle" tokenizer. Since this process trains the tokenizer only indirectly on private data, we can use the "postprocessing guarantee" of differential privacy and thus use no additional privacy budget.

Federated learning with differential privacy, or private federated learning, provides a strategy to train machine learning models while respecting users' privacy. However, differential privacy can disproportionately degrade the performance of the models on under-represented groups, as these parts of the distribution are difficult to learn in the presence of noise. Existing approaches for enforcing fairness in machine learning models have considered the centralized setting, in which the algorithm has access to the users' data. This paper introduces an algorithm to enforce group fairness in private federated learning, where users' data does not leave their devices. First, the paper extends the modified method of differential multipliers to empirical risk minimization with fairness constraints, thus providing an algorithm to enforce fairness in the central setting. Then, this algorithm is extended to the private federated learning setting. The proposed algorithm, \texttt{FPFL}, is tested on a federated version of the Adult dataset and an "unfair" version of the FEMNIST dataset. The experiments on these datasets show how private federated learning accentuates unfairness in the trained models, and how FPFL is able to mitigate such unfairness., Comment: Presented at PriML workshop at NeurIPS 2021. 20 pages: 11 of main content, 3 of references, and 6 of supplementary material

The sound identification of refactoring opportunities is still an open problem in software engineering. Recent studies have shown the effectiveness of machine learning models in recommending methods that should undergo different refactoring operations. In this work, we experiment with such approaches to identify methods that should undergo an Extract Method refactoring, in the context of ING, a large financial organization. More specifically, we (i) compare the code metrics distributions, which are used as features by the models, between open-source and ING systems, (ii) measure the accuracy of different machine learning models in recommending Extract Method refactorings, (iii) compare the recommendations given by the models with the opinions of ING experts. Our results show that the feature distributions of ING systems and open-source systems are somewhat different, that machine learning models can recommend Extract Method refactorings with high accuracy, and that experts tend to agree with most of the recommendations of the model.