Search

Simultaneous machine translation (SMT) takes streaming input utterances and incrementally produces target text. Existing SMT methods only use the partial utterance that has already arrived at the input and the generated hypothesis. Motivated by human interpreters' technique to forecast future words before hearing them, we propose $\textbf{T}$ranslation by $\textbf{A}$nticipating $\textbf{F}$uture (TAF), a method to improve translation quality while retraining low latency. Its core idea is to use a large language model (LLM) to predict future source words and opportunistically translate without introducing too much risk. We evaluate our TAF and multiple baselines of SMT on four language directions. Experiments show that TAF achieves the best translation quality-latency trade-off and outperforms the baselines by up to 5 BLEU points at the same latency (three words)., Comment: Under review

Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting-where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.

The Central Molecular Zone (CMZ) is the largest reservoir of dense molecular gas in the Galaxy and is heavily obscured in the optical and near-IR. We present an overview of the far-IR dust continuum, where the molecular clouds are revealed, provided by Herschel in the inner 40\deg($|l| <$ 20\deg) of the Milky Way with a particular focus on the CMZ. We report a total dense gas ($N$(H$_2$) $> 10^{23}$ cm$^{-2}$) CMZ mass of M=$2\substack{+2 \\ -1} \times 10^7$ M$_{\odot}$ and confirm that there is a highly asymmetric distribution of dense gas, with about 70-75% at positive longitudes. We create and publicly release complete fore/background-subtracted column density and dust temperature maps in the inner 40\deg ($|l| <$ 20\deg) of the Galaxy. We find that the CMZ clearly stands out as a distinct structure, with an average mass per longitude that is at least $3\times$ higher than the rest of the inner Galaxy contiguously from 1.8\deg $> \ell >$ -1.3\deg. This CMZ extent is larger than previously assumed, but is consistent with constraints from velocity information. The inner Galaxy's column density peaks towards the SgrB2 complex with a value of about 2 $\times$ 10$^{24}$ cm$^{-2}$, and typical CMZ molecular clouds are about N(H$_2$)=10$^{23}$ cm$^{-2}$. Typical CMZ dust temperatures range from about $12-35$ K with relatively little variation. We identify a ridge of warm dust in the inner CMZ that potentially traces the base of the northern Galactic outflow seen with MEERKAT., Comment: Submitted to ApJ, 16 pages, project website: https://centralmolecularzone.github.io/3D_CMZ/

The Central Molecular Zone (CMZ) is the way station at the heart of our Milky Way Galaxy, connecting gas flowing in from Galactic scales with the central nucleus. Key open questions remain about its 3-D structure, star formation properties, and role in regulating this gas inflow. In this work, we identify a hierarchy of discrete structures in the CMZ using column density and dust temperature maps from Paper I (Battersby et al., submitted). We calculate the physical ($N$(H$_2$), $T_{\rm{dust}}$, mass, radius) and kinematic (HNCO, HCN, and HC$_3$N moments) properties of each structure as well as their bolometric luminosities and star formation rates (SFRs). We compare these properties with regions in the Milky Way disk and external galaxies. We perform power-law fits to the column density probability distribution functions (N-PDFs) of the inner 100 pc, SgrB2, and the outer 100 pc of the CMZ as well as several individual molecular cloud structures and find generally steeper power-law slopes ($-9<\alpha<-2$) compared with the literature ($-6 < \alpha < -1$). We find that individual CMZ structures require a large external pressure ($P_e$/k$_B$ $> 10^{7-9}$ K cm$^{-3}$) to be considered bound. Despite the fact that the CMZ overall is well below the Gao-Solomon dense gas star-formation relation (and in modest agreement with the Schmidt-Kennicutt relation), individual structures on the scale of molecular clouds generally follow these star-formation relations and agree well with other Milky Way and extragalactic regions., Comment: Submitted to ApJ, 22 pages, project website: https://centralmolecularzone.github.io/3D_CMZ/

A comprehensive 3-D model of the central 300 pc of the Milky Way, the Central Molecular Zone (CMZ) is of fundamental importance in understanding energy cycles in galactic nuclei, since the 3-D structure influences the location and intensity of star formation, feedback, and black hole accretion. Current observational constraints are insufficient to distinguish between existing 3-D models. Dust extinction is one diagnostic tool that can help determine the location of dark molecular clouds relative to the bright Galactic Center emission. By combining Herschel and Spitzer observations, we developed three new dust extinction techniques to estimate the likely near/far locations for each cloud in the CMZ. We compare our results to four geometric CMZ orbital models. Our extinction methods show good agreement with each other, and with results from spectral line absorption analysis from Walker et al. (submitted). Our near/far results for CMZ clouds are inconsistent with a projected version of the Sofue (1995) two spiral arms model, and show disagreement in position-velocity space with the Molinari et al. (2011) closed elliptical orbit. Our results are in reasonable agreement with the Kruijssen et al. (2015) open streams. We find that a simplified toy-model elliptical orbit which conserves angular momentum shows promising fits in both position-position and position-velocity space. We conclude that all current CMZ orbital models lack the complexity needed to describe the motion of gas in the CMZ, and further work is needed to construct a complex orbital model to accurately describe gas flows in the CMZ., Comment: Submitted to ApJ; 36 pages; Project series website with relevant code, data, and results: https://centralmolecularzone.github.io/3D_CMZ/

The Milky Way's Central Molecular Zone (CMZ) is the largest concentration of dense molecular gas in the Galaxy, the structure of which is shaped by the complex interplay between Galactic-scale dynamics and extreme physical conditions. Understanding the 3-D geometry of this gas is crucial as it determines the locations of star formation and subsequent feedback. We present a catalogue of clouds in the CMZ using Herschel data. Using archival data from the APEX and MOPRA CMZ surveys, we measure averaged kinematic properties of the clouds at 1mm and 3mm. We use archival ATCA data of the H$_{2}$CO (1$_{1,0}$ - 1$_{1,1}$) 4.8 GHz line to search for absorption towards the clouds, and 4.85 GHz GBT C-band data to measure the radio continuum emission. We measure the absorption against the continuum to provide new constraints for the line-of-sight positions of the clouds relative to the Galactic centre, and find a highly asymmetric distribution, with most clouds residing in front of the Galactic centre. The results are compared with different orbital models, and we introduce a revised toy model of a vertically-oscillating closed elliptical orbit. We find that most models describe the PPV structure of the gas reasonably well, but find significant inconsistencies in all cases regarding the near vs. far placement of individual clouds. Our results highlight that the CMZ is likely more complex than can be captured by these simple geometric models, along with the need for new data to provide further constraints on the true 3-D structure of the CMZ., Comment: Submitted to ApJ, 30 pages, project website: https://centralmolecularzone.github.io/3D_CMZ/

Observations suggest that filaments in molecular clouds can grow by mass accretion while forming cores via fragmentation. Here we present one of the first large sample studies of filament accretion using velocity gradient measurements of star-forming filaments on the $\sim 0.05$ pc scale with NH$_3$ observations of the Perseus Molecular Cloud, primarily obtained as a part of the GBT Ammonia Survey (GAS). In this study, we find significant correlations between velocity gradient, velocity dispersion, mass per unit length, and the number of cores per unit length of the Perseus filaments. Our results suggest a scenario in which filaments not only grow through mass accretion but also form new cores continuously in the process well into the thermally supercritical regime. Such behavior is contrary to that expected from isolated filament models but consistent with how filaments form within a more realistic cloud environment, suggesting that the cloud environment plays a crucial role in shaping core formation and evolution in filaments. Furthermore, even though velocity gradients within filaments are not oriented randomly, we find no correlation between velocity gradient orientation and the filament properties we analyzed. This result suggests that gravity is unlikely the dominant mechanism imposing order on the $\sim 0.05$ pc scale for dense star-forming gas., Comment: Accepted to ApJ (Oct 17, 2024)

ALMA-IMF observed 15 massive protoclusters capturing multiple spectral lines and the continuum emission. We focus on the G351.77 protocluster ($\sim$ 2500 M$_{\odot}$, estimated from single-dish continuum observations) located at 2 kpc. We trace the dense gas emission and kinematics with N$_2$H$^+$ (1-0) at $\sim$ 4 kau resolution. We estimate an N$_2$H$^+$ relative abundance $\sim (1.7 \pm 0.5) \times 10^{-10}$. We decompose the N$_2$H$^+$ emission into up to two velocity components, highlighting the kinematic complexity in the dense gas. By examining the position-velocity (PV) diagrams on small scales, we observe clear inflow signatures (V-shapes) associated with dense cores. The most prominent V-shape has a mass inflow rate of $\sim 9.8 \times 10^{-4}$ M$_{\odot}$ yr$^{-1}$ and a short timescale of $\sim$ 15.6 kyr. We also observe V-shapes without associated cores. This suggests both that cores or centers of accretion exist below the 1.3 mm detection limit, and that the V-shapes may be viable tracers of very early accretion and star formation on $\sim$ 4 kau scales. The large-scale PV diagram shows that the protocluster is separated into 2 principal velocity structures. Combined with smaller scale DCN and H$_2$CO emission, we propose a scenario of larger scale slow contraction with rotation in the center based on simple toy models. This scenario leads the suggestion of outside-in evolution of the protocluster as it collapses. The gas depletion times implied by the V-shapes are short ($\sim$ 0.3 Myr), requiring either very fast cluster formation, and/or continuous mass feeding of the protocluster. The latter is possible via the Mother Filament G351.77 is forming out of. The similarities in the properties of G351.77 and the recently published work in G353.41 indicate that many of the physical conditions inferred via the ALMA-IMF N$_2$H$^+$ observations may be generic to protoclusters., Comment: Submitted in A&A, 28 pages, 31 figures, 4 interactive figures, 7 tables

We present James Webb Space Telescope (JWST)-NIRCam observations of the massive star-forming molecular cloud Sagittarius C (Sgr C) in the Central Molecular Zone (CMZ). In conjunction with ancillary mid-IR and far-IR data, we characterize the two most massive protostars in Sgr C via spectral energy distribution (SED) fitting, estimating that they each have current masses of $m_* \sim 20\:M_\odot$ and surrounding envelope masses of $\sim 100\:M_\odot$. We report a census of lower-mass protostars in Sgr C via a search for infrared counterparts to mm continuum dust cores found with ALMA. We identify 88 molecular hydrogen outflow knot candidates originating from outflows from protostars in Sgr C, the first such unambiguous detections in the infrared in the CMZ. About a quarter of these are associated with flows from the two massive protostars in Sgr C; these extend for over 1 pc and are associated with outflows detected in ALMA SiO line data. An additional $\sim 40$ features likely trace shocks in outflows powered by lower-mass protostars throughout the cloud. We report the discovery of a new star-forming region hosting two prominent bow shocks and several other line-emitting features driven by at least two protostars. We infer that one of these is forming a high-mass star given an SED-derived mass of $m_* \sim 9\:M_\odot$ and associated massive ($\sim 90\:M_\odot$) mm core and water maser. Finally, we identify a population of miscellaneous Molecular Hydrogen Objects (MHOs) that do not appear to be associated with protostellar outflows., Comment: Appendix figures B1 and B2 will be made into online-only figure sets for the eventual ApJ publication

The Central Molecular Zone (CMZ) of the Milky Way is fed by gas inflows from the Galactic disk along almost radial trajectories aligned with the major axis of the Galactic bar. However, despite being fundamental to all processes in the nucleus of the galaxy, these inflows have been studied significantly less than the CMZ itself. We present observations of various molecular lines between 215 and 230 GHz for 20 clouds with $|\ell| < 10^\circ$, which are candidates for clouds in the Galactic bar due to their warm temperatures and broad lines relative to typical Galactic disk clouds, using the Atacama Large Millimeter/submillimeter Array (ALMA) Atacama Compact Array (ACA). We measure gas temperatures, shocks, star formation rates, turbulent Mach numbers, and masses for these clouds. Although some clouds may be in the Galactic disk despite their atypical properties, nine clouds are likely associated with regions in the Galactic bar, and in these clouds, turbulent pressure is suppressing star formation. In clouds with no detected star formation, turbulence is the dominant heating mechanism, whereas photo-electric processes heat the star-forming clouds. We find that the ammonia (NH3) and formaldehyde (H2CO) temperatures probe different gas components, and in general each transition appears to trace different molecular gas phases within the clouds. We also measure the CO-to-H2 X-factor in the bar to be an order of magnitude lower than the typical Galactic value. These observations provide evidence that molecular clouds achieve CMZ-like properties before reaching the CMZ, Comment: 43 pages, 22 figures. Accepted to ApJ

We present \textbf{H}ybrid-\textbf{A}utoregressive \textbf{IN}ference Tr\textbf{AN}sducers (HAINAN), a novel architecture for speech recognition that extends the Token-and-Duration Transducer (TDT) model. Trained with randomly masked predictor network outputs, HAINAN supports both autoregressive inference with all network components and non-autoregressive inference without the predictor. Additionally, we propose a novel semi-autoregressive inference paradigm that first generates an initial hypothesis using non-autoregressive inference, followed by refinement steps where each token prediction is regenerated using parallelized autoregression on the initial hypothesis. Experiments on multiple datasets across different languages demonstrate that HAINAN achieves efficiency parity with CTC in non-autoregressive mode and with TDT in autoregressive mode. In terms of accuracy, autoregressive HAINAN outperforms TDT and RNN-T, while non-autoregressive HAINAN significantly outperforms CTC. Semi-autoregressive inference further enhances the model's accuracy with minimal computational overhead, and even outperforms TDT results in some cases. These results highlight HAINAN's flexibility in balancing accuracy and speed, positioning it as a strong candidate for real-world speech recognition applications.

We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs) by incorporating pre-trained speech models. However, these SLMs often undergo extensive speech instruction-tuning to bridge the gap between speech and text modalities. This requires significant annotation efforts and risks catastrophic forgetting of the original language capabilities. In this work, we present a simple yet effective automatic process for creating speech-text pair data that carefully injects speech paralinguistic understanding abilities into SLMs while preserving the inherent language capabilities of the text-based LLM. Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data, achieving impressive performance on Dynamic-SUPERB and AIR-Bench-Chat benchmarks. Furthermore, our model exhibits the ability to follow complex instructions derived from LLMs, such as specific output formatting and chain-of-thought reasoning. Our approach not only enhances the versatility and effectiveness of SLMs but also reduces reliance on extensive annotated datasets, paving the way for more efficient and capable speech understanding systems., Comment: Submitted to ICASSP 2025

A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time., Comment: 4 pages, submitted to ICASSP 2025

We propose a novel end-to-end multi-talker automatic speech recognition (ASR) framework that enables both multi-speaker (MS) ASR and target-speaker (TS) ASR. Our proposed model is trained in a fully end-to-end manner, incorporating speaker supervision from a pre-trained speaker diarization module. We introduce an intuitive yet effective method for masking ASR encoder activations using output from the speaker supervision module, a technique we term Meta-Cat (meta-information concatenation), that can be applied to both MS-ASR and TS-ASR. Our results demonstrate that the proposed architecture achieves competitive performance in both MS-ASR and TS-ASR tasks, without the need for traditional methods, such as neural mask estimation or masking at the audio or feature level. Furthermore, we demonstrate a glimpse of a unified dual-task model which can efficiently handle both MS-ASR and TS-ASR tasks. Thus, this work illustrates that a robust end-to-end multi-talker ASR framework can be implemented with a streamlined architecture, obviating the need for the complex speaker filtering mechanisms employed in previous studies.

The Milky Way's Central Molecular Zone (CMZ) differs dramatically from our local solar neighbourhood, both in the extreme interstellar medium conditions it exhibits (e.g. high gas, stellar, and feedback density) and in the strong dynamics at play (e.g. due to shear and gas influx along the bar). Consequently, it is likely that there are large-scale physical structures within the CMZ that cannot form elsewhere in the Milky Way. In this paper, we present new results from the Atacama Large Millimeter/submillimeter Array (ALMA) large programme ACES (ALMA CMZ Exploration Survey) and conduct a multi-wavelength and kinematic analysis to determine the origin of the M0.8$-$0.2 ring, a molecular cloud with a distinct ring-like morphology. We estimate the projected inner and outer radii of the M0.8$-$0.2 ring to be 79" and 154", respectively (3.1 pc and 6.1 pc at an assumed Galactic Centre distance of 8.2 kpc) and calculate a mean gas density $> 10^{4}$ cm$^{-3}$, a mass of $\sim$ $10^6$ M$_\odot$, and an expansion speed of $\sim$ 20 km s$^{-1}$, resulting in a high estimated kinetic energy ($> 10^{51}$ erg) and momentum ($> 10^7$ M$_\odot$ km s$^{-1}$). We discuss several possible causes for the existence and expansion of the structure, including stellar feedback and large-scale dynamics. We propose that the most likely cause of the M0.8$-$0.2 ring is a single high-energy hypernova explosion. To viably explain the observed morphology and kinematics, such an explosion would need to have taken place inside a dense, very massive molecular cloud, the remnants of which we now see as the M0.8$-$0.2 ring. In this case, the structure provides an extreme example of how supernovae can affect molecular clouds., Comment: 17 pages, 13 figures, and 2 tables. Published in Astronomy & Astrophysics

Large language models (LLMs) have demonstrated remarkable advancements in language understanding and generation. Building on the success of text-based LLMs, recent research has adapted these models to use speech embeddings for prompting, resulting in Speech-LLM models that exhibit strong performance in automatic speech recognition (ASR) and automatic speech translation (AST). In this work, we propose a novel approach to leverage ASR transcripts as prompts for AST in a Speech-LLM built on an encoder-decoder text LLM. The Speech-LLM model consists of a speech encoder and an encoder-decoder structure Megatron-T5. By first decoding speech to generate ASR transcripts and subsequently using these transcripts along with encoded speech for prompting, we guide the speech translation in a two-step process like chain-of-thought (CoT) prompting. Low-rank adaptation (LoRA) is used for the T5 LLM for model adaptation and shows superior performance to full model fine-tuning. Experimental results show that the proposed CoT prompting significantly improves AST performance, achieving an average increase of 2.4 BLEU points across 6 En->X or X->En AST tasks compared to speech prompting alone. Additionally, compared to a related CoT prediction method that predicts a concatenated sequence of ASR and AST transcripts, our method performs better by an average of 2 BLEU points.

Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model. To explore new capabilities in language modeling for speech processing, we introduce the generative speech transcription error correction (GenSEC) challenge. This challenge comprises three post-ASR language modeling tasks: (i) post-ASR transcription correction, (ii) speaker tagging, and (iii) emotion recognition. These tasks aim to emulate future LLM-based agents handling voice-based interfaces while remaining accessible to a broad audience by utilizing open pretrained language models or agent-based APIs. We also discuss insights from baseline evaluations, as well as lessons learned for designing future evaluations., Comment: IEEE SLT 2024. The initial draft version has been done in December 2023. Post-ASR Text Processing and Understanding Community and LlaMA-7B pre-training correction model: https://huggingface.co/GenSEC-LLM/SLT-Task1-Llama2-7b-HyPo-baseline

We propose Sortformer, a novel neural model for speaker diarization, trained with unconventional objectives compared to existing end-to-end diarization models. The permutation problem in speaker diarization has long been regarded as a critical challenge. Most prior end-to-end diarization systems employ permutation invariant loss (PIL), which optimizes for the permutation that yields the lowest error. In contrast, we introduce Sort Loss, which enables a diarization model to autonomously resolve permutation, with or without PIL. We demonstrate that combining Sort Loss and PIL achieves performance competitive with state-of-the-art end-to-end diarization models trained exclusively with PIL. Crucially, we present a streamlined multispeaker ASR architecture that leverages Sortformer as a speaker supervision model, embedding speaker label estimation within the ASR encoder state using a sinusoidal kernel function. This approach resolves the speaker permutation problem through sorted objectives, effectively bridging speaker-label timestamps and speaker tokens. In our experiments, we show that the proposed multispeaker ASR architecture, enhanced with speaker supervision, improves performance via adapter techniques. Code and trained models will be made publicly available via the NVIDIA NeMo framework

This paper presents a new method for training sequence-to-sequence models for speech recognition and translation tasks. Instead of the traditional approach of training models on short segments containing only lowercase or partial punctuation and capitalization (PnC) sentences, we propose training on longer utterances that include complete sentences with proper punctuation and capitalization. We achieve this by using the FastConformer architecture which allows training 1 Billion parameter models with sequences up to 60 seconds long with full attention. However, while training with PnC enhances the overall performance, we observed that accuracy plateaus when training on sequences longer than 40 seconds across various evaluation settings. Our proposed method significantly improves punctuation and capitalization accuracy, showing a 25% relative word error rate (WER) improvement on the Earnings-21 and Earnings-22 benchmarks. Additionally, training on longer audio segments increases the overall model accuracy across speech recognition and translation benchmarks. The model weights and training code are open-sourced though NVIDIA NeMo., Comment: Accepted at SLT 2024

Speech foundation models have achieved state-of-the-art (SoTA) performance across various tasks, such as automatic speech recognition (ASR) in hundreds of languages. However, multi-speaker ASR remains a challenging task for these models due to data scarcity and sparsity. In this paper, we present approaches to enable speech foundation models to process and understand multi-speaker speech with limited training data. Specifically, we adapt a speech foundation model for the multi-speaker ASR task using only telephonic data. Remarkably, the adapted model also performs well on meeting data without any fine-tuning, demonstrating the generalization ability of our approach. We conduct several ablation studies to analyze the impact of different parameters and strategies on model performance. Our findings highlight the effectiveness of our methods. Results show that less parameters give better overall cpWER, which, although counter-intuitive, provides insights into adapting speech foundation models for multi-speaker ASR tasks with minimal annotated data., Comment: Accepted by SLT 2024

This review paper describes the energy-upgraded CEBAF accelerator. This superconducting linac has achieved 12 GeV beam energy by adding 11 new high-performance cryomodules containing eighty-eight superconducting cavities that have operated CW at an average accelerating gradient of 20 MV/m. After reviewing the attributes and performance of the previous 6 GeV CEBAF accelerator, we discuss the upgraded CEBAF accelerator system in detail with particular attention paid to the new beam acceleration systems. In addition to doubling the acceleration in each linac, the upgrade included improving the beam recirculation magnets, adding more helium cooling capacity to allow the newly installed modules to run cold, adding a new experimental hall, and improving numerous other accelerator components. We review several of the techniques deployed to operate and analyze the accelerator performance, and document system operating experience and performance. In the final portion of the document, we present much of the current planning regarding projects to improve accelerator performance and enhance operating margins, and our plans for ensuring CEBAF operates reliably into the future. For the benefit of potential users of CEBAF, the performance and quality measures for beam delivered to each of the experimental halls is summarized in the appendix., Comment: 66 pages, 73 figures, 21 tables

Self-supervised learning has been proved to benefit a wide range of speech processing tasks, such as speech recognition/translation, speaker verification and diarization, etc. However, most of current approaches are computationally expensive. In this paper, we propose a simplified and more efficient self-supervised learning framework termed as NeMo Encoder for Speech Tasks (NEST). Specifically, we adopt the FastConformer architecture with 8x sub-sampling rate, which is faster than Transformer or Conformer architectures. Instead of clustering-based quantization, we use fixed random projection for its simplicity and effectiveness. We also implement a generalized noisy speech augmentation that teaches the model to disentangle the main speaker from noise or other speakers. Experiments show that \model improves over existing self-supervised models and achieves new state-of-the-art performance on a variety of speech processing tasks, such as speech recognition/translation, speaker diarization, spoken language understanding, etc. Code and checkpoints will be publicly available via NVIDIA NeMo framework.

We present ALMA observations of SiO, SiS, H$_2$O , NaCl, and SO line emission at ~30 to 50 mas resolution. These images map the molecular outflow and disk of Orion Source I (SrcI) on ~12 to 20 AU scales. Our observations show that the flow of material around SrcI creates a turbulent boundary layer in the outflow from SrcI which may dissipate angular momentum in the rotating molecular outflow into the surrounding medium. Additionally, the data suggests that the proper motion of SrcI may have a significant effect on the structure and evolution of SrcI and its molecular outflow. As the motion of SrcI funnels material between the disk and the outflow, some material may be entrained into the outflow and accrete onto the disk, creating shocks which excite the NaCl close to the disk surface., Comment: Accepted in The Astrophysical Journal

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Large Language Models (LLMs) rely on instruction samples for alignment, but creating these datasets poses challenges, particularly in expert-dependent tasks like coding, which can be cost-prohibitive. One approach to mitigate these challenges is synthesizing data using another LLM. In this paper, we introduce a scalable method for generating synthetic instructions to enhance the code generation capability of LLMs. The proposed algorithm, Genetic-Instruct, mimics evolutionary processes, utilizing self-instruction to create numerous synthetic samples from a limited number of seeds. Genetic-Instruct is designed for efficient scaling of the generation process. Fine-tuning multiple coding LLMs with the synthetic samples demonstrates a significant improvement in their code generation accuracy compared to the baselines.

The stellar initial mass function (IMF) is critical to our understanding of star formation and the effects of young stars on their environment. On large scales, it enables us to use tracers such as UV or Halpha emission to estimate the star formation rate of a system and interpret unresolved star clusters across the universe. So far, there is little firm evidence of large-scale variations of the IMF, which is thus generally considered universal. Stars form from cores and it is now possible to estimate core masses and compare the core mass function (CMF) with the IMF, which it presumably produces. The goal of the ALMA-IMF large program is to measure the core mass function at high linear resolution (2700 au) in 15 typical Milky Way protoclusters spanning a mass range of 2500 to 32700 Msun. In this work, we used two different core extraction algorithms to extract about 680 gravitationally bound cores from these 15 protoclusters. We adopt per core temperature using the temperature estimate from the PPMAP Bayesian method. A power-law fit to the CMF of the sub-sample of cores above the 1.64 Msun completeness limit, 330 cores, through the maximum likelihood estimate technique yields a slope of 1.97 +/- 0.06, significantly flatter than the 2.35 Salpeter slope. Assuming a self-similar mapping between the CMF and the IMF, this result implies that these 15 high-mass protoclusters will generate atypical IMFs. This sample is the largest to date produced and analysed self-consistently, derived at matched physical resolution, with per-core temperature estimates and cores as massive as 150 Msun. We provide the raw source extraction catalogues and the source derived size, temperature, mass, and spectral indices in the 15 protoclusters., Comment: Accepted by A&A

This paper proposes a generative speech enhancement model based on Schr\"odinger bridge (SB). The proposed model is employing a tractable SB to formulate a data-to-data process between the clean speech distribution and the observed noisy speech distribution. The model is trained with a data prediction loss, aiming to recover the complex-valued clean speech coefficients, and an auxiliary time-domain loss is used to improve training of the model. The effectiveness of the proposed SB-based model is evaluated in two different speech enhancement tasks: speech denoising and speech dereverberation. The experimental results demonstrate that the proposed SB-based outperforms diffusion-based models in terms of speech quality metrics and ASR performance, e.g., resulting in relative word error rate reduction of 20% for denoising and 6% for dereverberation compared to the best baseline model. The proposed model also demonstrates improved efficiency, achieving better quality than the baselines for the same number of sampling steps and with a reduced computational cost.

A crucial aspect in addressing the challenge of measuring the core mass function, that is pivotal for comprehending the origin of the initial mass function, lies in constraining the temperatures of the cores. We aim to measure the luminosity, mass, column density and dust temperature of star-forming regions imaged by the ALMA-IMF large program. High angular resolution mapping is required to capture the properties of protostellar and pre-stellar cores and to effectively separate them from larger features, such as dusty filaments. We employed the point process mapping (PPMAP) technique, enabling us to perform spectral energy distribution fitting of far-infrared and submillimeter observations across the 15 ALMA-IMF fields, at an unmatched 2.5" angular resolution. By combining the modified blackbody model with near-infrared data, we derived bolometric luminosity maps. We estimated the errors impacting values of each pixel in the temperature, column density, and luminosity maps. Subsequently, we employed the extraction algorithm getsf on the luminosity maps in order to detect luminosity peaks and measure their associated masses. We obtained high-resolution constraints on the luminosity, dust temperature, and mass of protoclusters, that are in agreement with previously reported measurements made at a coarser angular resolution. We find that the luminosity-to-mass ratio correlates with the evolutionary stage of the studied regions, albeit with intra-region variability. We compiled a PPMAP source catalog of 313 luminosity peaks using getsf on the derived bolometric luminosity maps. The PPMAP source catalog provides constraints on the mass and luminosity of protostars and cores, although one source may encompass several objects. Finally, we compare the estimated luminosity-to-mass ratio of PPMAP sources with evolutionary tracks and discuss the limitations imposed by the 2.5" beam., Comment: 35 pages, 19 figures, 5 tables Accepted by A&A

We use the H$41\alpha$ recombination line to create templates of the millimeter free-free emission in the ALMA-IMF continuum maps, which allows to separate it from dust emission. This method complements spectral-index information and extrapolation from centimeter wavelength maps. We use the derived maps to estimate the properties of up to 34 HII regions across the ALMA-IMF protoclusters. The hydrogen ionizing-photon rate $Q_0$ and spectral types follow the evolutionary trend proposed by Motte et al. The youngest protoclusters lack detectable ionized gas, followed by protoclusters with increasing numbers of OB stars. The total $Q_0$ increases from $\sim 10^{45}$ s$^{-1}$ to $> 10^{49}$ s$^{-1}$. We used the adjacent He$41\alpha$ line to measure the relative number abundances of helium, finding values consistent with the Galactic interstellar medium, although a few outliers are discussed. A search for sites of maser amplification of the H$41\alpha$ line returned negative results. We looked for possible correlations between the electron densities ($n_e$), emission measures (EM), and $Q_0$ with HII region size $D$. The latter are the better correlated, with $Q_0 \propto D^{2.49\pm0.18}$. This favors interpretations where smaller ultracompact HII regions are not necessarily the less dynamically evolved versions of larger ones, but rather are ionized by less massive stars. Moderate correlations were found between dynamical width $\Delta V_\mathrm{dyn}$ with $D$ and $Q_0$. $\Delta V_\mathrm{dyn}$ increases from about one to two times the ionized-gas sound speed. Finally, an outlier HII region south of W43-MM2 is discussed. We suggest that this source could harbor an embedded stellar or disk wind., Comment: Accepted to AAS Journals (ApJS). Data available at https://dataverse.harvard.edu/dataverse/h41a-freefree

We introduce romanization encoding for script-heavy languages to optimize multilingual and code-switching Automatic Speech Recognition (ASR) systems. By adopting romanization encoding alongside a balanced concatenated tokenizer within a FastConformer-RNNT framework equipped with a Roman2Char module, we significantly reduce vocabulary and output dimensions, enabling larger training batches and reduced memory consumption. Our method decouples acoustic modeling and language modeling, enhancing the flexibility and adaptability of the system. In our study, applying this method to Mandarin-English ASR resulted in a remarkable 63.51% vocabulary reduction and notable performance gains of 13.72% and 15.03% on SEAME code-switching benchmarks. Ablation studies on Mandarin-Korean and Mandarin-Japanese highlight our method's strong capability to address the complexities of other script-heavy languages, paving the way for more versatile and effective multilingual ASR systems.

Discrete speech representations have garnered recent attention for their efficacy in training transformer-based models for various speech-related tasks such as automatic speech recognition (ASR), translation, speaker verification, and joint speech-text foundational models. In this work, we present a comprehensive analysis on building ASR systems with discrete codes. We investigate different methods for codec training such as quantization schemes and time-domain vs spectral feature encodings. We further explore ASR training techniques aimed at enhancing performance, training efficiency, and noise robustness. Drawing upon our findings, we introduce a codec ASR pipeline that outperforms Encodec at similar bit-rate. Remarkably, it also surpasses the state-of-the-art results achieved by strong self-supervised models on the 143 languages ML-SUPERB benchmark despite being smaller in size and pretrained on significantly less data., Comment: Accepted at Interspeech 2024

Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categorized as: i) GPT-style, prepend speech prompts to the text prompts as a sequence of LLM inputs like a decoder-only model; ii) T5-style, introduce speech cross-attention to each layer of the pretrained LLMs. We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities. Moreover, there is no clear streaming solution for either style, especially considering the solution should generalize to speech multitask. We reformulate streamable SpeechLLM as a read-write policy problem and unifies the offline and streaming research with BESTOW architecture. Hence we demonstrate the first open-source SpeechLLM solution that enables Streaming and Multitask at scale (beyond ASR) at the same time. This streamable solution achieves very strong performance on a wide range of speech tasks (ASR, AST, SQA, unseen DynamicSuperb). It is end-to-end optimizable, with lower training/inference cost, and demonstrates LLM knowledge transferability to speech.

Recent advances in speech recognition and translation rely on hundreds of thousands of hours of Internet speech data. We argue that state-of-the art accuracy can be reached without relying on web-scale data. Canary - multilingual ASR and speech translation model, outperforms current state-of-the-art models - Whisper, OWSM, and Seamless-M4T on English, French, Spanish, and German languages, while being trained on an order of magnitude less data than these models. Three key factors enables such data-efficient model: (1) a FastConformer-based attention encoder-decoder architecture (2) training on synthetic data generated with machine translation and (3) advanced training techniques: data-balancing, dynamic data blending, dynamic bucketing and noise-robust fine-tuning. The model, weights, and training code will be open-sourced., Comment: Accepted at Interspeech-2024

Recent speech language models (SLMs) typically incorporate pre-trained speech models to extend the capabilities from large language models (LLMs). In this paper, we propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities, enabling SLMs to interpret and generate comprehensive natural language descriptions, thereby facilitating the capability to understand both linguistic and non-linguistic features in speech. Enhanced with the proposed approach, our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. Moreover, we discover that the aligned model exhibits a zero-shot instruction-following capability without explicit speech instruction tuning. These findings highlight the potential to reshape instruction-following SLMs by incorporating rich, descriptive speech captions., Comment: Accepted to Interspeech 2024

Large Language Model (LLM) based text-to-speech (TTS) systems have demonstrated remarkable capabilities in handling large speech datasets and generating natural speech for new speakers. However, LLM-based TTS models are not robust as the generated output can contain repeating words, missing words and mis-aligned speech (referred to as hallucinations or attention errors), especially when the text contains multiple occurrences of the same token. We examine these challenges in an encoder-decoder transformer model and find that certain cross-attention heads in such models implicitly learn the text and speech alignment when trained for predicting speech tokens for a given text. To make the alignment more robust, we propose techniques utilizing CTC loss and attention priors that encourage monotonic cross-attention over the text tokens. Our guided attention training technique does not introduce any new learnable parameters and significantly improves robustness of LLM-based TTS models., Comment: Published as a conference paper at INTERSPEECH 2024

In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages., Comment: Accepted for Interspeech 2024

We release the Nemotron-4 340B model family, including Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. Our models are open access under the NVIDIA Open Model License Agreement, a permissive model license that allows distribution, modification, and use of the models and its outputs. These models perform competitively to open access models on a wide range of evaluation benchmarks, and were sized to fit on a single DGX H100 with 8 GPUs when deployed in FP8 precision. We believe that the community can benefit from these models in various research studies and commercial applications, especially for generating synthetic data to train smaller language models. Notably, over 98% of data used in our model alignment process is synthetically generated, showcasing the effectiveness of these models in generating synthetic data. To further support open research and facilitate model development, we are also open-sourcing the synthetic data generation pipeline used in our model alignment process.

Accurate recognition of rare and new words remains a pressing problem for contextualized Automatic Speech Recognition (ASR) systems. Most context-biasing methods involve modification of the ASR model or the beam-search decoding algorithm, complicating model reuse and slowing down inference. This work presents a new approach to fast context-biasing with CTC-based Word Spotter (CTC-WS) for CTC and Transducer (RNN-T) ASR models. The proposed method matches CTC log-probabilities against a compact context graph to detect potential context-biasing candidates. The valid candidates then replace their greedy recognition counterparts in corresponding frame intervals. A Hybrid Transducer-CTC model enables the CTC-WS application for the Transducer model. The results demonstrate a significant acceleration of the context-biasing recognition with a simultaneous improvement in F-score and WER compared to baseline methods. The proposed method is publicly available in the NVIDIA NeMo toolkit., Comment: Accepted by Interspeech 2024

This paper introduces a highly efficient greedy decoding algorithm for Transducer-based speech recognition models. We redesign the standard nested-loop design for RNN-T decoding, swapping loops over frames and labels: the outer loop iterates over labels, while the inner loop iterates over frames searching for the next non-blank symbol. Additionally, we represent partial hypotheses in a special structure using CUDA tensors, supporting parallelized hypotheses manipulations. Experiments show that the label-looping algorithm is up to 2.0X faster than conventional batched decoding when using batch size 32. It can be further combined with other compiler or GPU call-related techniques to achieve even more speedup. Our algorithm is general-purpose and can work with both conventional Transducers and Token-and-Duration Transducers. We open-source our implementation to benefit the research community., Comment: Accepted at IEEE SLT 2024

Historically, most speech models in machine-learning have used the mel-spectrogram as a speech representation. Recently, discrete audio tokens produced by neural audio codecs have become a popular alternate speech representation for speech synthesis tasks such as text-to-speech (TTS). However, the data distribution produced by such codecs is too complex for some TTS models to predict, hence requiring large autoregressive models to get reasonable quality. Typical audio codecs compress and reconstruct the time-domain audio signal. We propose a spectral codec which compresses the mel-spectrogram and reconstructs the time-domain audio signal. A study of objective audio quality metrics suggests that our spectral codec has comparable perceptual quality to equivalent audio codecs. Furthermore, non-autoregressive TTS models trained with the proposed spectral codec generate audio with significantly higher quality than when trained with mel-spectrograms or audio codecs.

This paper proposes a flexible multichannel speech enhancement system with the main goal of improving robustness of automatic speech recognition (ASR) in noisy conditions. The proposed system combines a flexible neural mask estimator applicable to different channel counts and configurations and a multichannel filter with automatic reference selection. A transform-attend-concatenate layer is proposed to handle cross-channel information in the mask estimator, which is shown to be effective for arbitrary microphone configurations. The presented evaluation demonstrates the effectiveness of the flexible system for several seen and unseen compact array geometries, matching the performance of fixed configuration-specific systems. Furthermore, a significantly improved ASR performance is observed for configurations with randomly-placed microphones.

Recent observations have demonstrated that giant molecular clouds (GMCs) are short-lived entities, surviving for the order of a dynamical time before turning a few percent of their mass into stars and dispersing, leaving behind an isolated young stellar population. The key question has been whether this GMC dispersal actually marks a point of GMC destruction by stellar feedback from the new-born stars, or if GMCs might be `immortal' and only dynamically decouple from their nascent stars due to stellar drift. We address this question in six nearby galaxies, by quantifying how the gas-star formation relation depends on the spatial scale for scales between the GMC diameter and the GMC separation length, i.e. the scales where an excess of GMCs would be expected to be found in the stellar drift scenario. Our analysis reveals a consistent dearth of GMCs near young stellar populations regardless of the spatial scale, discounting the notion of `immortal' GMCs that decouple from their nascent stars through stellar drift. Instead, our findings demonstrate that stellar feedback destroys most GMCs at the end of their lifecycle. Employing a variety of statistical techniques to test both hypotheses, we find that the probability that stellar feedback concludes the GMC lifecycle is about 2,000 times higher than the probability that stellar drift separates GMCs and young stellar regions. This observation strengthens the emerging picture that galaxies consist of dynamic building blocks undergoing vigorous, feedback-driven lifecycles that collectively regulate star formation and drive the baryon cycle within galaxies., Comment: 10 pages, 4 figures, 3 tables; submitted to The Open Journal of Astrophysics (April 24, 2024); Figure 1 explains the experiment, Figure 2 shows the measurements, Table 3 shows the main result

A compact source, G0.02467-0.0727, was detected in ALMA \threemm observations in continuum and very broad line emission. The continuum emission has a spectral index $\alpha\approx3.3$, suggesting that the emission is from dust. The line emission is detected in several transitions of CS, SO, and SO$_2$ and exhibits a line width FWHM $\approx160$ \kms. The line profile appears Gaussian. The emission is weakly spatially resolved, coming from an area on the sky $\lesssim1"$ in diameter ($\lesssim10^4$ AU at the distance of the Galactic Center; GC). The centroid velocity is $v_{LSR}\approx40$-$50$ \kms, which is consistent with a location in the Galactic Center. With multiple SO lines detected, and assuming local thermodynamic equilibrium (LTE) conditions, $T_\mathrm{LTE} = 13$ K, which is colder than seen in typical GC clouds, though we cannot rule out low-density, subthermally excited, warmer gas. Despite the high velocity dispersion, no emission is observed from SiO, suggesting that there are no strong ($\gtrsim10~\mathrm{km~s}^{-1}$) shocks in the molecular gas. There are no detections at other wavelengths, including X-ray, infrared, and radio. We consider several explanations for the Millimeter Ultra-Broad Line Object (MUBLO), including protostellar outflow, explosive outflow, collapsing cloud, evolved star, stellar merger, high-velocity compact cloud, intermediate mass black hole, and background galaxy. Most of these conceptual models are either inconsistent with the data or do not fully explain it. The MUBLO is, at present, an observationally unique object., Comment: Accepted to ApJL

The ALMA-IMF Large Program provides multi-tracer observations of 15 Galactic massive protoclusters at matched sensitivity and spatial resolution. We focus on the dense gas kinematics of the G353.41 protocluster traced by N$_2$H$^+$ (1$-$0), with an spatial resolution $\sim$0.02 pc. G353.41, at a distance of $\sim$2 kpc, has a mass of $\sim$2500 M$_{\odot}$ within $1.3\times1.3$ pc$^2$. We extract the N$_2$H$^+$ isolated line component and we decompose it by fitting up to 3 Gaussian velocity components. This allows us to identify velocity structures that are impossible to identify in the traditional position-velocity diagram. We identify multiple velocity gradients on large ($\sim$1 pc) and small scales ($\sim$0.2 pc). We find good agreement between the N$_2$H$^+$ velocities and the previously reported DCN core velocities, suggesting that cores are kinematically coupled to the dense gas in which they form. We measure 9 converging ``V-shaped'' velocity gradients ($\sim20$ km/s/pc), located in filaments, which are sometimes associated with cores near their point of convergence. The average inflow timescale is $\sim67$ kyr, or about twice the free-fall time of cores in the same area ($\sim33$ kyr) but substantially shorter than protostar lifetime estimates ($\sim$0.5 Myr). We derive mass accretion rates in the range of $(0.35-8.77)\,\times\,10^{-4}$ M$_{\odot}$/yr. This feeding might lead to further filament collapse and formation of new cores. We suggest that the protocluster is collapsing on large scales, but the velocity signature of collapse is slow compared to pure free-fall. These data are consistent with a comparatively slow global protocluster contraction under gravity, and faster core formation within, suggesting the formation of multiple generations of stars over the protocluster lifetime., Comment: Accepted in A&A, 22 pages, 21 figures, 1 interactive figure, 4 tables

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs., Comment: COLM 2024; Code is available at https://github.com/hsiehjackson/RULER

This paper proposes Transducers with Pronunciation-aware Embeddings (PET). Unlike conventional Transducers where the decoder embeddings for different tokens are trained independently, the PET model's decoder embedding incorporates shared components for text tokens with the same or similar pronunciations. With experiments conducted in multiple datasets in Mandarin Chinese and Korean, we show that PET models consistently improve speech recognition accuracy compared to conventional Transducers. Our investigation also uncovers a phenomenon that we call error chain reactions. Instead of recognition errors being evenly spread throughout an utterance, they tend to group together, with subsequent errors often following earlier ones. Our analysis shows that PET models effectively mitigate this issue by substantially reducing the likelihood of the model generating additional errors following a prior one. Our implementation will be open-sourced with the NeMo toolkit., Comment: accepted at the ICASSP 2024 conference

We aim to identify and characterize cores in the high-mass proto-cluster W49, determine their evolutionary stages and measure the associated lifetimes. We built a catalog of 129 cores extracted from an ALMA 1.3 mm continuum image at 0.26" (2900 au) angular resolution. The association between cores and Hyper/Ultra Compact HII (H/UC HII) regions was established from the analysis of VLA 3.3 cm continuum and H30$\alpha$ line observations. We also looked for emission of hot molecular cores (HMCs) using the methyl formate doublet at 218.29 GHz. We identified 40 cores associated with an H/UC HII region and 19 HMCs over the ALMA mosaic. The 52 cores with an H/UC HII region and/or a HMC are assumed to be high-mass protostellar cores, while the rest of the core population likely consists in prestellar cores and low-mass protostellar cores. We found a good agreement between the two tracers of ionized gas, with 23 common detections and only four cores detected at 3.3 cm and not in H30$\alpha$. The spectral indexes from 3.3 cm to 1.3 mm range from 1, for the youngest cores with partially optically thick free-free emission, to about -0.1, that is optically thin free-free emission obtained for cores likely more evolved. Using as a reference the H/UC HII regions, we found the statistical lifetimes of the HMC and massive protostellar phases in W49N to be about $6\times10^4$ yr and $1.4\times10^5$ yr, respectively. We also showed that HMC can co-exist with H/UC HII regions during a short fraction of the core lifetime, about $2\times10^4$ yr. This indicates a rapid dispersal of the inner molecule envelope once the HC HII is formed., Comment: 15 pages, 7 figures. Accepted for publication in A&A