Search

Analyzing real-world multimodal signals is an essential and challenging task for intelligent voice assistants (IVAs). Mainstream approaches have achieved remarkable performance on various downstream tasks of IVAs with pre-trained audio models and text models. However, these models are pre-trained independently and usually on tasks different from target domains, resulting in sub-optimal modality representations for downstream tasks. Moreover, in many domains, collecting enough language-audio pairs is extremely hard, and transcribing raw audio also requires high professional skills, making it difficult or even infeasible to joint pre-training. To address these painpoints, we propose DSCLAP, a simple and effective framework that enables language-audio pre-training with only raw audio signal input. Specifically, DSCLAP converts raw audio signals into text via an ASR system and combines a contrastive learning objective and a language-audio matching objective to align the audio and ASR transcriptions. We pre-train DSCLAP on 12,107 hours of in-vehicle domain audio. Empirical results on two downstream tasks show that while conceptually simple, DSCLAP significantly outperforms the baseline models in all metrics, showing great promise for domain-specific IVAs applications.

Contrastive learning has become one of the most impressive approaches for multi-modal representation learning. However, previous multi-modal works mainly focused on cross-modal understanding, ignoring in-modal contrastive learning, which limits the representation of each modality. In this paper, we propose a novel contrastive learning strategy, called $Turbo$, to promote multi-modal understanding by joint in-modal and cross-modal contrastive learning. Specifically, multi-modal data pairs are sent through the forward pass twice with different hidden dropout masks to get two different representations for each modality. With these representations, we obtain multiple in-modal and cross-modal contrastive objectives for training. Finally, we combine the self-supervised Turbo with the supervised multi-modal classification and demonstrate its effectiveness on two audio-text classification tasks, where the state-of-the-art performance is achieved on a speech emotion recognition benchmark dataset.

With the goal of more natural and human-like interaction with virtual voice assistants, recent research in the field has focused on full duplex interaction mode without relying on repeated wake-up words. This requires that in scenes with complex sound sources, the voice assistant must classify utterances as device-oriented or non-device-oriented. The dual-encoder structure, which is jointly modeled by text and speech, has become the paradigm of device-directed speech detection. However, in practice, these models often produce incorrect predictions for unaligned input pairs due to the unavoidable errors of automatic speech recognition (ASR).To address this challenge, we propose M$^{3}$V, a multi-modal multi-view approach for device-directed speech detection, which frames we frame the problem as a multi-view learning task that introduces unimodal views and a text-audio alignment view in the network besides the multi-modal. Experimental results show that M$^{3}$V significantly outperforms models trained using only single or multi-modality and surpasses human judgment performance on ASR error data for the first time.

Electron beam shaping allows the control of the temporal properties of x-ray free-electron laser pulses from femtosecond to attosecond timescales. Here we demonstrate the use of a laser heater to shape electron bunches and enable the generation of attosecond x-ray pulses. We demonstrate that this method can be applied in a selective way, shaping a targeted subset of bunches while leaving the remaining bunches unchanged. This experiment enables the delivery of shaped x-ray pulses to multiple undulator beamlines, with pulse properties tailored to specialized scientific applications.

Attosecond photoemission or photoionization delays are a unique probe of the structure and the electronic dynamics of matter. However, spectral congestion and spatial delocalization of valence electron wave functions set fundamental limits to the complexity of systems that can be studied and the information that can be retrieved, respectively. Using attosecond X-ray pulses from LCLS, we demonstrate the key advantages of measuring core-level delays: the photoelectron spectra remain atom-like, the measurements become element specific and the observed scattering dynamics originate from a point-like source. We exploit these unique features to reveal the effects of electronegativity and symmetry on attosecond scattering dynamics by measuring the photoionization delays between N-1s and C-1s core shells of a series of aromatic azabenzene molecules. Remarkably, the delays systematically increase with the number of nitrogen atoms in the molecule and reveal multiple resonances. We identify two previously unknown mechanisms regulating the associated attosecond dynamics, namely the enhanced confinement of the trapped wavefunction with increasing electronegativity of the atoms and the decrease of the coupling strength among the photoemitted partial waves with increasing symmetry. This study demonstrates the unique opportunities opened by measurements of core-level photoionization delays for unravelling attosecond electron dynamics in complex matter.

The photoelectric effect is not truly instantaneous, but exhibits attosecond delays that can reveal complex molecular dynamics. Sub-femtosecond duration light pulses provide the requisite tools to resolve the dynamics of photoionization. Accordingly, the past decade has produced a large volume of work on photoionization delays following single photon absorption of an extreme ultraviolet (XUV) photon. However, the measurement of time-resolved core-level photoionization remained out of reach. The required x-ray photon energies needed for core-level photoionization were not available with attosecond tabletop sources. We have now measured the x-ray photoemission delay of core-level electrons, and here report unexpectedly large delays, ranging up to 700 attoseconds in NO near the oxygen K-shell threshold. These measurements exploit attosecond soft x-ray pulses from a free-electron laser (XFEL) to scan across the entire region near the K-shell threshold. Furthermore, we find the delay spectrum is richly modulated, suggesting several contributions including transient trapping of the photoelectron due to shape resonances, collisions with the Auger-Meitner electron that is emitted in the rapid non-radiative relaxation of the molecule, and multi-electron scattering effects. The results demonstrate how x-ray attosecond experiments, supported by comprehensive theoretical modelling, can unravel the complex correlated dynamics of core-level photoionization.

Vesicle fusion is an important process underlying cell division, transport, and membrane trafficking. In phospholipid systems, a range of fusogens including divalent cations and depletants have been shown to induce adhesion, hemifusion, and then full content fusion between vesicles. This works shows that these fusogens do not perform the same function for fatty acid vesicles, which are used as model protocells (primitive cells). Even when fatty acid vesicles appear adhered or hemifused to each other, the intervening barriers between vesicles do not rupture. This difference is likely because fatty acids have a single aliphatic tail, and are more dynamic than their phospholipid counterparts. To address this, we postulate that fusion could instead occur under conditions, such as lipid exchange, that disrupt lipid packing. Using both experiments and molecular dynamics simulations, we verify that fusion in fatty acid systems can indeed be induced by lipid exchange. These results begin to probe how membrane biophysics could constrain the evolutionary dynamics of protocells., Comment: 15 pages, 7 figures

The hydrodynamic effects of macromolecular crowding inside cells is often studied in vitro by using polymers as crowding reagents. Confinement of polymers inside cell-sized droplets has been shown to affect the diffusion of small molecules. Here we develop a method, based on digital holographic microscopy, to measure the diffusion of polystyrene microspheres that are confined within lipid vesicles containing a high concentration of solute. We apply the method to three solutes of varying complexity: sucrose, dextran, and PEG, prepared at ~7 % (w/w). We find that diffusion inside and outside the vesicles is the same when the solute is sucrose or dextran that is prepared below the critical overlap concentration. For polyethylene glycol, which is present at a concentration higher than the critical overlap concentration, the diffusion of microspheres inside vesicles is slower, hinting at the potential effects of confinement on crowding agents., Comment: 4 figures, 6 pages

Sub-diffraction resolution, gentle sample illumination, and the possibility to image in multiple colors make Structured Illumination Microscopy (SIM) an imaging technique which is particularly well suited for live cell observations. Here, we present Machine learning Assisted Interferometric-SIM (MAI-SIM), an easy-to-implement method for high speed SIM imaging in multiple colors. The instrument is based on an interferometer design in which illumination patterns are generated, rotated, and stepped in phase through movement of a single galvanometric mirror element. The design is robust, flexible, and the pattern generation process works for all wavelengths. We complement the unique properties of interferometric SIM with a machine learning toolbox that is simple and efficient to use and is superior to existing methods for the reconstruction of super-resolved images recorded by the instrument. The framework permits real-time SIM reconstructions to be performed in multiple colors, providing the user with instant visualization of the super-resolved images. We demonstrate the capability of MAI-SIM on live biological samples and capture super-resolution images in multiple colors simultaneously over large fields of view. Finally, we embrace a fully open design philosophy to bring the advantages of MAI-SIM to as many users as possible and provide full details on system design and software., Comment: 31 pages, 7 figures, 9 supporting figures

We report efforts to quantify the loading of cell-sized lipid vesicles using in-line digital holographic microscopy. This method does not require fluorescent reporters, fluorescent tracers, or radioactive tracers. A single-color LED light source takes the place of conventional illumination to generate holograms rather than bright field images. By modelling the vesicle's scattering in a microscope with a Lorenz-Mie light scattering model, and comparing the results to data holograms, we are able to measure the vesicle's refractive index and thus loading. Performing the same comparison for bulk light scattering measurements enables retrieval of vesicle loading for nanoscale vesicles., Comment: 7 figures

Imaging through scattering media is a challenging problem owing to speckle decorrelations from perturbations in the media itself. For in-line imaging modalities, which are appealing because they are compact, require no moving parts, and are robust, negating the effects of such scattering becomes particularly challenging. Here we explore the effect of stationary scattering media on light scattering in in-line geometries, including digital holographic microscopy. We consider various object-scatterer scenarios where the object is distorted or obscured by additional stationary scatterers, and use an advanced deep learning (DL) generative methodology, generative adversarial networks (GANs), to mitigate the effects of the additional scatterers. Using light scattering simulations and experiments on objects of interest with and without additional scatterers, we find that conditional GANs can be quickly trained with minuscule datasets and can also efficiently learn the one-to-one statistical mapping between the cross-domain input-output image pairs. Training such a network yields a standalone model, that can be used later to inverse or negate the effect of scattering, yielding clear object reconstructions for object retrieval and downstream processing. Moreover, it is well-known that the coherent point spread function (c-PSF) of a stationary scattering optical system is a speckle pattern which is spatially shift variant. We show that with rapid training using only 20 image pairs, it is possible to negate this undesired scattering to accurately localize diffraction-limited impulses with high spatial accuracy, therefore transforming the earlier shift variant system to a linear shift invariant (LSI) system., Comment: 4 figures

We demonstrate an accurate phase retrieval of XUV atomic ionization by streaking the photoelectron in a circularly polarized IR laser field. The streaking phase can then be converted to the atomic time delay containing the Wigner and continuum-continuum components. Our demonstration is based on a numerical solution of the time-dependent Schr\"odinger equation. We test this technique using the hydrogen atom ionized by an isolated attosecond XUV pulse across a wide range of photon energies. In parallel, we run a series of RABBITT simulations and demonstrate equivalence of the phase and timing information provided by the two methods. This validates the proposed technique and makes it a useful tool that can be applied to a broad range of atomic and molecular targets exposed to XUV radiation from novel free-electron laser sources., Comment: 7 pages, 5 figures

The newly constructed Time-resolved atomic, Molecular and Optical science instrument (TMO), is configured to take full advantage of both linear accelerators at SLAC National Accelerator Laboratory, the copper accelerator operating at a repetition rate of 120 Hz providing high per pulse energy, as well as the superconducting accelerator operating at a repetition rate of about 1 MHz providing high average intensity. Both accelerators build a soft X-ray free electron laser with the new variable gab undulator section. With this flexible light sources, TMO supports many experimental techniques not previously available at LCLS and will have two X-ray beam focus spots in line. Thereby, TMO supports Atomic, Molecular and Optical (AMO), strong-field and nonlinear science and will host a designated new dynamic reaction microscope with a sub-micron X-ray focus spot. The flexible instrument design is optimized for studying ultrafast electronic and molecular phenomena and can take full advantage of the sub-femtosecond soft X-ray pulse generation program.

In quantum systems, coherent superpositions of electronic states evolve on ultrafast timescales (few femtosecond to attosecond, 1 as = 0.001 fs = 10^{-18} s), leading to a time dependent charge density. Here we exploit the first attosecond soft x-ray pulses produced by an x-ray free-electron laser to induce a coherent core-hole excitation in nitric oxide. Using an additional circularly polarized infrared laser pulse we create a clock to time-resolve the electron dynamics, and demonstrate control of the coherent electron motion by tuning the photon energy of the x-ray pulse. Core-excited states offer a fundamental test bed for studying coherent electron dynamics in highly excited and strongly correlated matter.

The desire to create cell-like models for fundamental science and applications has spurred extensive effort towards creating giant unilamellar vesicles (GUVs). However, a route to selectively self-assemble GUVs in bulk has remained elusive. In bulk solution, membrane-forming molecules such as phospholipids, single-tailed surfactants, and block copolymers typically self-assemble into multilamellar, onion-like structures. So although self-assembly processes can form nanoscale unilamellar vesicles, scaffolding by droplets or surfaces is required to create GUVs. Here we show that surprisingly, it is possible to bulk self-assemble cell-sized GUVs with almost complete selectivity over other vesicle topologies. The seemingly paradoxical pair of features that enables this appears to be having very dynamic molecules at the nanoscale, that create unusually rigid membranes. The resultant self-assembly pathway enables encapsulation of molecules and colloids, and can also generate model primitive cells that can grow and divide., Comment: 9 pages, 4 figures

The recent demonstration of isolated attosecond pulses from an X-ray free-electron laser (XFEL) opens the possibility for probing ultrafast electron dynamics at X-ray wavelengths. An established experimental method for probing ultrafast dynamics is X-ray transient absorption spectroscopy, where the X-ray absorption spectrum is measured by scanning the central photon energy and recording the resultant photoproducts. The spectral bandwidth inherent to attosecond pulses is wide compared to the resonant features typically probed, which generally precludes the application of this technique in the attosecond regime. In this paper we propose and demonstrate a new technique to conduct transient absorption spectroscopy with broad bandwidth attosecond pulses with the aid of ghost imaging, recovering sub-bandwidth resolution in photoproduct-based absorption measurements., Comment: 13 pages, 3 figures

A technique for measuring photoionization time delays with attosecond precision is combined with calculations of photoionization matrix elements to demonstrate how multi-electron dynamics affect photoionization time delays in carbon dioxide. Electron correlation is observed to affect the time delays through two mechanisms: autoionization of molecular Rydberg states and accelerated escape from a continuum shape resonance.

The quantum mechanical motion of electrons in molecules and solids occurs on the sub-femtosecond timescale. Consequently, the study of ultrafast electronic phenomena requires the generation of laser pulses shorter than 1 fs and of sufficient intensity to interact with their target with high probability. Probing these dynamics with atomic-site specificity requires the extension of sub-femtosecond pulses to the soft X-ray spectral region. Here we report the generation of isolated GW-scale soft X-ray attosecond pulses with an X-ray free-electron laser. Our source has a pulse energy that is six orders of magnitude larger than any other source of isolated attosecond pulses in the soft X-ray spectral region, with a peak power in the tens of gigawatts. This unique combination of high intensity, high photon energy and short pulse duration enables the investigation of electron dynamics with X-ray non-linear spectroscopy and single-particle imaging.

A holographic microscope captures interference patterns, or holograms, that encode three-dimensional (3D) information about the object being viewed. Computation is essential to extracting that 3D information. By wrapping low-level scattering codes and taking advantage of Python's data analysis ecosystem, HoloPy makes it easy for experimentalists to use modern, sophisticated inference methods to analyze holograms. The resulting data can be used to understand how small particles or microorganisms move and interact. The project illustrates how computational tools can enable experimental methods and new experiments.