Search

Labelled data are limited and self-supervised learning is one of the most important approaches for reducing labelling requirements. While it has been extensively explored in the image domain, it has so far not received the same amount of attention in the acoustic domain. Yet, reducing labelling is a key requirement for many acoustic applications. Specifically in bioacoustic, there are rarely sufficient labels for fully supervised learning available. This has led to the widespread use of acoustic recognisers that have been pre-trained on unrelated data for bioacoustic tasks. We posit that training on the actual task data and combining self-supervised pre-training with few-shot classification is a superior approach that has the ability to deliver high accuracy even when only a few labels are available. To this end, we introduce and evaluate a new architecture that combines CNN-based preprocessing with feature extraction based on state space models (SSMs). This combination is motivated by the fact that CNN-based networks alone struggle to capture temporal information effectively, which is crucial for classifying acoustic signals. SSMs, specifically S4 and Mamba, on the other hand, have been shown to have an excellent ability to capture long-range dependencies in sequence data. We pre-train this architecture using contrastive learning on the actual task data and subsequent fine-tuning with an extremely small amount of labelled data. We evaluate the performance of this proposed architecture for ($n$-shot, $n$-class) classification on standard benchmarks as well as real-world data. Our evaluation shows that it outperforms state-of-the-art architectures on the few-shot classification problem.

Deep learning has the potential to enhance speech signals and increase their intelligibility for users of hearing aids. Deep models suited for real-world application should feature a low computational complexity and low processing delay of only a few milliseconds. In this paper, we explore deep speech enhancement that matches these requirements and contrast monaural and binaural processing algorithms in two complex acoustic scenes. Both algorithms are evaluated with objective metrics and in experiments with hearing-impaired listeners performing a speech-in-noise test. Results are compared to two traditional enhancement strategies, i.e., adaptive differential microphone processing and binaural beamforming. While in diffuse noise, all algorithms perform similarly, the binaural deep learning approach performs best in the presence of spatial interferers. Through a post-analysis, this can be attributed to improvements at low SNRs and to precise spatial filtering., Comment: This work is published in IEEE/ACM TASLP. This version corresponds to the accepted version

In this paper, we introduce a causal low-latency low-complexity approach for binaural multichannel blind speaker separation in noisy reverberant conditions. The model, referred to as Group Communication Binaural Filter and Sum Network (GCBFSnet) predicts complex filters for filter-and-sum beamforming in the time-frequency domain. We apply Group Communication (GC), i.e., latent model variables are split into groups and processed with a shared sequence model with the aim of reducing the complexity of a simple model only containing one convolutional and one recurrent module. With GC we are able to reduce the size of the model by up to 83 % and the complexity up to 73 % compared to the model without GC, while mostly retaining performance. Even for the smallest model configuration, GCBFSnet matches the performance of a low-complexity TasNet baseline in most metrics despite the larger size and higher number of required operations of the baseline., Comment: Accepted for publication at IEEE ICASSP 2024 OJSP track

Ab initio molecular dynamics (AIMD) based on density functional theory (DFT) has become a workhorse for studying the structure, dynamics, and reactions in condensed matter systems. Currently, AIMD simulations are primarily carried out at the level of generalized gradient approximation (GGA), which is at the 2nd rung of DFT-functionals in terms of accuracy. Hybrid DFT functionals which form the 4th rung in the accuracy ladder, are not commonly used in AIMD simulations as the computational cost involved is $100$ times or higher. To facilitate AIMD simulations with hybrid functionals, we propose here an approach that could speed up the calculations by ~30 times or more for systems with a few hundred of atoms. We demonstrate that, by achieving this significant speed up and making the compute time of hybrid functional based AIMD simulations at par with that of GGA functionals, we are able to study several complex condensed matter systems and model chemical reactions in solution with hybrid functionals that were earlier unthinkable to be performed., Comment: 45 pages, 9 figures, 5 tables

The scarcity of labelled data makes training Deep Neural Network (DNN) models in bioacoustic applications challenging. In typical bioacoustics applications, manually labelling the required amount of data can be prohibitively expensive. To effectively identify both new and current classes, DNN models must continue to learn new features from a modest amount of fresh data. Active Learning (AL) is an approach that can help with this learning while requiring little labelling effort. Nevertheless, the use of fixed feature extraction approaches limits feature quality, resulting in underutilization of the benefits of AL. We describe an AL framework that addresses this issue by incorporating feature extraction into the AL loop and refining the feature extractor after each round of manual annotation. In addition, we use raw audio processing rather than spectrograms, which is a novel approach. Experiments reveal that the proposed AL framework requires 14.3%, 66.7%, and 47.4% less labelling effort on benchmark audio datasets ESC-50, UrbanSound8k, and InsectWingBeat, respectively, for a large DNN model and similar savings on a microcontroller-based counterpart. Furthermore, we showcase the practical relevance of our study by incorporating data from conservation biology projects. All codes are publicly available on GitHub.

The state of protonation/deprotonation of surfaces has far-ranging implications in all areas of chemistry: from acid-base catalysis$^1$ and the electro- and photocatalytic splitting of water$^2$, to the behavior of minerals$^3$ and biochemistry$^4$. The acidity of a molecule or a surface site is described by its proton affinity (PA) and pK$_\mathrm{a}$ value (the negative logarithm of the equilibrium constant of the proton transfer reaction in solution). For solids, in contrast to molecules, the acidity of individual sites is difficult to assess. For mineral surfaces such as oxides they are estimated by semi-empirical concepts such as bond-order valence sums$^5$, and also increasingly modeled with first-principles molecular dynamics simulations$^{6,7}$. Currently such predictions cannot be tested - the experimental measures used for comparison are typically average quantities integrated over the whole surface or, in some cases, individual crystal facets$^8$, such as the point of zero charge (pzc)$^9$. Here we assess individual hydroxyls on In$_2$O$_3$(111), a model oxide with four different types of surface oxygen atoms, and probe the strength of their hydrogen bond with the tip of a non-contact atomic force microscope (AFM). The force curves are in quantitative agreement with density-functional theory (DFT) calculations. By relating the results to known proton affinities and pK$_\mathrm{a}$ values of gas-phase molecules, we provide a direct measure of proton affinity distributions at the atomic scale.

Clean oxide surfaces are generally hydrophilic. Water molecules anchor at undercoordinated surface metal atoms that act as Lewis-acid sites, and they are stabilized by H bonds to undercoordinated surface oxygens. The large unit cell of In2O3(111) provides surface atoms in various configurations, which leads to chemical heterogeneity and a local deviation from this general rule. Experiments (TPD, XPS, ncAFM) agree quantitatively with DFT calculations and show a series of distinct phases. The first three water molecules dissociate at one specific area of the unit cell and desorb above room temperature. The next three adsorb as molecules in the adjacent region. Three more water molecules rearrange this structure and an additional nine pile up above the OH groups. Despite offering undercoordinated In and O sites, the rest of the unit cell is unfavorable for adsorption and remains water-free. The first water layer thus shows ordering into nanoscopic 3D water clusters separated by hydrophobic pockets.

Speech enhancement in hearing aids is a challenging task since the hardware limits the number of possible operations and the latency needs to be in the range of only a few milliseconds. We propose a deep-learning model compatible with these limitations, which we refer to as Group-Communication Filter-and-Sum Network (GCFSnet). GCFSnet is a causal multiple-input single output enhancement model using filter-and-sum processing in the time-frequency domain and a multi-frame deep post filter. All filters are complex-valued and are estimated by a deep-learning model using weight-sharing through Group Communication and quantization-aware training for reducing model size and computational footprint. For a further increase in performance, a low bit rate binaural link for delayed binaural features is proposed to use binaural information while retaining a latency of 2ms. The performance of an oracle binaural LCMV beamformer in non-low-latency configuration can be matched even by a unilateral configuration of the GCFSnet in terms of objective metrics., Comment: Accepted at WASPAA 2023

In this paper, we propose a multilingual query-by-example keyword spotting (KWS) system based on a residual neural network. The model is trained as a classifier on a multilingual keyword dataset extracted from Common Voice sentences and fine-tuned using circle loss. We demonstrate the generalization ability of the model to new languages and report a mean reduction in EER of 59.2 % for previously seen and 47.9 % for unseen languages compared to a competitive baseline. We show that the word embeddings learned by the KWS model can be accurately predicted from the phoneme sequences using a simple LSTM model. Our system achieves a promising accuracy for streaming keyword spotting and keyword search on Common Voice audio using just 5 examples per keyword. Experiments on the Hey-Snips dataset show a good performance with a false negative rate of 5.4 % at only 0.1 false alarms per hour., Comment: Accepted to ICASSP 2023

Tracking individuals is a vital part of many experiments conducted to understand collective behaviour. Ants are the paradigmatic model system for such experiments but their lack of individually distinguishing visual features and their high colony densities make it extremely difficult to perform reliable tracking automatically. Additionally, the wide diversity of their species' appearances makes a generalized approach even harder. In this paper, we propose a data-driven multi-object tracker that, for the first time, employs domain adaptation to achieve the required generalisation. This approach is built upon a joint-detection-and-tracking framework that is extended by a set of domain discriminator modules integrating an adversarial training strategy in addition to the tracking loss. In addition to this novel domain-adaptive tracking framework, we present a new dataset and a benchmark for the ant tracking problem. The dataset contains 57 video sequences with full trajectory annotation, including 30k frames captured from two different ant species moving on different background patterns. It comprises 33 and 24 sequences for source and target domains, respectively. We compare our proposed framework against other domain-adaptive and non-domain-adaptive multi-object tracking baselines using this dataset and show that incorporating domain adaptation at multiple levels of the tracking pipeline yields significant improvements. The code and the dataset are available at https://github.com/chamathabeysinghe/da-tracker.

Kohn-Sham density functional theory and plane wave basis set based ab initio molecular dynamics (AIMD) simulation is a powerful tool for studying complex reactions in solutions, such as electron transfer (ET) reactions involving Fe$^{2+}$/Fe$^{3+}$ ions in water. In most cases, such simulations are performed using density functionals at the level of Generalized Gradient Approximation (GGA). The challenge in modelling ET reactions is the poor quality of GGA functionals in predicting properties of such open-shell systems due to the inevitable self-interaction error (SIE). While hybrid functionals can minimize SIE, AIMD at that level of theory is typically 150 times slower than GGA for systems containing ~100 atoms. Among several approaches reported to speed-up AIMD simulations with hybrid functionals, the noise-stabilized MD (NSMD) procedure, together with the use of localized orbitals to compute the required exchange integrals, is an attractive option. In this work, we demonstrate the application of the NSMD approach for studying the Fe$^{2+}$/Fe$^{3+}$ redox reaction in water. It is shown here that long AIMD trajectories at the level of hybrid density functionals can be obtained using this approach. Redox properties of the aqueous Fe$^{2+}$/Fe$^{3+}$ system computed from these simulations are compared with the available experimental data for validation., Comment: 9 pages, 7 figures

Molecules consisting of a donor and an acceptor moiety can exhibit large intrinsic dipole moments. Upon deposition on a metal surface, the dipole may be effectively screened and the charge distribution altered due to hybridization with substrate electronic states. Here, we deposit Ethyl-Diaminodicyanoquinone molecules, which exhibit a large dipole moment in gas phase, on a Au(111) surface. Employing a combination of scanning tunneling microscopy and non-contact atomic force microscopy, we find that a significant dipole moment persists in the flat-lying molecules. Density-functional theory calculations reveal that the dipole moment is even increased on the metal substrate as compared to the gas phase. We also show that the local contact potential across the molecular islands is decreased by several tens of meV with respect to the bare metal. We explain this by the induced charge-density redistribution due to the adsorbed molecules, which confine the substrate's wavefunction at the interface. Our local measurements provide direct evidence of this so-called push-back or cushion effect at the scale of individual molecules., Comment: This document is the unedited Author's version of a Submitted Work that was subsequently accepted for publication in Journal of Physical Chemistry C

This paper introduces a real-time time-domain packet loss concealment (PLC) neural-network (tPLCnet). It efficiently predicts lost frames from a short context buffer in a sequence-to-one (seq2one) fashion. Because of its seq2one structure, a continuous inference of the model is not required since it can be triggered when packet loss is actually detected. It is trained on 64h of open-source speech data and packet-loss traces of real calls provided by the Audio PLC Challenge. The model with the lowest complexity described in this paper reaches a robust PLC performance and consistent improvements over the zero-filling baseline for all metrics. A configuration with higher complexity is submitted to the PLC Challenge and shows a performance increase of 1.07 compared to the zero-filling baseline in terms of PLC-MOS on the blind test set and reaches a competitive 3rd place in the challenge ranking., Comment: Submitted to Interspeech 2022

This paper presents a speech intelligibility model based on automatic speech recognition (ASR), combining phoneme probabilities from deep neural networks (DNN) and a performance measure that estimates the word error rate from these probabilities. This model does not require the clean speech reference nor the word labels during testing as the ASR decoding step, which finds the most likely sequence of words given phoneme posterior probabilities, is omitted. The model is evaluated via the root-mean-squared error between the predicted and observed speech reception thresholds from eight normal-hearing listeners. The recognition task consists of identifying noisy words from a German matrix sentence test. The speech material was mixed with eight noise maskers covering different modulation types, from speech-shaped stationary noise to a single-talker masker. The prediction performance is compared to five established models and an ASR-model using word labels. Two combinations of features and networks were tested. Both include temporal information either at the feature level (amplitude modulation filterbanks and a feed-forward network) or captured by the architecture (mel-spectrograms and a time-delay deep neural network, TDNN). The TDNN model is on par with the DNN while reducing the number of parameters by a factor of 37; this optimization allows parallel streams on dedicated hearing aid hardware as a forward-pass can be computed within the 10ms of each frame. The proposed model performs almost as well as the label-based model and produces more accurate predictions than the baseline models.

Industrial upscaling frequently results in a different coating microstructure than the laboratory prototypes presented in the literature. Here, we investigate the wear behavior of physical vapor deposited (PVD) MoS2 coatings: A dense, nanocrystalline MoS2 coating, and a porous, prismatic-textured MoS2 coating. Transmission electron microscopy (TEM) investigations before and after wear testing evidence a crystallographic reorientation towards a basal texture in both samples. A basal texture is usually desirable due to its low-friction properties. This favorable reorientation is associated to a tribological compaction of the porous specimens. Following running-in, sliding under high contact pressure ultimately leads to a wear rate as small as for an ideal grown bulk MoS2 single crystal grown by chemical vapor deposition (CVD). This suggests that the imperfections of industrial grade MoS2 coatings can be remediated by a suitable pretreatment.

Listening to the audio of TV broadcast signals can be challenging for hearing-impaired as well as normal-hearing listeners, especially when background sounds are prominent or too loud compared to the speech signal. This can result in a reduced satisfaction and increased listening effort of the listeners. Since the broadcast sound is usually premixed, we perform a subjective evaluation for quantifying the potential of speech enhancement systems based on audio source separation and recurrent neural networks (RNN). Recently, RNNs have shown promising results in the context of sound source separation and real-time signal processing. In this paper, we separate the speech from the background signals and remix the separated sounds at a higher signal-to-noise ratio. This differs from classic speech enhancement, where usually only the extracted speech signal is exploited. The subjective evaluation with 20 normal-hearing subjects on real TV-broadcast material shows that our proposed enhancement system is able to reduce the listening effort by around 2 points on a 13-point listening effort rating scale and increases the perceived sound quality compared to the original mixture., Comment: Accepted by IEEE/ACM Transactions on Audio, Speech and Language Processing. This version is the authors' version and may vary from the final publication in details

Density functionals at the level of the Generalized Gradient Approximation (GGA) and a plane-wave basis set are widely used today to perform ab initio molecular dynamics (AIMD) simulations. Going up in the ladder of accuracy of density functionals from GGA (2nd rung) to hybrid density functionals (4th rung) is much desired pertaining to the accuracy of the latter in describing structure, dynamics, and energetics of molecular and condensed matter systems. On the other hand, hybrid density functional based AIMD simulations are about two orders of magnitude slower than GGA based AIMD for systems containing ~100 atoms using ~100 compute cores. Two methods, namely MTACE and s-MTACE, based on a multiple time step integrator and adaptively compressed exchange operator formalism are able to provide a speed-up of about 7-9 in performing hybrid density functional based AIMD. In this work, we report an implementation of these methods using a task-group based parallelization within the CPMD program package, with the intention to take advantage of the large number of compute cores available on modern high-performance computing platforms. We present here the boost in performance achieved through this algorithm. This work also identifies the computational bottleneck in the s-MTACE method, and proposes a way to overcome that., Comment: 11 pages, 7 figures, 5 tables

Deep learning has celebrated resounding successes in many application areas of relevance to the Internet of Things (IoT), such as computer vision and machine listening. These technologies must ultimately be brought directly to the edge to fully harness the power of deep learning for the IoT. The obvious challenge is that deep learning techniques can only be implemented on strictly resource-constrained edge devices if the models are radically downsized. This task relies on different model compression techniques, such as network pruning, quantization, and the recent advancement of XNOR-Net. This study examines the suitability of these techniques for audio classification on microcontrollers. We present an application of XNOR-Net for end-to-end raw audio classification and a comprehensive empirical study comparing this approach with pruning-and-quantization methods. We show that raw audio classification with XNOR yields comparable performance to regular full precision networks for small numbers of classes while reducing memory requirements 32-fold and computation requirements 58-fold. However, as the number of classes increases significantly, performance degrades, and pruning-and-quantization based compression techniques take over as the preferred technique being able to satisfy the same space constraints but requiring approximately 8x more computation. We show that these insights are consistent between raw audio classification and image classification using standard benchmark sets. To the best of our knowledge, this is the first study to apply XNOR to end-to-end audio classification and evaluate it in the context of alternative techniques. All codes are publicly available on GitHub., Comment: 13 pages

Significant efforts are being invested to bring state-of-the-art classification and recognition to edge devices with extreme resource constraints (memory, speed, and lack of GPU support). Here, we demonstrate the first deep network for acoustic recognition that is small, flexible and compression-friendly yet achieves state-of-the-art performance for raw audio classification. Rather than handcrafting a once-off solution, we present a generic pipeline that automatically converts a large deep convolutional network via compression and quantization into a network for resource-impoverished edge devices. After introducing ACDNet, which produces above state-of-the-art accuracy on ESC-10 (96.65%), ESC-50 (87.10%), UrbanSound8K (84.45%) and AudioEvent (92.57%), we describe the compression pipeline and show that it allows us to achieve 97.22% size reduction and 97.28% FLOP reduction while maintaining close to state-of-the-art accuracy 96.25%, 83.65%, 78.27% and 89.69% on these datasets. We describe a successful implementation on a standard off-the-shelf microcontroller and, beyond laboratory benchmarks, report successful tests on real-world datasets.

This paper applies the dual-signal transformation LSTM network (DTLN) to the task of real-time acoustic echo cancellation (AEC). The DTLN combines a short-time Fourier transformation and a learned feature representation in a stacked network approach, which enables robust information processing in the time-frequency and in the time domain, which also includes phase information. The model is only trained on 60~h of real and synthetic echo scenarios. The training setup includes multi-lingual speech, data augmentation, additional noise and reverberation to create a model that should generalize well to a large variety of real-world conditions. The DTLN approach produces state-of-the-art performance on clean and noisy echo conditions reducing acoustic echo and additional noise robustly. The method outperforms the AEC-Challenge baseline by 0.30 in terms of Mean Opinion Score (MOS)., Comment: Submitted in to ICASSP 2021