Search

The human superior temporal gyrus (STG) is critical for extracting meaningful linguistic features from speech input. Local neural populations are tuned to acoustic-phonetic features of all consonants and vowels and to dynamic cues for intonational pitch. These populations are embedded throughout broader functional zones that are sensitive to amplitude-based temporal cues. Beyond speech features, STG representations are strongly modulated by learned knowledge and perceptual goals. Currently, a major challenge is to understand how these features are integrated across space and time in the brain during natural speech comprehension. We present a theory that temporally recurrent connections within STG generate context-dependent phonological representations, spanning longer temporal sequences relevant for coherent percepts of syllables, words, and phrases.

Hearing loss is one of the most common conditions affecting older adults worldwide. Frequent complaints from the users of modern hearing aids include poor speech intelligibility in noisy environments and high cost, among other issues. However, the signal processing and audiological research needed to address these problems has long been hampered by proprietary development systems, underpowered embedded processors, and the difficulty of performing tests in real-world acoustical environments. To facilitate existing research in hearing healthcare and enable new investigations beyond what is currently possible, we have developed a modern, open-source hearing research platform, Open Speech Platform (OSP). This paper presents the system design of the complete OSP wearable platform, from hardware through firmware and software to user applications. The platform provides a complete suite of basic and advanced hearing aid features which can be adapted by researchers. It serves web apps directly from a hotspot on the wearable hardware, enabling users and researchers to control the system in real time. In addition, it can simultaneously acquire high-quality electroencephalography (EEG) or other electrophysiological signals closely synchronized to the audio. All of these features are provided in a wearable form factor with enough battery life for hours of operation in the field.

In this work, we demonstrate the existence of universal adversarial audio perturbations that cause mis-transcription of audio signals by automatic speech recognition (ASR) systems. We propose an algorithm to find a single quasi-imperceptible perturbation, which when added to any arbitrary speech signal, will most likely fool the victim speech recognition model. Our experiments demonstrate the application of our proposed technique by crafting audio-agnostic universal perturbations for the state-of-the-art ASR system - Mozilla DeepSpeech. Additionally, we show that such perturbations generalize to a significant extent across models that are not available during training, by performing a transferability test on a WaveNet based ASR system.

The personality in the present world plays a critical role in social interactions, the use of modern technologies, and individuals' success. Therefore, in the last two decades, the study of Automatic Personality Perception (APP) and Automatic Personality Recognition (APR) has become more prevalent than speech processing. These studies have shown that personality traits affect acoustic features. However, the intrinsic imbalanced distribution of personality classes across the dataset is an issue mentioned in most previous studies and the classification results suffer from it. In this paper, an innovative supervised k-fold Cross-Validation (CV) method was proposed to cope with the problem of affecting the imbalanced distribution of data across different classes. The classification outcomes showed better performance in comparison with three traditional data balancing methods. Moreover, the obtained results of the proposed evaluation method indicated that the proposed method acts as a k-fold CV method if the data distribution is balanced; otherwise, it will improve the classification results.

Abstract Human speech signals may contain specific information regarding a speaker's characteristics, and these signals can be very useful in applications involving interactive voice response (IVR) and automatic speech recognition (ASR). For IVR and ASR applications, speaker classification into different ages and gender groups can be applied in human–machine interaction or computer‐based interaction systems for customised advertisement, translation (text generation), machine dialog systems, or self‐service applications. Hence, an IVR‐based system dictates that ASR should function through users' voices (specific voice‐frequency bands) to identify customers' age and gender and interact with a host system. In the present study, we intended to combine a pitch detection (PD)‐based extractor and a voice classifier for gender identification. The Yet Another Algorithm for Pitch Tracking (YAAPT)‐based PD method was designed to extract the voice fundamental frequency (F0) from non‐stationary speaker's voice signals, allowing us to achieve gender identification, by distinguishing differences in F0 between adult females and males, and classify voices into adult and children groups. Then, in vowel voice signal classification, a one‐dimensional (1D) convolutional neural network (CNN), consisted of a multi‐round 1D kernel convolutional layer, a 1D pooling process, and a vowel classifier that could preliminary divide feature patterns into three level ranges of F0, including adult and children groups. Consequently, a classifier was used in the classification layer to identify the speakers' gender. The proposed PD‐based extractor and voice classifier could reduce complexity and improve classification efficiency. Acoustic datasets were selected from the Hillenbrand database for experimental tests on 12 vowels classifications, and K‐fold cross‐validations were performed. The experimental results demonstrated that our approach is a very promising method to quantify the proposed classifier's performance in terms of recall (%), precision (%), accuracy (%), and F1 score.

Abstract Although current attention‐based speech enhancement methods have been proven to be capable of significantly improving the noise reduction performance, a bottleneck has arisen in juggling both detailed features and high‐level features: The more attention paid to such performance indicators as speech intelligibility index and articulation index will lead to the loss of subtle features such as syllable continuity and timbre distortion. To tackle such a problem, we divide the speech enhancement model into multiple stages, with a special attention mechanism introduced in each stage so that the detailed speech information is retained and the features of high levels are captured. In this research, a Multi‐Stage Attention Network (MSANet) is implemented in cascade by using three different attention modules: Self Attention, Channel Attention and Spatial Attention. The attention module in each stage needs only to focus on the features of its corresponding stage, thus giving full play to their respective advantages without affecting or damaging the features of other stages, helping decouple the feature extraction process and obtaining the features capable of complementing one another. The comparisons and ablation experiments show that the performance of MSANet is superior to those of current mainstream time domain or time‐frequency domain state‐of‐the‐art methods. Compared to the baseline, PESQ score of their model (3.11) has increased by 5%, CSIG (4.44) has increased by 2.5%, CBAK (3.63) has increased by 2.8% and COVL (3.81) has increased by 3.8%, which demonstrates the potential of MSANet as speech feature extraction backbones. An implementation of the PyTorch version is available here: http://www.msp‐lab.cn:1436/msp/MSANet.

Unsupervised learning-based approaches for training speech vector representations (SVR) have recently been widely applied. While pretrained SVR models excel in relatively clean automatic speech recognition (ASR) tasks, such as those recorded in laboratory environments, they are still insufficient for practical applications with various types of noise, intonation, and dialects. To cope with this problem, we present a novel unsupervised SVR learning method for practical end-to-end ASR models. Our approach involves designing a speech feature masking method to stabilize SVR model learning and improve the performance of the ASR model in a downstream task. By introducing a noise masking strategy into diverse combinations of the time and frequency regions of the spectrogram, the SVR model becomes a robust representation extractor for the ASR model in practical scenarios. In pretraining experiments, we train the SVR model using approximately 18,000 h of Korean speech datasets that included diverse speakers and were recorded in environments with various amounts of noise. The weights of the pretrained SVR extractor are then frozen, and the extracted speech representations are used for ASR model training in a downstream task. The experimental results show that the ASR model using our proposed SVR extractor significantly outperforms conventional methods. [ABSTRACT FROM AUTHOR]

The functioning of language as a communication system is described by a variety of linguistic theories which are not always consistent with each other and often do not rely on cognitive and/or neurobiological data. One of the most striking examples is the "null morpheme" proposed by the theory of universal grammar, which is a concept that has only an abstract meaning as it has no phonological implementation (for example, the null ending of nouns in some cases: cf. stol-∅ (tableNOM) vs. stol-a (tableGEN). With the aim of testing the processing of null morphemes by the brain, we conducted an EEG experiment in which subjects were presented with heard phrases with null (e.g., on kupil-∅) (he bought-∅) or marked (ona kupil-a) (she boughtFEM) verb endings, with correct and incorrect gender agreement between the pronoun and the verb (e.g., on kupil-∅ (he bought-∅) vs. *ona kupil-∅ (she bought-∅, where the correct ending is –a)) and the verb presented alone (kupil-∅) (bought-∅) as a control condition. Event-related potentials analysis results demonstrated an increase in cerebral responses where the null ending disagreed. The latency of this response was ~200 msec, corresponding to the ELAN component, which reflects the early automatic processing of (morpho)syntactic information. These results can be explained by the theory of morphosyntactic priming and indicate that a representation of the null morpheme (ending) exists and is activated in the process of speech perception. [ABSTRACT FROM AUTHOR]

Speech is the primary entity for personal communication however ambient quality generally impairs speech signal quality and understanding of communication. Therefore, it is required that the distorted speech signal be improved in its quality and comprehension. In the field of speech processing, great efforts have been made to develop speech enhancement techniques that restore speech signals by reducing the amount of interfering noise. This work focuses on a critical analysis of single channel speech enhancement technique that performs noise reduction through spectral subtraction based on minimal statistics. Minimal statistics implies estimating the power spectrum of a non-standard noise signal by avoiding the problem of detecting speech activity by finding the smallest value for a smooth power spectrum of a noisy speech signal. The performance of the spectral subtraction method is evaluated over a wide range of noise types with varying sound levels using single channel speech data. This estimator is used to find the optimal value for the method parameter and improve this algorithm to make it more suitable for voice communication purposes. The system can be implemented in MATLAB and also validated against a variety of performance measures and various improvements in signal-to-noise ratio (SNRI) and spectral distortion (SD). This approach provides effective speech enhancement in SNRI and SD performance metrics. A comparatively new method has been proposed in this paper named Spectral Statistics Based on Minimum Statistics (SSBMS) which customarily follows the transient noise and provides a better response in the process of speech enhancement. [ABSTRACT FROM AUTHOR]

Speech enhancement is the process of enhancing the clarity and intelligibility of speech signals that have been degraded due to background noise. With the assistance of deep learning, a novel speech signal enhancement model is introduced in this research. The proposed model is divided into two phases: (i) Training (ii) Testing. In the training phase, the noise spectrum and signal spectrum are estimated via a Non-negative Matrix Factorization (NMF) from the noisy input signal. Then, Empirical Mean Decomposition (EMD) features are extracted from the Wiener filter. The de-noised signal is acquired from EMD, the bark frequency is evaluated and the Fractional Delta AMS features are extracted. The key contribution of this study is the use of the Long Short Term Memory (LSTM) model to properly estimate the tuning factor η of the Wiener filter for all input signals. The LSTM was trained by the extracted features (EMD) via a modified wiener filter for decomposing the spectral signal and the output of EMD is the denoised enhanced speech signal. A comparative evaluation is carried out between the proposed and existing models in terms of error measures. [ABSTRACT FROM AUTHOR]

In this work, we present an approach to understand the computational methods and decision-making involved in the identification of emotions in spontaneous speech. The selected task consists of Spanish TV debates, which entail a high level of complexity as well as additional subjectivity in the human perception-based annotation procedure. A simple convolutional neural model is proposed, and its behaviour is analysed to explain its decision-making. The proposed model slightly outperforms commonly used CNN architectures such as VGG16, while being much lighter. Internal layer-by-layer transformations of the input spectrogram are visualised and analysed. Finally, a class model visualisation is proposed as a simple interpretation approach whose usefulness is assessed in the work. [ABSTRACT FROM AUTHOR]

Introduction: The paper examines the discrimination of lexical stress contrasts in a foreign language from a neural perspective. The aim of the study was to identify the areas associated with word stress processing (in comparison with vowel processing), when listeners of a fixed‐stress language have to process stress in a foreign free‐stress language. Methods: We asked French‐speaking participants to process stress and vowel contrasts in Spanish, a foreign language that the participants did not know. Participants performed a discrimination task on Spanish word pairs differing either with respect to word stress (penultimate or final stressed word) or with respect to the final vowel while functional magnetic resonance imaging data was acquired. Results: Behavioral results showed lower accuracy and longer reaction times for discriminating stress contrasts than vowel contrasts. The contrast Stress > Vowel revealed an increased bilateral activation of regions shown to be associated with stress processing (i.e., supplementary motor area, insula, middle/superior temporal gyrus), as well as a stronger involvement of areas related to more domain‐general cognitive control functions (i.e., bilateral inferior frontal gyrus). The contrast Vowel > Stress showed an increased activation in regions typically associated with the default mode network (known for decreasing its activity during attentionally more demanding tasks). Conclusion: When processing Spanish stress contrasts as compared to processing vowel contrasts, native listeners of French activated to a higher degree anterior networks including regions related to cognitive control. They also show a decrease in regions related to the default mode network. These findings, together with the behavioral results, reflect the higher cognitive demand, and therefore, the larger difficulties, for French‐speaking listeners during stress processing as compared to vowel processing. [ABSTRACT FROM AUTHOR]

A cochlear implant (CI) can partially restore hearing in individuals with profound sensorineural hearing loss. However, electrical hearing with a CI is limited and highly variable. The current study aimed to better understand the different factors contributing to this variability by examining how age affects cognitive functions and cortical speech processing in CI users. Electroencephalography (EEG) was applied while two groups of CI users (young and elderly; N = 13 each) and normal-hearing (NH) listeners (young and elderly; N = 13 each) performed an auditory sentence categorization task, including semantically correct and incorrect sentences presented either with or without background noise. Event-related potentials (ERPs) representing earlier, sensory-driven processes (N1-P2 complex to sentence onset) and later, cognitive-linguistic integration processes (N400 to semantically correct/incorrect sentence-final words) were compared between the different groups and speech conditions. The results revealed reduced amplitudes and prolonged latencies of auditory ERPs in CI users compared to NH listeners, both at earlier (N1, P2) and later processing stages (N400 effect). In addition to this hearing-group effect, CI users and NH listeners showed a comparable background-noise effect, as indicated by reduced hit rates and reduced (P2) and delayed (N1/P2) ERPs in conditions with background noise. Moreover, we observed an age effect in CI users and NH listeners, with young individuals showing improved specific cognitive functions (working memory capacity, cognitive flexibility and verbal learning/retrieval), reduced latencies (N1/P2), decreased N1 amplitudes and an increased N400 effect when compared to the elderly. In sum, our findings extend previous research by showing that the CI users' speech processing is impaired not only at earlier (sensory) but also at later (semantic integration) processing stages, both in conditions with and without background noise. Using objective ERP measures, our study provides further evidence of strong age effects on cortical speech processing, which can be observed in both the NH listeners and the CI users. We conclude that elderly individuals require more effortful processing at sensory stages of speech processing, which however seems to be at the cost of the limited resources available for the later semantic integration processes. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Harbin University of Science & Technology is the property of Journal of Harbin University of Science & Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Abstract Various time‐frequency (T‐F) masks are being applied to sound source localization tasks. Moreover, deep learning has dramatically advanced T‐F mask estimation. However, existing masks are usually designed for speech separation tasks and are suitable only for single‐channel signals. A novel complex‐valued T‐F mask is proposed that reserves the head‐related transfer function (HRTF), customized for binaural sound source localization. In addition, because the convolutional neural network that is exploited to estimate the proposed mask takes binaural spectral information as the input and output, accurate binaural cues can be preserved. Compared with conventional T‐F masks that emphasize single speech source–dominated T‐F units, HRTF‐reserved masks eliminate the speech component while keeping the direct propagation path. Thus, the estimated HRTF is capable of extracting more reliable localization features for the final direction of arrival estimation. Hence, binaural sound source localization guided by the proposed T‐F mask is robust under noisy and reverberant acoustic environments. The experimental results demonstrate that the new T‐F mask is superior to conventional T‐F masks and lead to the better performance of sound source localization in adverse environments.

• Infant brain lateralization within STG predicted long-term language skills. • The effect was only observed for functional activation for speech perception. • Neither infant language skills nor structural measurements were predictors. • Results suggest the sensitivity of infant task-based fMRI in predicting later skills. • Results support the hypothesis that functional asymmetries are advantageous. Brain asymmetries are hypothesized to reduce functional duplication and thus have evolutionary advantages. The goal of this study was to examine whether early brain lateralization contributes to skill development within the speech-language domain. To achieve this goal, 25 infants (2–13 months old) underwent behavioral language examination and fMRI during sleep while listening to forward and backward speech, and then were assessed on various language skills at 55–69 months old. We observed that infant functional lateralization of the superior temporal gyrus (STG) for forward > backward speech was associated with phonological, vocabulary, and expressive language skills 4 to 5 years later. However, we failed to observe that infant language skills or the anatomical lateralization of STG were related to subsequent language skills. Overall, our findings suggest that infant functional lateralization of STG for speech perception may scaffold subsequent language acquisition, supporting the hypothesis that functional hemisphere asymmetries are advantageous. [ABSTRACT FROM AUTHOR]

This paper provides a methodology to better understand the relationships between different aspects of vocal fold motion, which are used as features in machine learning-based approaches for detecting respiratory infections from voice recordings. The relationships are derived through a joint multivariate analysis of the vocal fold oscillations of speakers. Specifically, the multivariate setting explores the displacements and velocities of the left and right vocal folds derived from recordings of five extended vowel sounds for each speaker (/aa/, /iy/, /ey/, /uw/, and /ow/). In this multivariate setting, the differences between the bivariate and conditional interactions are analyzed by information-theoretic quantities based on transfer entropy. Incorporation of the conditional quantities reveals information regarding the confounding factors that can influence the statistical interactions among other pairs of variables. This is demonstrated on a vector autoregressive process where the analytical derivations can be carried out. As a proof of concept, the methodology is applied on a clinically curated dataset of COVID-19. The findings suggest that the interaction between the vocal fold oscillations can change according to individuals and presence of any respiratory infection, such as COVID-19. The results are important in the sense that the proposed approach can be utilized to determine the selection of appropriate features as a supplementary or early detection tool in voice-based diagnostics in future studies.

Parkinson’s disease (PD) classification through speech has been an advancing field of research because of its ease of acquisition and processing. The minimal infrastructure requirements of the system have also made it suitable for telemonitoring applications. Researchers have studied the effects of PD on speech from various perspectives using different speech tasks. Typical speech deficits due to PD include voice monotony (e.g., monopitch), breathy or rough quality, and articulatory errors. In connected speech, these symptoms are more emphatic, which is also the basis for speech assessment in popular rating scales used for PD, like the Unified Parkinson’s Disease Rating Scale (UPDRS) and Hoehn and Yahr (HY). The current study introduces an innovative framework that integrates pitch-synchronous segmentation and an optimized set of features to investigate and analyze continuous speech from both PD patients and healthy controls (HC). Comparison of the proposed framework against existing methods has shown its superiority in classification performance and mitigation of overfitting in machine learning models. A set of optimal classifiers with unbiased decision-making was identified after comparing several machine learning models. The outcomes yielded by the classifiers demonstrate that the framework effectively learns the intrinsic characteristics of PD from connected speech, which can potentially offer valuable assistance in clinical diagnosis.

Two studies investigate the production and perception of speech chunks in Estonian. A corpus study examines to what degree the boundaries of syntactic constituents and frequent collocations influence the distribution of prosodic information in spontaneously spoken utterances. A perception experiment tests to what degree prosodic information, constituent structure, and collocation frequencies interact in the perception of speech chunks. Two groups of native Estonian speakers rated spontaneously spoken utterances for the presence of disjunctures, whilst listening to these utterances (N=47N=47) or reading them (N=40N=40). The results of the corpus study reveal a rather weak correspondence between the distribution of prosodic information and boundaries of the syntactic constituents and collocations. The results of the perception experiments demonstrate a strong influence of clause boundaries on the perception of prosodic discontinuities as prosodic breaks. Thus, the results indicate that there is no direct relationship between the semantico-syntactic characteristics of utterances and the distribution of prosodic information. The percept of a prosodic break relies on the rapid recognition of constituent structure, i.e. structural information.

Self-supervised learning has recently been implemented widely in speech processing areas, replacing conventional acoustic feature extraction to extract meaningful information from speech. One of the challenging applications of speech processing is to extract affective information from speech, commonly called speech emotion recognition. Until now, it is not clear the position of these speech representations compared to the classical acoustic feature. This paper evaluates nineteen self-supervised speech representations and one classical acoustic feature for five distinct speech emotion recognition datasets on the same classifier. We calculate the effect size among twenty speech representations to show the magnitude of relative differences from the top to the lowest performance. The top three are WavLM Large, UniSpeech-SAT Large, and HuBERT Large, with negligible effect sizes among them. The significance test supports the difference among self-supervised speech representations. The best prediction for each dataset is shown in the form of a confusion matrix to gain insights into the best performance of speech representations for each emotion category based on the training data from balanced vs. unbalanced datasets, English vs. Japanese corpus, and five vs. six emotion categories. Despite showing their competitiveness, this exploration of self-supervised learning for speech emotion recognition also shows their limitations on models pre-trained on small data and trained on unbalanced datasets.

The fully convolutional time-domain speech separation network (Conv-TasNet) has been used as a backbone model in various studies because of its structural excellence. To maximize the performance and efficiency of Conv-TasNet, we attempt to apply a neural architecture search (NAS). NAS is a branch of automated machine learning that automatically searches for an optimal model structure while minimizing human intervention. In this study, we introduce a candidate operation to define the search space of NAS for Conv-TasNet. In addition, we introduce a low computational cost NAS to overcome the limitations of the backbone model that consumes large GPU memory for training. Next, we determine the optimized separation module structures using two search strategies based on gradient descent and reinforcement learning. In addition, when NAS is simply applied, there is an imbalance in the updating of architecture parameters, which are NAS parameters. Therefore, we introduce an auxiliary loss method that is appropriate for the Conv-TasNet architecture for a balanced architecture parameter update of the entire model. Furthermore, we determine that the auxiliary loss technique mitigates the imbalance of architecture parameter updates and improves the separation accuracy.

This paper evaluates speech emotion and naturalness recognitions by utilizing deep learning models with multitask learning and single-task learning approaches. The emotion model accommodates valence, arousal, and dominance attributes known as dimensional emotion. The naturalness ratings are labeled on a five-point scale as dimensional emotion. Multitask learning predicts both dimensional emotion (as the main task) and naturalness scores (as an auxiliary task) simultaneously. The single-task learning predicts either dimensional emotion (valence, arousal, and dominance) or naturalness score independently. The results with multitask learning show improvement from previous studies on single-task learning for both dimensional emotion recognition and naturalness predictions. Within this study, single-task learning still shows superiority over multitask learning for naturalness recognition. The scatter plots of emotion and naturalness prediction scores against the true labels in multitask learning exhibit the lack of the model; it fails to predict the low and extremely high scores. The low score of naturalness prediction in this study is possibly due to a low number of samples of unnatural speech samples since the MSP-IMPROV dataset promotes the naturalness of speech. The finding that jointly predicting naturalness with emotion helps improve the performance of emotion recognition may be embodied in the emotion recognition model in future work.

Hearing loss is a common problem affecting the quality of life for thousands of people. However, many individuals with hearing loss are dissatisfied with the quality of modern hearing aids. Amplification is the main method of compensating for hearing loss in modern hearing aids. One common amplification technique is dynamic range compression, which maps audio signals onto a person’s hearing range using an amplification curve. However, due to the frequency dependent nature of the human cochlea, compression is often performed independently in different frequency bands. This paper presents a real-time multirate multiband amplification system for hearing aids, which includes a multirate channelizer for separating an audio signal into eleven standard audiometric frequency bands, and an automatic gain control system for accurate control of the steady state and dynamic behavior of audio compression as specified by ANSI standards. The spectral channelizer offers high frequency resolution with low latency of 5.4 ms and about $14\times $ improvement in complexity over a baseline design. Our automatic gain control includes a closed-form solution for satisfying any designated attack and release times for any desired compression parameters. The increased frequency resolution and precise gain adjustment allow our system to more accurately fulfill audiometric hearing aid prescriptions.

IntroductionThe effective fusion of text and audio information for categorical and dimensional speech emotion recognition (SER) remains an open issue, especially given the vast potential of deep neural networks (DNNs) to provide a tighter integration of the two.MethodsIn this contribution, we investigate the effectiveness of deep fusion of text and audio features for categorical and dimensional SER. We propose a novel, multistage fusion method where the two information streams are integrated in several layers of a DNN, and contrast it with a single-stage one where the streams are merged in a single point. Both methods depend on extracting summary linguistic embeddings from a pre-trained BERT model, and conditioning one or more intermediate representations of a convolutional model operating on log-Mel spectrograms.ResultsExperiments on the MSP-Podcast and IEMOCAP datasets demonstrate that the two fusion methods clearly outperform a shallow (late) fusion baseline and their unimodal constituents, both in terms of quantitative performance and qualitative behavior.DiscussionOverall, our multistage fusion shows better quantitative performance, surpassing alternatives on most of our evaluations. This illustrates the potential of multistage fusion in better assimilating text and audio information.

Background: Older adults process speech differently, but it is not yet clear how aging affects different levels of processing natural, continuous speech, both in terms of bottom-up acoustic analysis and top-down generation of linguistic-based predictions. We studied natural speech processing across the adult lifespan via electroencephalography (EEG) measurements of neural tracking.Goals: Our goals are to analyze the unique contribution of linguistic speech processing across the adult lifespan using natural speech, while controlling for the influence of acoustic processing. Moreover, we also studied acoustic processing across age. In particular, we focus on changes in spatial and temporal activation patterns in response to natural speech across the lifespan.Methods: 52 normal-hearing adults between 17 and 82 years of age listened to a naturally spoken story while the EEG signal was recorded. We investigated the effect of age on acoustic and linguistic processing of speech. Because age correlated with hearing capacity and measures of cognition, we investigated whether the observed age effect is mediated by these factors. Furthermore, we investigated whether there is an effect of age on hemisphere lateralization and on spatiotemporal patterns of the neural responses.Results: Our EEG results showed that linguistic speech processing declines with advancing age. Moreover, as age increased, the neural response latency to certain aspects of linguistic speech processing increased. Also acoustic neural tracking (NT) decreased with increasing age, which is at odds with the literature. In contrast to linguistic processing, older subjects showed shorter latencies for early acoustic responses to speech. No evidence was found for hemispheric lateralization in neither younger nor older adults during linguistic speech processing. Most of the observed aging effects on acoustic and linguistic processing were not explained by age-related decline in hearing capacity or cognition. However, our results suggest that the effect of decreasing linguistic neural tracking with advancing age at word-level is also partially due to an age-related decline in cognition than a robust effect of age.Conclusion: Spatial and temporal characteristics of the neural responses to continuous speech change across the adult lifespan for both acoustic and linguistic speech processing. These changes may be traces of structural and/or functional change that occurs with advancing age.

The ability to process speech in noise (SPiN) declines with age, with a detrimental impact on life quality. Music-making activities such as singing and playing a musical instrument have raised interest as potential prevention strategies for SPiN perception decline because of their positive impact on several brain system, especially the auditory system, which is critical for SPiN. However, the literature on the effect of musicianship on SPiN performance has yielded mixed results. By critically assessing the existing literature with a systematic review and a meta-analysis, we aim to provide a comprehensive portrait of the relationship between music-making activities and SPiN in different experimental conditions. 38/49 articles, most focusing on young adults, were included in the quantitative analysis. The results show a positive relationship between music-making activities and SPiN, with the strongest effects found in the most challenging listening conditions, and little to no effect in less challenging situations. This pattern of results supports the notion of a relative advantage for musicians on SPiN performance and clarify the scope of this effect. However, further studies, especially with older adults, using adequate randomization methods, are needed to extend the present conclusions and assess the potential for musical activities to be used to mitigate SPiN decline in seniors.

Abstract Introduction The paper examines the discrimination of lexical stress contrasts in a foreign language from a neural perspective. The aim of the study was to identify the areas associated with word stress processing (in comparison with vowel processing), when listeners of a fixed‐stress language have to process stress in a foreign free‐stress language. Methods We asked French‐speaking participants to process stress and vowel contrasts in Spanish, a foreign language that the participants did not know. Participants performed a discrimination task on Spanish word pairs differing either with respect to word stress (penultimate or final stressed word) or with respect to the final vowel while functional magnetic resonance imaging data was acquired. Results Behavioral results showed lower accuracy and longer reaction times for discriminating stress contrasts than vowel contrasts. The contrast Stress > Vowel revealed an increased bilateral activation of regions shown to be associated with stress processing (i.e., supplementary motor area, insula, middle/superior temporal gyrus), as well as a stronger involvement of areas related to more domain‐general cognitive control functions (i.e., bilateral inferior frontal gyrus). The contrast Vowel > Stress showed an increased activation in regions typically associated with the default mode network (known for decreasing its activity during attentionally more demanding tasks). Conclusion When processing Spanish stress contrasts as compared to processing vowel contrasts, native listeners of French activated to a higher degree anterior networks including regions related to cognitive control. They also show a decrease in regions related to the default mode network. These findings, together with the behavioral results, reflect the higher cognitive demand, and therefore, the larger difficulties, for French‐speaking listeners during stress processing as compared to vowel processing.

A cochlear implant (CI) can partially restore hearing in individuals with profound sensorineural hearing loss. However, electrical hearing with a CI is limited and highly variable. The current study aimed to better understand the different factors contributing to this variability by examining how age affects cognitive functions and cortical speech processing in CI users. Electroencephalography (EEG) was applied while two groups of CI users (young and elderly; N = 13 each) and normal-hearing (NH) listeners (young and elderly; N = 13 each) performed an auditory sentence categorization task, including semantically correct and incorrect sentences presented either with or without background noise. Event-related potentials (ERPs) representing earlier, sensory-driven processes (N1-P2 complex to sentence onset) and later, cognitive-linguistic integration processes (N400 to semantically correct/incorrect sentence-final words) were compared between the different groups and speech conditions. The results revealed reduced amplitudes and prolonged latencies of auditory ERPs in CI users compared to NH listeners, both at earlier (N1, P2) and later processing stages (N400 effect). In addition to this hearing-group effect, CI users and NH listeners showed a comparable background-noise effect, as indicated by reduced hit rates and reduced (P2) and delayed (N1/P2) ERPs in conditions with background noise. Moreover, we observed an age effect in CI users and NH listeners, with young individuals showing improved specific cognitive functions (working memory capacity, cognitive flexibility and verbal learning/retrieval), reduced latencies (N1/P2), decreased N1 amplitudes and an increased N400 effect when compared to the elderly. In sum, our findings extend previous research by showing that the CI users’ speech processing is impaired not only at earlier (sensory) but also at later (semantic integration) processing stages, both in conditions with and without background noise. Using objective ERP measures, our study provides further evidence of strong age effects on cortical speech processing, which can be observed in both the NH listeners and the CI users. We conclude that elderly individuals require more effortful processing at sensory stages of speech processing, which however seems to be at the cost of the limited resources available for the later semantic integration processes.

Within speech processing, articulatory-to-acoustic mapping (AAM) methods can apply ultrasound tongue imaging (UTI) as an input. (Micro)convex transducers are mostly used, which provide a wedge-shape visual image. However, this process is optimized for the visual inspection of the human eye, and the signal is often post-processed by the equipment. With newer ultrasound equipment, now it is possible to gain access to the raw scanline data (i.e., ultrasound echo return) without any internal post-processing. In this study, we compared the raw scanline representation with the wedge-shaped processed UTI as the input for the residual network applied for AAM, and we also investigated the optimal size of the input image. We found no significant differences between the performance attained using the raw data and the wedge-shaped image extrapolated from it. We found the optimal pixel size to be 64 × 43 in the case of the raw scanline input, and 64 × 64 when transformed to a wedge. Therefore, it is not necessary to use the full original 64 × 842 pixels raw scanline, but a smaller image is enough. This allows for the building of smaller networks, and will be beneficial for the development of session and speaker-independent methods for practical applications. AAM systems have the target application of a "silent speech interface", which could be helpful for the communication of the speaking-impaired, in military applications, or in extremely noisy conditions. [ABSTRACT FROM AUTHOR]

The development of skills related to executive function (EF) in infancy, including their emergence, underlying neural mechanisms, and interconnections to other cognitive skills, is an area of increasing research interest. Here, we report on findings from a multidimensional dataset demonstrating that infants' behavioral performance on a flexible learning task improved across development and that the task performance is highly correlated with both neural structure and neural function. The flexible learning task probed infants' ability to learn two different associations, concurrently, over 16 trials, requiring multiple skills relevant to EF. We examined infants' neural structure by measuring myelin density in the brain, using a novel macromolecular proton fraction (MPF) mapping method. We further examined an important neural function of speech processing by characterizing the mismatch response (MMR) to speech contrasts using magnetoencephalography (MEG). All measurements were performed longitudinally in monolingual English‐learning infants at 7‐ and 11‐months of age. At the group level, 11‐month‐olds, but not 7‐month‐olds, demonstrated evidence of learning both associations in the behavioral task. Myelin density in the prefrontal region at 7 months of age was found to be highly predictive of behavioral task performance at 11 months of age, suggesting that myelination may support the development of these skills. Furthermore, a machine‐learning regression analysis revealed that individual differences in the behavioral task are predicted by concurrent neural speech processing at both ages, suggesting that these skills do not develop in isolation. Together, these cross‐modality results revealed novel insights into EF‐related skills. Highlight: Monolingual infants demonstrated flexible learning on a task requiring executive function skills at 11 months, but not at 7 months.Infants' myelin density at 7 months is highly predictive of their behavioral performance in the flexible learning task at 11 months of age.Individual differences in the flexible learning task performance are also correlated with concurrent neural processing of speech at both ages. [ABSTRACT FROM AUTHOR]

The motivation of this paper is to propose a set of best-quality linguistic materials for Vietnamese speech processing, which can be used for Vietnamese TTS and ASR problems. This proposed material includes: (1) a pronunciation dictionary, which adapts from X-SAMPA, (2) a rule-based grapheme to phoneme for Vietnamese. In order to test and evaluate, we have built a Vietnamese TTS system based on the Merlin engine, using the above materials, and evaluating the quality of speech and the accuracy of pronunciation. The results show that the applicability of these materials is favorable for further research and development on Vietnamese speech processing. [ABSTRACT FROM AUTHOR]

BackgroundLanguage is a highly lateralized function, with typically developing individuals showing left hemispheric specialization. Individuals with autism spectrum disorder (ASD) often show reduced or reversed hemispheric lateralization in response to language. However, it is unclear when this difference emerges and whether or not it can serve as an early ASD biomarker. Additionally, atypical language lateralization is not specific to ASD as it is also seen more frequently in individuals with mixed- and left-handedness. Here, we examined early asymmetry patterns measured through neural responses to speech sounds at 12 months and behavioral observations of handedness at 36 months in children with and without ASD.MethodsThree different groups of children participated in the study: low-risk controls (LRC), high risk for ASD (HRA; infants with older sibling with ASD) without ASD, and HRA infants who later receive a diagnosis of ASD (ASD). Event-related potentials (ERPs) to speech sounds were recorded at 12 months. Utilizing a novel observational approach, handedness was measured by hand preference on a variety of behaviors at 36 months.ResultsAt 12 months, lateralization patterns of ERPs to speech stimuli differed across the groups with the ASD group showing reversed lateralization compared to the LRC group. At 36 months, factor analysis of behavioral observations of hand preferences indicated a one-factor model with medium to high factor loadings. A composite handedness score was derived; no group differences were observed. There was no association between lateralization to speech at 12 months and handedness at 36 months in the LRC and HRA groups. However, children with ASD did show an association such that infants with lateralization patterns more similar to the LRC group at 12 months were stronger right-handers at 36 months.ConclusionsThese results highlight early developmental patterns that might be specific to ASD, including a potential early biomarker of reversed lateralization to speech stimuli at 12 months, and a relation between behavioral and neural asymmetries. Future investigations of early asymmetry patterns, especially atypical hemispheric specialization, may be informative in the early identification of ASD.

In the cocktail party situation, people with normal hearing usually follow a single speaker among multiple concurrent ones. However, there is no agreement in the literature as to whether the background is segregated into multiple streams/speakers. The current study varied the number of concurrent speech streams and investigated target detection and memory for the contents of a target stream as well as the processing of distractors. A malevoiced target stream was either presented alone (single-speech), together with one male-voiced distractor (one-distractor), or a male- and a femalevoiced distractor (two-distractor). Behavioral measures of target detection and content tracking performance as well as target- and distractor detection related event-related brain potentials (ERPs) were assessed. We found that the N2 amplitude decreased whereas the P3 amplitude increased from the single-speech to the concurrent speech streams conditions. Importantly, the behavioral effect of distractors differed between the conditions with one vs. two distractor speech streams and the non-zero voltages in the N2 time window for distractor numerals and in the P3 time window for syntactic violations appearing in the non-target speech stream significantly differed between the one- and two-distractor conditions for the same (male) speaker. These results support the notion that the two background speech streams are segregated, as they show that distractors and syntactic violations appearing in the non-target streams are processed even when two speech non-target speech streams are delivered together with the target stream. [ABSTRACT FROM AUTHOR]

One of the obstacles in developing speech emotion recognition (SER) systems is the data scarcity problem, i.e., the lack of labeled data for training these systems. Data augmentation is an effective method for increasing the amount of training data. In this paper, we propose a cycle-generative adversarial network (cycle-GAN) for data augmentation in the SER systems. For each of the five emotions considered, an adversarial network is designed to generate data that have a similar distribution to the main data in that class but have a different distribution to those of other classes. These networks are trained in an adversarial way to produce feature vectors similar to those in the training set, which are then added to the original training sets. Instead of using the common cross-entropy loss to train cycle-GANs, we use the Wasserstein divergence to mitigate the gradient vanishing problem and to generate high-quality samples. The proposed network has been applied to SER using the EMO-DB dataset. The quality of the generated data is evaluated using two classifiers based on support vector machine and deep neural network. The results showed that the recognition accuracy in unweighted average recall was about 83.33%, which is better than the baseline methods compared. [ABSTRACT FROM AUTHOR]