Search

The brain evolved as a sensory-motor machine that drives behavior while being linked to the world through sensors. Human cognition abstracts from these sensory-motor roots but retains intimate ties. The brain's structure reflects this history. How do neural processes at different distances from the sensory and motor surfaces integrate to achieve meaningful and grounded cognition? This is a challenge given the time-continuous and graded nature of sensory-motor processing, which enables continuous online updating. It is also a major challenge to understanding development and autonomous learning, in which the coupling across functional boundaries evolves under the influence of online activation patterns.

We investigated the impact of teachers' multimodal cues on L2 word learning in naturalistic teaching. 169 university students randomly watched 12 of 54 clips of English vocabulary instructions and took subsequent word recognition and learning tests. The learning outcomes were analysed as a function of teachers' prosodic, linguistic and gestural input during the instruction of each vocabulary while controlling for students' characteristics and varying teachers' influences. Results showed that a shorter mean length of utterances, fewer L2 English words, and more questions for students and “phrase” teaching predicted better learning outcomes. Furthermore, students learning improved with teachers' slower speaking rate but fewer pauses and more iconic gestures. These results were robust even after controlling for other significant factors such as students' English proficiency, working memory, degree of liking of teachers and different teachers. Overall, multimodal cues enhance L2 vocabulary learning, with implications for educators, linguists, and cognitive scientists.

It has long been assumed that infants' ability to discriminate between languages stems from their sensitivity to speech rhythm, i.e., organized temporal structure of vowels and consonants in a language. However, the relationship between speech rhythm and language discrimination has not been directly demonstrated. Here, we use computational modeling and train models of speech perception with and without access to information about rhythm. We test these models on language discrimination, and find that access to rhythm does not affect the success of the model in replicating infant language discrimination results. Our findings challenge the relationship between rhythm and language discrimination, and have implications for theories of language acquisition.

Speech perception involves storing and integrating sequentially presented items. Recent work in cognitive neuroscience has identified temporal and contextual characteristics in humans' neural encoding of speech that may facilitate this temporal processing. In this study, we simulated similar analyses with representations extracted from a computational model that was trained on unlabelled speech with the learning objective of predicting upcoming acoustics. Our simulations revealed temporal dynamics similar to those in brain signals, implying that these properties can arise without linguistic knowledge. Another property shared between brains and the model is that the encoding patterns of phonemes support some degree of cross-context generalization. However, we found evidence that the effectiveness of these generalizations depends on the specific contexts, which suggests that this analysis alone is insufficient to support the presence of context-invariant encoding.

Autistic and neurotypical children do not handle audiovisual speech in the same manner. Current evidence suggests that this difference occurs at the level of cue combination. Here, we test whether differences in autistic and neurotypical audiovisual speech perception can be explained by a neural theory of sensory perception in autism, which proposes that heightened levels of neural excitation can account for sensory differences in autism. Through a linking hypothesis that integrates a standard probabilistic cognitive model of cue integration with representations of neural activity, we derive a model that can simulate audio-visual speech perception at a neural population level. Simulations of an audiovisual lexical identification task demonstrate that heightened levels of neural excitation at the level of cue combination cannot account for the observed differences in autistic and neurotypical children's audiovisual speech perception.

Bilinguals show linguistic transfer effects at several processing levels. Focusing on phonology, we investigate the transfer of optional assimilation rules during speech production. Specifically, we examine to what extent bilinguals apply their native assimilation rule and/or fail to apply an L2 assimilation rule in their L2 speech. Both early and advanced late English-French bilinguals read a short French text. Using a speech recognizer with specific pronunciation variants, we found that the late bilinguals showed evidence for transfer of place assimilation, as well as a reduction in the amount of voicing assimilation compared to that of native French controls. The early bilinguals did not differ from the French controls in terms of place assimilation, but their voicing assimilation rate was intermediate between those of the French controls and the late bilinguals.

This study investigated whether infants use iconicity in speech and gesture to interpret words. Thirty-six 14-17-month-old infants participated in a preferential looking task in which they heard a spoken non-word (e.g., “zudzud”) while observing a small and a large object (e.g., a small and a large square). All infants were presented with an iconic cue for object size (small or large) in 1) the pitch of the spoken non-word (high vs. low), 2) in gesture (small or large), or 3) congruently in both pitch and gesture (e.g., a high pitch and a small gesture indicating a small square). Infants did not show a preference for congruently sized objects in any iconic cue condition. Bayes Factor analyses supported the null hypotheses. In conclusion, we found no evidence that infants link the pitch of spoken non-words, or the iconic gestures accompanying those spoken non-words, to object size.

Experiments in psycholinguistics allow us to test hypotheses and build theories. However, psycholinguistic experiments often suffer from low ecological validity, because participants are often required to perform an unusual task in the face of unusual materials. In the current experiment, we test the predictions of Noisy Channel Processing in a naturalistic task: identifying the lyrics of a song. We conducted an experiment where participants heard short excerpts from songs and then indicated which one out of four possible transcriptions they had heard. We found that the predictions of Noisy Channel Processing bear out: options with higher prior and likelihood were chosen more often by participants as the perceived song lyrics. Thus, Noisy Channel Processing is successful in explaining the everyday phenomenon of mis-heard song lyrics. More broadly, this suggests that Noisy Channel Processing captures everyday language processing, and that it is not dependent on unnatural experimental tasks and materials.

Although children are on the receiving end of pedagogical questions (asked with an intent to teach) and information-seeking questions (asked with an intent to seek information), little is known about how children differentiate between the two types of questions. Here, we tested if parents spontaneously modulate their prosody when asking pedagogical as opposed to information- seeking questions. To test this, we asked 35 parent- child pairs to engage in a learning game where parents were asking questions while being in the role of a teacher (pedagogical) or in the role of a student (information- seeking). Next, 128 naïve listeners judged the questions produced by parents. We found that naïve listeners could reliably differentiate the two types of questions on the basis of prosody alone. This finding highlights the importance of prosody as a mechanism for communicating pedagogical intent in parent-child interactions.

Voice identity recognition in auditory-only conditions is facilitated by knowing the face of the speaker. This effect is called the ‘face-benefit'. Based on neuroscience findings, we hypothesized that this benefit emerges from two factors: First, a generative world model integrates information from multiple senses to better predict the sensory dynamics. Second, the model substitutes absent sensory information, e.g., facial dynamics, with internal simulations. We have developed a deep generative model that learns to simulate such multisensory dynamics, developing latent speaker characteristic contexts. We trained our model on synthetic audio-visual data of talking faces and tested its ability to recognize speakers from their voice only. We found that the model recognizes previously seen speakers better than previously unseen speakers when given their voice only. The modeling results confirm that multisensory simulations and predictive substitutions of missing visual inputs result in the face-benefit

Most alphabets are based on visual coding of phonemes and syllables, and similar visual codes were developed for visually conveying the sounds of speech to deaf people. Notably, Cued Speech (CS) allows for syllables to be specified by a combination of lip configuration, hand location and hand shape. The use of this communication system has been proven to improve general language skills in a deaf community characterized by low literacy. Meanwhile, the mechanisms of CS perception remain largely unknown.In an fMRI study involving 3 groups of participants (deaf and hearing people proficient in CS and a group of hearing people naïve to CS), we identify the brain areas that process and, more specifically, encode the various components of CS. Particular attention is given to the role of expertise, and to the links between CS and reading, two coexisting visual codes for language that both compete and support each other.

This research explores the potential of machine learning (ML) in identifying vocal markers for schizophrenia. While previous research showed that voice-based ML models can accurately predict schizophrenia diagnosis and symptoms, it is unclear to what extent such ML markers generalize to different clinical subpopulations and languages: the assessment of generalization performance is however crucial for testing their clinical applicability.We systematically examined voice-based ML model performance on a large cross-linguistic dataset (3 languages: Danish, German, Chinese). Employing a rigorous pipeline to minimize overfitting, including cross-validated training sets and multilingual models, we assessed generalization on participants with schizophrenia and controls speaking the same or different languages. Model performance was comparable to state-of-the art findings (F1-score ~ 0.75) within the same language; however, models did not generalize well - showing a substantial decrease - when tested on new languages, and the performance of multilingual models was also generally low (F1-score ~ 0.50).

The McGurk effect is a demonstration of the multimodal nature of speech perception; listening to /b/ while watching visual mouth movements for /g/ is expected to result in a “fusion” perception of /d/. A majority of studies on the effect use isolated syllables, whereas our goal was to enhance ecological validity by examining word stimuli. We varied task (forced-choice vs. open-ended) and stimuli (words vs. non-words) between participants. In the word condition, all three stimuli formed words (e.g., beer/deer/gear), and in the non-word condition, the B, D, or G stimulus was a word while the other two were nonwords (e.g., besk/desk/gesk). Fusion responses were much lower than in previous studies, but importantly, participants showed the most fusion responses when the D stimulus was a word and B and G were non-words. These results challenge assumptions about the underlying mechanisms of the McGurk effect, arguing against a purely perceptual illusion.

By 2050, 1 in 4 people worldwide will be living with hearing impairment. We propose a digital Speech Hearing Screener (dSHS) using short nonsense word recognition to measure speech-hearing ability. The importance of hearing screening is increasing due to the anticipated increase in individuals with hearing impairment globally. We compare dSHS outcomes with standardized pure-tone averages (PTA) and speech-recognition thresholds (SRT). Fifty participants (aged 55 or older underwent pure-tone and speech-recognition thresholding. One-way ANOVA was used to compare differences between hearing impaired and hearing not-impaired groups, by the dSHS, with a clinical threshold of moderately impaired hearing at 35 dB and severe hearing impairment at 50 dB. dSHS results significantly correlated with PTAs/SRTs. ANOVA results revealed the dSHS was significantly different (F(1,47) = 38.1, p < 0.001) between hearing impaired and unimpaired groups. Classification analysis using a 35 dB threshold, yielded accuracy of 85.7% for PTA-based impairment and 81.6% for SRT-based impairment. At a 50 dB threshold, dSHS classification accuracy was 79.6% for PTA-based impairment (Negative Predictive Value (NPV)-93%) and 83.7% (NPV-100%) for SRT-based impairment. The dSHS successfully differentiates between hearing-impaired and unimpaired individuals in under 3 min. This hearing screener offers a time-saving, in-clinic hearing screening to streamline the triage of those with likely hearing impairment to the appropriate follow-up assessment, thereby improving the quality of services. Future work will investigate the ability of the dSHS to help rule out hearing impairment as a cause or confounder in clinical and research applications. [ABSTRACT FROM AUTHOR]

Deep convolutional neural network (CNN) has been widely used in speech recognition technology. The model based on deep CNN can effectively improve the quality of human-computer interaction. However, the existing CNN with fixed convolutional kernel size has a disadvantage on extracting data features. It is hard to effectively identify whether the extracted features sufficient or not. As a result, a self-tuning convolutional kernel (STCK) algorithm is proposed to solve the mentioned problem. Firstly, the computational process of STCK algorithm is derived. Then the calculation formula of the convolutional kernel size is obtained. Meanwhile, Bark-spectrum is introduced to extract the spectrogram of speech signal, which is used as the CNN input to adapt to the human hearing. In addition, the data enhancement strategies are proposed, namely frame channel shielding and Bark-band channel shielding. The presented strategies can further improve the generalization ability of the recognition model. The experimental results show that, compared with another two models (the CNN model without STCK algorithm and the CNN model without the data enhancement strategy), the training loss of the proposed method is minimum. And the recognition error rates for the test samples are reduced by 3.9% and 1%, respectively. [ABSTRACT FROM AUTHOR]

This paper describes how automatic speech recognition (ASR) can be used to explore electronic patient records (EPR) in a sterile environment. As an information display, we used a system that was developed for the presentation of patient data and for supporting surgical hand disinfection. The speech recognition was performed using the Mozilla Web Speech API. A room microphone, a Jabra Box, and a directional microphone, a RØDE NT-USB Mini, were tested at various positions. Interactions with the EPR are triggered with a list of commands. The interaction was working as indented and test persons reported the system as intuitive. [ABSTRACT FROM AUTHOR]

Speech recognition is a critical task in spoken language applications. Globally known models such as DeepSpeech2 are effective for English speech recognition, however, they are not well-suited for languages like Arabic. This paper is interested in recognizing the Arabic language, especially Modern Standard Arabic (MSA). This paper proposed two models that utilize "Franco-Arabic" as an encoding mechanism and additional enhancements to recognize MSA. The first model uses Mel-Frequency Cepstral Coefficients (MFCCs) as input features, while the second employs six sequential Gated Recurrent Unit (GRU) layers. Each model is then followed by a fully connected layer with a dropout layer which helped reduce overfitting. The Connectionist Temporal Classification (CTC) loss is used to calculate the prediction error and to maximize the correct transcription likelihood. Two experiments were conducted for each model. The first experiment involved 41 h of continuous speech over 15 epochs. Whereas, the second one utilized 69 h over 30 epochs. The experiments showed that the first model excels in speed while the second excels in accuracy, and both outperformed the well-known DeepSpeech2. [ABSTRACT FROM AUTHOR]

Background: In cochlear implant (CI) treatment, there is a large variability in outcome. The aim of our study was to identify the independent audiometric measures that are most directly relevant for describing this variability in outcome characteristics of CI recipients. An extended audiometric test battery was used with selected adult patients in order to characterize the full range of CI outcomes. Methods: CI users were recruited for this study on the basis of their postoperative results and divided into three groups: low (1st quartile), moderate (medium decentile), and high hearing performance (4th quartile). Speech recognition was measured in quiet by using (i) monosyllabic words (40–80 dB SPL), (ii) speech reception threshold (SRT) for numbers, and (iii) the German matrix test in noise. In order to reconstruct demanding everyday listening situations in the clinic, the temporal characteristics of the background noise and the spatial arrangements of the signal sources were varied for tests in noise. In addition, a survey was conducted using the Speech, Spatial, and Qualities (SSQ) questionnaire and the Listening Effort (LE) questionnaire. Results: Fifteen subjects per group were examined (total N = 45), who did not differ significantly in terms of age, time after CI surgery, or CI use behavior. The groups differed mainly in the results of speech audiometry. For speech recognition, significant differences were found between the three groups for the monosyllabic tests in quiet and for the sentences in stationary (S0°N0°) and fluctuating (S0°NCI) noise. Word comprehension and sentence comprehension in quiet were both strongly correlated with the SRT in noise. This observation was also confirmed by a factor analysis. No significant differences were found between the three groups for the SSQ questionnaire and the LE questionnaire results. The results of the factor analysis indicate that speech recognition in noise provides information highly comparable to information from speech intelligibility in quiet. Conclusions: The factor analysis highlighted three components describing the postoperative outcome of CI patients. These were (i) the audiometrically measured supra-threshold speech recognition and (ii) near-threshold audibility, as well as (iii) the subjective assessment of the relationship to real life as determined by the questionnaires. These parameters appear well suited to setting up a framework for a test battery to assess CI outcomes. [ABSTRACT FROM AUTHOR]

As per the universal principles of usability and sustainable development goal defined for reducing inequality, each individual will have special abilities and challenges. Individuals with visual challenges have trouble interacting with digital platforms. In order to achieve inclusivity, there is a need to integrate universal accessibility into the web portals which are used as the most popular digital platforms. This study mainly focuses on the requirements of visually impaired users. The research work discusses the proposed approach for a telemedicine web platform by integrating voice navigation system. With this system, users can orally interact with the platform by using defined commands. Users can also receive the audio feedback from the computer for the specific command. The proposed speech recognition engine is implemented using deep learning models and tested on various browsers. The engine captures the commands in the form of user inputs and generates the proper audio feedback after executing the commands. Users with visual impairment have been involved in the evaluation by allowing them to interact on the telemedicine platform with verbal commands. The evaluation questions have been asked after each interaction to capture the response time, accuracy, experience and satisfaction. The outcome of the evaluation shows that individuals are showing significant progress in accessing required information and navigating the web pages on their choice of browsers. The study also observed that speech recognition engine along with speech grammar, acoustic model and synthesis has improved the usage of the system for all types of users. Ultimately, integrating voice navigation into web platforms provides a satisfactory experience to the users and improves accessibility, inclusivity and reduces inequality. [ABSTRACT FROM AUTHOR]

Speech digit recognition research is growing decisively, and a bulk of digit recognition algorithms are used in European and a few Asian languages. Kui is a low-resourced tribal language locally used in several states of India. Despite its significance, there is not much research on Kui's speech. This research aims to present an in-depth analysis of novel Kui digit recognition using predefined machine learning (ML) techniques. For this purpose, we first gathered spoken numbers i.e. from 0 to 9 of eight different speakers containing a total of 200 words. Secondly, we choose the numbers: ଶନୂ (zero), ଏକ (one), ଦୁଇ (two), ତିନି (three), ସାରି (four), ପାସ (five), ସଅ (six), ସାତ (seven), ଆଟ (eight), ନଅ (nine). Meanwhile, we build nine different ML models to recognize Kui digits that take the Mel-frequency cepstral coefficients (MFCCs) method to extract the relevant features for model predictions. Finally, we compared the performance of ML models for both augmented and non-augmented Kui data. The result shows that the SVM+Augmentation method for Kui digit recognition combined obtained the highest accuracy of 83% than other methods. Moreover, the difficulties and potential prospects for Kui digit recognition are also highlighted in this work. [ABSTRACT FROM AUTHOR]

Abstract By 2050, 1 in 4 people worldwide will be living with hearing impairment. We propose a digital Speech Hearing Screener (dSHS) using short nonsense word recognition to measure speech-hearing ability. The importance of hearing screening is increasing due to the anticipated increase in individuals with hearing impairment globally. We compare dSHS outcomes with standardized pure-tone averages (PTA) and speech-recognition thresholds (SRT). Fifty participants (aged 55 or older underwent pure-tone and speech-recognition thresholding. One-way ANOVA was used to compare differences between hearing impaired and hearing not-impaired groups, by the dSHS, with a clinical threshold of moderately impaired hearing at 35 dB and severe hearing impairment at 50 dB. dSHS results significantly correlated with PTAs/SRTs. ANOVA results revealed the dSHS was significantly different (F(1,47) = 38.1, p

The traditional translation robot is limited to the translation of single-mode text images and text videos, which has the problem of low translation accuracy. Therefore, speech recognition and intelligent translation in multimodal human–computer interaction (HCI) system are proposed. First, the network structure of speech recognition model in multi-channel HCI system is established, and the multi-head self-attention mechanism is constructed. Then, the artificial intelligence voice wake-up function is designed, and a multimodal machine translation model is constructed. On this basis, selective attention is added to obtain visual recognition of perceived text, and the decoder is used for multimodal gating fusion to realize the output of encoder translation results. Experimental results show that this method has high BLUE value and high translation accuracy.

In this paper, we present a novel approach for text-independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (Wav2Vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained using forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.

As per the universal principles of usability and sustainable development goal defined for reducing inequality, each individual will have special abilities and challenges. Individuals with visual challenges have trouble interacting with digital platforms. In order to achieve inclusivity, there is a need to integrate universal accessibility into the web portals which are used as the most popular digital platforms. This study mainly focuses on the requirements of visually impaired users. The research work discusses the proposed approach for a telemedicine web platform by integrating voice navigation system. With this system, users can orally interact with the platform by using defined commands. Users can also receive the audio feedback from the computer for the specific command. The proposed speech recognition engine is implemented using deep learning models and tested on various browsers. The engine captures the commands in the form of user inputs and generates the proper audio feedback after executing the commands. Users with visual impairment have been involved in the evaluation by allowing them to interact on the telemedicine platform with verbal commands. The evaluation questions have been asked after each interaction to capture the response time, accuracy, experience and satisfaction. The outcome of the evaluation shows that individuals are showing significant progress in accessing required information and navigating the web pages on their choice of browsers. The study also observed that speech recognition engine along with speech grammar, acoustic model and synthesis has improved the usage of the system for all types of users. Ultimately, integrating voice navigation into web platforms provides a satisfactory experience to the users and improves accessibility, inclusivity and reduces inequality.

Abstract Artificial intelligence (AI) telephone is reliable for the follow‐up and management of hypertensives. It takes less time and is equivalent to manual follow‐up to a high degree. We conducted a reliability study to evaluate the efficiency of AI telephone follow‐up in the management of hypertension. During May 18 and June 30, 2020, 350 hypertensives managed by the Pengpu Community Health Service Center in Shanghai were recruited for follow‐up, once by AI and once by a human. The second follow‐up was conducted within 3–7 days (mean 5.5 days). The mean length time of two calls were compared by paired t‐test, and Cohen's Kappa coefficient was used to evaluate the reliability of the results between the two follow‐up visits. The mean length time of AI calls was shorter (4.15 min) than that of manual calls (5.24 min, P

Abstract Multilingual speakers can find speech recognition in everyday environments like restaurants and open-plan offices particularly challenging. In a world where speaking multiple languages is increasingly common, effective clinical and educational interventions will require a better understanding of how factors like multilingual contexts and listeners’ language proficiency interact with adverse listening environments. For example, word and phrase recognition is facilitated when competing voices speak different languages. Is this due to a “release from masking” from lower-level acoustic differences between languages and talkers, or higher-level cognitive and linguistic factors? To address this question, we created a “one-man bilingual cocktail party” selective attention task using English and Mandarin speech from one bilingual talker to reduce low-level acoustic cues. In Experiment 1, 58 listeners more accurately recognized English targets when distracting speech was Mandarin compared to English. Bilingual Mandarin–English listeners experienced significantly more interference and intrusions from the Mandarin distractor than did English listeners, exacerbated by challenging target-to-masker ratios. In Experiment 2, 29 Mandarin–English bilingual listeners exhibited linguistic release from masking in both languages. Bilinguals experienced greater release from masking when attending to English, confirming an influence of linguistic knowledge on the “cocktail party” paradigm that is separate from primarily energetic masking effects. Effects of higher-order language processing and expertise emerge only in the most demanding target-to-masker contexts. The “one-man bilingual cocktail party” establishes a useful tool for future investigations and characterization of communication challenges in the large and growing worldwide community of Mandarin–English bilinguals.

Abstract Most of the adversarial attacks against speech recognition systems focus on specific adversarial perturbations, which are generated by adversaries for each normal example to achieve the attack. Universal adversarial perturbations (UAPs), which are independent of the examples, have recently received wide attention for their enhanced real-time applicability and expanded threat range. However, most of the UAP research concentrates on the image domain, and less on speech. In this paper, we propose a staged perturbation generation method that constructs CommanderUAP, which achieves a high success rate of universal adversarial attack against speech recognition models. Moreover, we apply some methods from model training to improve the generalization in attack and we control the imperceptibility of the perturbation in both time and frequency domains. In specific scenarios, CommanderUAP can also transfer attack some commercial speech recognition APIs.

Zaki Brahmi,1 Mohammad Mahyoob,2 Mohammed Al-Sarem,1 Jeehaan Algaraady,3 Khadija Bousselmi,4 Abdulaziz Alblwi1 1Department of Computer Science, Taibah University, Madina, Kingdom of Saudi Arabia; 2Department of Languages and Translation, Taibah University, Madina, Kingdom of Saudi Arabia; 3Department of English, Taiz University, Taiz, Yemen; 4Department of Computer Science, LISTIC, University of Savoie Mont Blanc, Chambéry, FranceCorrespondence: Mohammad Mahyoob, Email mqassem@taibahu.edu.saPurpose: Speech disorders profoundly impact the overall quality of life by impeding social operations and hindering effective communication. This study addresses the gap in systematic reviews concerning machine learning-based assistive technology for individuals with speech disorders. The overarching purpose is to offer a comprehensive overview of the field through a Systematic Literature Review (SLR) and provide valuable insights into the landscape of ML-based solutions and related studies.Methods: The research employs a systematic approach, utilizing a Systematic Literature Review (SLR) methodology. The study extensively examines the existing literature on machine learning-based assistive technology for speech disorders. Specific attention is given to ML techniques, characteristics of exploited datasets in the training phase, speaker languages, feature extraction techniques, and the features employed by ML algorithms.Originality: This study contributes to the existing literature by systematically exploring the machine learning landscape in assistive technology for speech disorders. The originality lies in the focused investigation of ML-speech recognition for impaired speech disorder users over ten years (2014– 2023). The emphasis on systematic research questions related to ML techniques, dataset characteristics, languages, feature extraction techniques, and feature sets adds a unique and comprehensive perspective to the current discourse.Findings: The systematic literature review identifies significant trends and critical studies published between 2014 and 2023. In the analysis of the 65 papers from prestigious journals, support vector machines and neural networks (CNN, DNN) were the most utilized ML technique (20%, 16.92%), with the most studied disease being Dysarthria (35/65, 54% studies). Furthermore, an upsurge in using neural network-based architectures, mainly CNN and DNN, was observed after 2018. Almost half of the included studies were published between 2021 and 2022).Keywords: speech disorder, speech recognition, dysarthria, machine learning, assistive technologies

Speech-recognition tests are widely used in both clinical and research audiology. The purpose of this study was the development of a novel speech-recognition test that combines concepts of different speech-recognition tests to reduce training effects and allows for a large set of speech material. The new test consists of four different words per trial in a meaningful construct with a fixed structure, the so-called phrases. Various free databases were used to select the words and to determine their frequency. Highly frequent nouns were grouped into thematic categories and combined with related adjectives and infinitives. After discarding inappropriate and unnatural combinations, and eliminating duplications of (sub-)phrases, a total number of 772 phrases remained. Subsequently, the phrases were synthesized using a text-to-speech system. The synthesis significantly reduces the effort compared to recordings with a real speaker. After excluding outliers, measured speech-recognition scores for the phrases with 31 normal-hearing participants at fixed signal-to-noise ratios (SNR) revealed speech-recognition thresholds (SRT) for each phrase varying up to 4 dB. The median SRT was −9.1 dB SNR and thus comparable to existing sentence tests. The psychometric function's slope of 15 percentage points per dB is also comparable and enables efficient use in audiology. Summarizing, the principle of creating speech material in a modular system has many potential applications. [ABSTRACT FROM AUTHOR]

This work presents a lightweight phoneme recognition model using object detection techniques. This model is mainly proposed to run on devices with low processing power, such as tablets and mobile phones. The use of the combination of hardware network architecture research complemented by the NetAdapt algorithm has led to the use of a simpler and lighter network architecture called MobileNet. The MobileNetV3 convolutional network architecture was combined with the Single-Shot Detection. The databases used in model training were TIMIT and LibriSpeech, both have spoken audios in English. To generate a graphical representation using the audiobases, for each audio, its spectrogram was calculated on the Mel scale. To train the algorithm of phoneme location detection, the temporal position of the occurrence of each phoneme in respective spectrogram is used. Additionally, it was necessary to increase the training dataset, in order to provide improvement in the generalization of the model. Therefore, the two databases were joined and data augmentation techniques were applied to audios. The main idea was to achieve learning using a lightweight architecture that can be used on devices with low processing power, such as tablets and mobile phones. Thus, this research used the MobileNet-Large architecture, which obtained an accuracy of 0.72 mAP@0.5IOU. For comparison, the MobileNet-Small architecture was also used, which obtained an accuracy of 0.63 mAP@0.5IOU. [ABSTRACT FROM AUTHOR]

In recent years, embedded system technologies and products for sensor networks and wearable devices used for monitoring people's activities and health have become the focus of the global IT industry. In order to enhance the speech recognition capabilities of wearable devices, this article discusses the implementation of audio positioning and enhancement in embedded systems using embedded algorithms for direction detection and mixed source separation. The two algorithms are implemented using different embedded systems: direction detection developed using TI TMS320C6713 DSK and mixed source separation developed using Raspberry Pi 2. For mixed source separation, in the first experiment, the average signal-to-interference ratio (SIR) at 1 m and 2 m distances was 16.72 and 15.76, respectively. In the second experiment, when evaluated using speech recognition, the algorithm improved speech recognition accuracy to 95%. [ABSTRACT FROM AUTHOR]

Speech comprehension can be challenging due to multiple factors, causing inconvenience for both the speaker and the listener. In such situations, using a humanoid robot, Pepper, can be beneficial as it can display the corresponding text on its screen. However, prior to that, it is essential to carefully assess the accuracy of the audio recordings captured by Pepper. Therefore, in this study, an experiment is conducted with eight participants with the primary objective of examining Pepper's speech recognition system with the help of audio features such as Mel-Frequency Cepstral Coefficients, spectral centroid, spectral flatness, the Zero-Crossing Rate, pitch, and energy. Furthermore, the K-means algorithm was employed to create clusters based on these features with the aim of selecting the most suitable cluster with the help of the speech-to-text conversion tool Whisper. The selection of the best cluster is accomplished by finding the maximum accuracy data points lying in a cluster. A criterion of discarding data points with values of WER above 0.3 is imposed to achieve this. The findings of this study suggest that a distance of up to one meter from the humanoid robot Pepper is suitable for capturing the best speech recordings. In contrast, age and gender do not influence the accuracy of recorded speech. The proposed system will provide a significant strength in settings where subtitles are required to improve the comprehension of spoken statements. [ABSTRACT FROM AUTHOR]

Introduction In clinical practice, patients with the same degree and configuration of hearing loss, or even with normal audiometric thresholds, present substantially different performances in terms of speech perception. This probably happens because other factors, in addition to auditory sensitivity, interfere with speech perception. Thus, studies are needed to investigate the performance of listeners in unfavorable listening conditions to identify the processes that interfere in the speech perception of these subjects. Objective To verify the influence of age, temporal processing, and working memory on speech recognition in noise. Methods Thirty-eight adult and elderly individuals with normal hearing thresholds participated in the study. Participants were divided into two groups: The adult group (G1), composed of 10 individuals aged 21 to 33 years, and the elderly group (G2), with 28 participants aged 60 to 81 years. They underwent audiological assessment with the Portuguese Sentence List Test, Gaps-in-Noise test, Digit Span Memory test, Running Span Task, Corsi Block-Tapping test, and Visual Pattern test. Results The Running Span Task score proved to be a statistically significant predictor of the listening-in-noise variable. This result showed that the difference in performance between groups G1 and G2 in relation to listening in noise is due not only to aging, but also to changes in working memory. Conclusion The study showed that working memory is a predictor of listening performance in noise in individuals with normal hearing, and that this task can provide important information for investigation in individuals who have difficulty hearing in unfavorable environments. [ABSTRACT FROM AUTHOR]

Background and Aim: The output Signal-to-Noise Ratio (SNR) is one of the essential factors in hearing aid benefits. There is limited evidence regarding SNR improvement by the Channel-Free (CFHA) and Multi-Channel Hearing Aid (MCHA) and the speech understanding in noise through them. This study aimed to investigate the extent to which output SNR was modified by CFHA and MCHA processing and the variation in aided speech recognition abilities with a change in output SNR. Methods: Thirty-six participants aged 50–65 years were included. A chosen CFHA and MCHA were used to obtain the output SNR and sentence recognition in noise in four different processing algorithms (linear, linear+noise reduction, WDRC, WDRC+noise reduction). Hagerman’s phase inversion technique was used to measure the attenuation of noise and, in turn, to obtain the output SNR of the hearing aid. Results: In all hearing aid processing algorithms among those with normal hearing and people with hearing loss, the output of CFHA revealed higher attenuation values than that of MCHA. There was a significant effect of the hearing aids and processing algorithms in both normal and individuals with hearing impairment on the mean SNR. Further, multiple linear regression analysis results showed that whether the hearing is channel-free or multichannel significantly predicted speech recognition scores, while output SNR and processing algorithms did not. Conclusion: The signal processing algorithms in CFHA had greater noise attenuation values, better output SNR, and speech recognition scores, showing an advantage over the modern MCHA among individuals with hearing impairment. [ABSTRACT FROM AUTHOR]

Featured Application: The proposed system can automatically generate conference/meeting minutes with labeled speakers and produce keyword spotting. So, the proposed system reduces the heavy task of recording the meeting minutes and improves concentration during meetings. Producing conference/meeting minutes requires a person to simultaneously identify a speaker and the speaking content during the course of the meeting. This recording process is a heavy task. Reducing the workload for meeting minutes is an essential task for most people. In addition, providing conference/meeting highlights in real time is helpful to the meeting process. In this study, we aim to implement an automatic meeting minutes generation system (AMMGS) for recording conference/meeting minutes. A speech recognizer transforms speech signals to obtain the conference/meeting text. Accordingly, the proposed AMMGS can reduce the effort in recording the minutes. All meeting members can concentrate on the meeting; taking minutes is unnecessary. The AMMGS includes speaker identification for Mandarin Chinese speakers, keyword spotting, and speech recognition. Transferring learning on YAMNet lets the network identify specified speakers. So, the proposed AMMGS can automatically generate conference/meeting minutes with labeled speakers. Furthermore, the AMMGS applies the Jieba segmentation tool for keyword spotting. The system detects the frequency of words' occurrence. Keywords are determined from the highly segmented words. These keywords help an attendant to stay with the agenda. The experimental results reveal that the proposed AMMGS can accurately identify speakers and recognize speech. Accordingly, the AMMGS can generate conference/meeting minutes while the keywords are spotted effectively. [ABSTRACT FROM AUTHOR]

The teaching of Chinese as a second language has become increasingly crucial for promoting cross-cultural exchange and mutual learning worldwide. However, traditional approaches to international Chinese language teaching have limitations that hinder their effectiveness, such as outdated teaching materials, lack of qualified instructors, and limited access to learning facilities. To overcome these challenges, it is imperative to develop intelligent and visually engaging methods for teaching international Chinese language learners. In this article, we propose leveraging speech recognition technology within artificial intelligence to create an oral assistance platform that provides visualized pinyin-formatted feedback to learners. Additionally, this system can identify accent errors and provide vocational skills training to improve learners' communication abilities. To achieve this, we propose the Attention-Connectionist Temporal Classification (CTC) model, which utilizes a specific temporal convolutional neural network to capture the location information necessary for accurate speech recognition. Our experimental results demonstrate that this model outperforms similar approaches, with significant reductions in error rates for both validation and test sets, compared with the original Attention model, Claim, Evidence, Reasoning (CER) is reduced by 0.67%. Overall, our proposed approach has significant potential for enhancing the efficiency and effectiveness of vocational skills training for international Chinese language learners. [ABSTRACT FROM AUTHOR]

Existing end-to-end speech recognition methods typically employ hybrid decoders based on CTC and Transformer. However, the issue of error accumulation in these hybrid decoders hinders further improvements in accuracy. Additionally, most existing models are built upon Transformer architecture, which tends to be complex and unfriendly to small datasets. Hence, we propose a Nonlinear Regularization Decoding Method for Speech Recognition. Firstly, we introduce the nonlinear Transformer decoder, breaking away from traditional left-to-right or right-to-left decoding orders and enabling associations between any characters, mitigating the limitations of Transformer architectures on small datasets. Secondly, we propose a novel regularization attention module to optimize the attention score matrix, reducing the impact of early errors on later outputs. Finally, we introduce the tiny model to address the challenge of overly large model parameters. The experimental results indicate that our model demonstrates good performance. Compared to the baseline, our model achieves recognition improvements of 0.12%, 0.54%, 0.51%, and 1.2% on the Aishell1, Primewords, Free ST Chinese Corpus, and Common Voice 16.1 datasets of Uyghur, respectively. [ABSTRACT FROM AUTHOR]

Multilingual speakers can find speech recognition in everyday environments like restaurants and open-plan offices particularly challenging. In a world where speaking multiple languages is increasingly common, effective clinical and educational interventions will require a better understanding of how factors like multilingual contexts and listeners' language proficiency interact with adverse listening environments. For example, word and phrase recognition is facilitated when competing voices speak different languages. Is this due to a "release from masking" from lower-level acoustic differences between languages and talkers, or higher-level cognitive and linguistic factors? To address this question, we created a "one-man bilingual cocktail party" selective attention task using English and Mandarin speech from one bilingual talker to reduce low-level acoustic cues. In Experiment 1, 58 listeners more accurately recognized English targets when distracting speech was Mandarin compared to English. Bilingual Mandarin–English listeners experienced significantly more interference and intrusions from the Mandarin distractor than did English listeners, exacerbated by challenging target-to-masker ratios. In Experiment 2, 29 Mandarin–English bilingual listeners exhibited linguistic release from masking in both languages. Bilinguals experienced greater release from masking when attending to English, confirming an influence of linguistic knowledge on the "cocktail party" paradigm that is separate from primarily energetic masking effects. Effects of higher-order language processing and expertise emerge only in the most demanding target-to-masker contexts. The "one-man bilingual cocktail party" establishes a useful tool for future investigations and characterization of communication challenges in the large and growing worldwide community of Mandarin–English bilinguals. [ABSTRACT FROM AUTHOR]

Most of the adversarial attacks against speech recognition systems focus on specific adversarial perturbations, which are generated by adversaries for each normal example to achieve the attack. Universal adversarial perturbations (UAPs), which are independent of the examples, have recently received wide attention for their enhanced real-time applicability and expanded threat range. However, most of the UAP research concentrates on the image domain, and less on speech. In this paper, we propose a staged perturbation generation method that constructs CommanderUAP, which achieves a high success rate of universal adversarial attack against speech recognition models. Moreover, we apply some methods from model training to improve the generalization in attack and we control the imperceptibility of the perturbation in both time and frequency domains. In specific scenarios, CommanderUAP can also transfer attack some commercial speech recognition APIs. [ABSTRACT FROM AUTHOR]

Introduction: Various types of noise have been used with speech material to assess speech perception in noise (SPIN) abilities. The literature suggests that speech identification varies with different types of background noise, and it has been reported that the target native language and the language of the babbling influence performance. Such efforts in an Indian context have not yet been reported. The aim of the study is to evaluate the speech perception in noise performance of Malayalam-speaking young adults with normal hearing using three different background noises. Material and methods: A repeated measure research design were adopted with a random sampling method. 30 native Malayalam speakers with normal hearing between the ages of 18 and 25 participated in the study. A standardized sentence list in Malayalam was used as the speech stimulus. Nine lists were chosen and randomly divided so that there were three lists to each background noise. Noises were speech spectrumshaped noise, non-native language multi-talker babble (Kannada), and native language multi-talker babble (Malayalam). Each successfully repeated keyword received a '1' and each incorrectly repeated word received a '0.' Because each sentence had four important words, each collection of 10 sentences scored a maximum of 40. The percentage of correct answers was determined and further analyzed. Results: Scores were significantly different in all three different background noises across different SNRs. The highest scores were obtained at +5 dB SNR and the poorest scores at --5 dB SNR. Among the three different background noises, native multi-talker babble (Malayalam) yielded better scores than non-native multi-talker babble (Kannada), followed last by speech spectrum-shaped noise. Conclusions: The findings of the current study may be attributed to the increased efficacy of speech spectrum noise due to its energetic masking characteristics and the similarity between the two languages in terms of its origin and acoustic-phonetic properties. [ABSTRACT FROM AUTHOR]

Artificial intelligence (AI) telephone is reliable for the follow‐up and management of hypertensives. It takes less time and is equivalent to manual follow‐up to a high degree. We conducted a reliability study to evaluate the efficiency of AI telephone follow‐up in the management of hypertension. During May 18 and June 30, 2020, 350 hypertensives managed by the Pengpu Community Health Service Center in Shanghai were recruited for follow‐up, once by AI and once by a human. The second follow‐up was conducted within 3–7 days (mean 5.5 days). The mean length time of two calls were compared by paired t‐test, and Cohen's Kappa coefficient was used to evaluate the reliability of the results between the two follow‐up visits. The mean length time of AI calls was shorter (4.15 min) than that of manual calls (5.24 min, P <.001). The answers related to the symptoms showed moderate to substantial consistency (κ:.465–.624, P <.001), and those related to the complications showed fair consistency (κ:.349, P <.001). In terms of lifestyle, the answer related to smoking showed a very high consistency (κ:.915, P <.001), while those addressing salt consumption, alcohol consumption, and exercise showed moderate to substantial consistency (κ:.402–.645, P <.001). There was moderate consistency in regular usage of medication (κ:.484, P <.001). [ABSTRACT FROM AUTHOR]

ThisSpecial Issue presents the latest advances in research and novel applications of speech and language technologies based on the works presented at the sixth edition of the IberSPEECH conference held in Granada in 2022, paying special attention to those focused on Iberian languages. IberSPEECH is the international conference of the Special Interest Group on Iberian Languages (SIG-IL) of the International Speech Communication Association (ISCA) and the Spanish Thematic Network on Speech Technologies (Red Temática en Tecnologías del Habla, or RTTH for short). Several researchers were invited to extend the contributions presented at IberSPEECH2022 due to their interest and quality. As a result, the Special Issue is composed of 11 papers that cover different research topics related to speech perception, speech analysis and enhancement, speaker verification and identification, speech production and synthesis, natural language processing, together with several applications and evaluation challenges. [ABSTRACT FROM AUTHOR]

This study investigates the utilization of the k-nearest-neighbor algorithm within the framework of machine learning for speech recognition applications. The AudioMNIST dataset is used for performing the evaluations in which the model predicts the spoken digit, namely from 0 to 9. Two different training-to-test percentage splits of the dataset are used, 70%-30% and 80%- 20%, while the k parameter ranges from 1 to 12. To better adapt the prediction model, the Melfrequency cepstrum coefficients are extracted from each audio sample, and the 13 filters are averaged over 25 ms frame windows with 10 ms frame overlap. In both training-to-test configurations the value for the k parameter that obtained the highest accuracy (> 95%) is k=5, while the easiest to predict digits was “7”. These findings underscore the efficacy of k-nearest-neighbor in speech recognition tasks and highlight the importance of parameter selection and feature extraction techniques in optimizing model performance. Further exploration of kNN's applicability in diverse speech recognition contexts holds promise for advancing the field's understanding and practical implementations. [ABSTRACT FROM AUTHOR]

In environments where silent communication is essential, such as libraries and conference rooms, the need for a discreet means of interaction is paramount. Here, we present a single-electrode, contact-separated triboelectric nanogenerator (CS-TENG) characterized by robust high-frequency sensing capabilities and long-term stability. Integrating this TENG onto the inner surface of a mask allows for the capture of conversational speech signals through airflow vibrations, generating a comprehensive dataset. Employing advanced signal processing techniques, including short-time Fourier transform (STFT), Mel-frequency cepstral coefficients (MFCC), and deep learning neural networks, facilitates the accurate identification of speaker content and verification of their identity. The accuracy rates for each category of vocabulary and identity recognition exceed 92% and 90%, respectively. This system represents a pivotal advancement in facilitating secure and efficient unobtrusive communication in quiet settings, with promising implications for smart home applications, virtual assistant technology, and potential deployment in security and confidentiality-sensitive contexts. [ABSTRACT FROM AUTHOR]

The classification of musical emotion is a challenging problem due to the subjectivity of human perception. In order to analyse the emotion contained in music from the perspective of product supply chain management, the psychological model and characteristics of music emotion are introduced. Additionally, the current product supply chain and the composition of the two contracts are analysed based on the characteristics of online music products. Finally, a speech emotion recognition model based on a Fuzzy Cognitive Map (FCM) is proposed. Two corpora containing five emotions (sadness, anger, happiness, surprise, and boredom) are selected to test the performance and characteristics of the network. In order to evaluate the system performance, different acoustic features are extracted, including prosodic features, formant features, marginal spectral features, and Mel‐Frequency Cepstral Coefficients (MFCC). The results show that the average recognition rate of MFCC features on the Berlin emotional speech database is 70.36%, which is 0.75% higher than the second‐highest edge spectrum features and achieves better results. Meanwhile, on this database, the Musical Emotion Recognition System based on FCM (MERS‐FCM) is improved by 20.36%, 7.34%, and 4.12% compared with Back Propagation, K‐nearest neighbour, and support vector machine, respectively, which shows that MERS‐FCM has better performance. The research of the music emotion recognition system based on FCM can provide more reference paths for the use and development of music emotion information. [ABSTRACT FROM AUTHOR]

Speech recognition becomes increasingly important in the modern society, especially for human–machine interactions, but its deployment is still severely thwarted by the struggle of machines to recognize voiced commands in challenging real‐life settings: oftentimes, ambient noise drowns the acoustic sound signals, and walls, face masks or other obstacles hide the mouth motion from optical sensors. To address these formidable challenges, an experimental prototype of a microwave speech recognizer empowered by programmable metasurface is presented here that can remotely recognize human voice commands and speaker identities even in noisy environments and if the speaker's mouth is hidden behind a wall or face mask. The programmable metasurface is the pivotal hardware ingredient of the system because its large aperture and huge number of degrees of freedom allows the system to perform a complex sequence of sensing tasks, orchestrated by artificial‐intelligence tools. Relying solely on microwave data, the system avoids visual privacy infringements. The developed microwave speech recognizer can enable privacy‐respecting voice‐commanded human–machine interactions is experimentally demonstrated in many important but to‐date inaccessible application scenarios. The presented strategy will unlock new possibilities and have expectations for future smart homes, ambient‐assisted health monitoring, as well as intelligent surveillance and security. [ABSTRACT FROM AUTHOR]

In this study, from the perspective of English speech feature parameters, two feature parameters, the mel-frequency cepstral coefficient (MFCC) and filter bank (Fbank), were selected to identify English speech. The algorithms used for recognition employed the classical back-propagation neural network (BPNN), recurrent neural network (RNN), and long short-term memory (LSTM) that were obtained by improving RNN. The three recognition algorithms were compared in the experiments, and the effects of the two feature parameters on the performance of the recognition algorithms were also compared. The LSTM model had the best identification performance among the three neural networks under different experimental environments; the neural network model using the MFCC feature parameter outperformed the neural network using the Fbank feature parameter; the LSTM model had the highest correct rate and the highest speed, while the RNN model ranked second, and the BPNN model ranked worst. The results confirm that the application of the LSTM model in combination with MFCC feature parameter extraction to English speech recognition can achieve higher speech recognition accuracy compared to other neural networks. [ABSTRACT FROM AUTHOR]

The abundance of openly available audio data in English enables pretraining of automatic speech recognition (ASR) systems on hundreds of thousands to millions of hours of recorded speech. As a result speech recognition systems are approaching human level robustness in English. Other languages' performance in multilingual speech recognition systems tend to stand in proportion to the amount of data included from the language – or the language family – in question. For low to mid resource languages with fewer speakers, the amount of openly available data may be limited, and as a consequence these languages tend to be underrepresented in large scale efforts to train multilingual speech recognition systems. The National Library of Sweden hosts large collections of audio recordings. These resources allow us to potentially bridge some of the existing speech recognition performance gaps between Swedish and higher resource languages. We believe Swedish speech recognition can be further improved upon by scaling up the amount of training data. We have constructed an inclusive and transparent speech corpus with emphasis on all variations of spoken Swedish that we will use to train speech-to-text models for Swedish. [ABSTRACT FROM AUTHOR]

Since the advent of ChatGPT in November 2022, there has been a growing interest and widespread speculation on how Artificial Intelligence (AI), more specifically generative AI, has the potential to revolutionize research and applications across disciplines, with real-world implications. Applied linguistics researchers and practitioners have long adapted to the use of technology in language learning and teaching, where AI already plays a role in the form of Natural Language Processing (NLP), Machine Learning (ML), and other related technologies. This Expositions article introduces generative AI, explains how it works and what distinguishes it from other AI technologies, and discusses its growing influence in the applications relevant to applied linguists. The article concludes with some guidance on how to navigate the generative AI space as an applied linguist while acknowledging the current limitations, including how to use generative AI in research and practice. [ABSTRACT FROM AUTHOR]

During a conversation, speakers in multilingual societies frequently switch between two or more spoken languages. A linguistic action known as "code-switching" particularly alters or merges two or more languages. The development of software or tools for detecting code-switching has received very little attention. This paper proposes a Deep Learning based methods for detecting code-switched English-Marathi data. These suggested methods can be applied to various applications, including phone call merging, Intelligent AI assistants, Intelligent travelling systems to assist travellers in navigation and reservations, call centres to handle customer service issues, etc. To create a system for code switch detection, our study demonstrates a detailed analysis of extracting several audio features such as the Mel-Spectrogram, Mel-frequency Cepstral Coefficient (MFCC), and Perceptual Linear Predictive coefficients (PLP). Our team's EnglishMarathi code-switched dataset served as the testing ground for our methodologies. Our model's accuracy was 92.99%, with 40 MFCC coefficients having energy coefficient serving as the zeroth coefficient. [ABSTRACT FROM AUTHOR]