Search

AbstractObjectiveDesignStudy sampleResultsConclusionsExploring applications of automatic speech recognition and text-to-speech technologies in hearing assessment and evaluations of hearing aids.Review protocol was registered at the INPLASY database and was performed following the PRISMA scoping review guidelines. A search in ten databases was conducted in January 2023 and updated in June 2024.Studies that used automatic speech recognition or text-to-speech to assess measures of hearing ability (e.g. speech reception threshold), or to configure hearing aids were retrieved. Of the 2942 records found, 28 met the inclusion criteria.The results indicated that text-to-speech could effectively replace recorded stimuli in speech intelligibility tests, requiring less effort for experimenters, without negatively impacting outcomes (n = 5). Automatic speech recognition captured verbal responses accurately, allowing for reliable speech reception threshold measurements without human supervision (n = 7). Moreover, automatic speech recognition was employed to simulate participants’ hearing, with high correlations between simulated and empirical data (n = 14). Finally, automatic speech recognition was used to optimise hearing aid configurations, leading to higher speech intelligibility for wearers compared to the original configuration (n = 3).There is the potential for automatic speech recognition and text-to-speech systems to enhance accessibility of, and efficiency in, hearing assessments, offering unsupervised testing options, and facilitating hearing aid personalisation. [ABSTRACT FROM AUTHOR]

Enhancing the naturalness and rhythmicity of generated audio in end-to-end speech synthesis is crucial. The current state-of-the-art (SOTA) model, VITS, utilizes a conditional variational autoencoder architecture. However, it faces challenges, such as limited robustness, due to training solely on text and spectrum data from the training set. Particularly, the posterior encoder struggles with mid- and high-frequency feature extraction, impacting waveform reconstruction. Existing efforts mainly focus on prior encoder enhancements or alignment algorithms, neglecting improvements to spectrum feature extraction. In response, we propose BERTIVITS, a novel model integrating BERT into VITS. Our model features a redesigned posterior encoder with residual connections and utilizes pre-trained models to enhance spectrum feature extraction. Compared to VITS, BERTIVITS shows significant subjective MOS score improvements (0.16 in English, 0.36 in Chinese) and objective Mel-Cepstral coefficient reductions (0.52 in English, 0.49 in Chinese). BERTIVITS is tailored for single-speaker scenarios, improving speech synthesis technology for applications like post-class tutoring or telephone customer service. [ABSTRACT FROM AUTHOR]

Recent developments on artificial intelligence (AI) and machine learning (ML) techniques are expected to have significant impact on public health in several ways. Indeed, modern AI/ML methods have been applied on multiple occasions on topics ranging from drug discovery and disease diagnostics to personalized medicine, medical imaging, and healthcare operations. While such developments may improve several quality-of-life aspects (such as access to health services and education), it is important considering that some individuals may face more challenges, particularly in extreme or emergency situations. In this work, we focus on utilizing AI/ML components to support scenarios when visual impairment or other limitations hinder the ability to interpret the world in this way. Specifically, we discuss the potential and the feasibility of automatically transferring key visual information into audio communication, in different languages and in real-time—a setting which we name ‘audible reality’ (AuRa). We provide a short guide to practical options currently available for implementing similar solutions and summarize key aspects for evaluating their scope. Finally, we discuss diverse settings and functionalities that AuRA applications could have in terms of broader impact, from a social and public health context, and invite the community to further such digital solutions and perspectives soon.

Abstract In the era of advanced text-to-speech (TTS) systems capable of generating high-fidelity, human-like speech by referring a reference speech, voice cloning (VC), or zero-shot TTS (ZS-TTS), stands out as an important subtask. A primary challenge in VC is maintaining speech quality and speaker similarity with limited reference data for a specific speaker. However, existing VC systems often rely on naive combinations of embedded speaker vectors for speaker control, which compromises the capture of speaking style, voice print, and semantic accuracy. To overcome this, we introduce the Two-branch Speaker Control Module (TSCM), a novel and highly adaptable voice cloning module designed to precisely processing speaker or style control for a target speaker. Our method uses an advanced fusion of local-level features from a Gated Convolutional Network (GCN) and utterance-level features from a gated recurrent unit (GRU) to enhance speaker control. We demonstrate the effectiveness of TSCM by integrating it into advanced TTS systems like FastSpeech 2 and VITS architectures, significantly optimizing their performance. Experimental results show that TSCM enables accurate voice cloning for a target speaker with minimal data through both zero-shot or few-shot fine-tuning of pretrained TTS models. Furthermore, our TSCM-based VITS (TSCM-VITS) showcases superior performance in zero-shot scenarios compared to existing state-of-the-art VC systems, even with basic dataset configurations. Our method’s superiority is validated through comprehensive subjective and objective evaluations. A demonstration of our system is available at https://great-research.github.io/tsct-tts-demo/ , providing practical insights into its application and effectiveness.

In the era of advanced text-to-speech (TTS) systems capable of generating high-fidelity, human-like speech by referring a reference speech, voice cloning (VC), or zero-shot TTS (ZS-TTS), stands out as an important subtask. A primary challenge in VC is maintaining speech quality and speaker similarity with limited reference data for a specific speaker. However, existing VC systems often rely on naive combinations of embedded speaker vectors for speaker control, which compromises the capture of speaking style, voice print, and semantic accuracy. To overcome this, we introduce the Two-branch Speaker Control Module (TSCM), a novel and highly adaptable voice cloning module designed to precisely processing speaker or style control for a target speaker. Our method uses an advanced fusion of local-level features from a Gated Convolutional Network (GCN) and utterance-level features from a gated recurrent unit (GRU) to enhance speaker control. We demonstrate the effectiveness of TSCM by integrating it into advanced TTS systems like FastSpeech 2 and VITS architectures, significantly optimizing their performance. Experimental results show that TSCM enables accurate voice cloning for a target speaker with minimal data through both zero-shot or few-shot fine-tuning of pretrained TTS models. Furthermore, our TSCM-based VITS (TSCM-VITS) showcases superior performance in zero-shot scenarios compared to existing state-of-the-art VC systems, even with basic dataset configurations. Our method's superiority is validated through comprehensive subjective and objective evaluations. A demonstration of our system is available at https://great-research.github.io/tsct-tts-demo/, providing practical insights into its application and effectiveness. [ABSTRACT FROM AUTHOR]

In this paper, we build neural-network model-based automatic speech recognition (ASR) systems incrementally for performance improvement. First, we add an adversarial text discriminator module to train the speech recognition model to correct typos in recognition results. Experiments show that the character error rate (CER) and word error rate (WER) of the ASR system achieved 12.3% and 31.4%. Second, we insert a pre-trained speech synthesis (text-to-speech, TTS) module to the ASR model. When we exploit a pre-trained TTS in ASR training, the CER and WER are reduced from 12.6% and 31.7% to 10.8% and 24.4%, demonstrating that pre-trained TTS can improve ASR. Finally, we include both pre-trained TTS and text discriminator in ASR training. The performance of this ASR system is further improved, achieving the CER and WER of 9.9% and 22.7% respectively. On Formosa Speech Recognition Challenge task using Taibun Hàn-jī transcription, the proposed method also achieves better CER than a system based on hybrid DNN-HMM chain model. [ABSTRACT FROM AUTHOR]

Automatically describing the content of a picture with meaningful and contextually appropriate textual descriptions is a difficult challenge at the interface of computer vision and natural language processing. This review paper provides a thorough summary of recent developments in picture caption generating methods, including both conventional strategies and cutting-edge deep learning-based solutions. We examine the changes in datasets, evaluation criteria, and model designs that have influenced this field's advancement. In order to improve the standard and variety of generated captions, we also integrate visual attention processes, transformer-based models, and reinforcement learning techniques. The focus of this study is on the algorithmic overlap between text to speech and visual. [ABSTRACT FROM AUTHOR]

In 2023, global data revealed that approximately 2.2 billion people are affected by some form of vision impairment. By addressing the specific challenges faced by individuals who are blind and visually impaired, this research paper introduces a novel system that integrates Google Speech input and text-to-speech technology. This innovative approach enables users to perform a variety of tasks, such as reading, detecting weather conditions, and finding locations, using simple voice commands. The user-friendly application is designed to significantly improve social interaction and daily activities for individuals who are visually impaired, underscoring the critical role of accessibility in technological advancements. This research effectively demonstrates the potential of this system to enhance the quality of life for those with visual impairments. [ABSTRACT FROM AUTHOR]

We propose a two-stage emotion-controllable text-to-speech (TTS) model that can increase the diversity of intra-emotion variation and also preserve inter-emotion controllability in synthesized speech. Conventional emotion-controllable TTS models increase the diversity of intra-emotion variation by controlling fine-grained emotion strengths; however, such models cannot control various prosodic factors (e.g., pitch). While other methods directly condition TTS models on intuitive prosodic factors, they cannot control emotions. Our proposed two-stage emotion-controllable TTS model extends the Tacotron2 model with a speech emotion recognizer (SER) and a prosodic factor generator (PFG) to solve this problem. In the first stage, we condition our model on emotion soft labels predicted by the SER model to enable inter-emotion controllability. In the second stage, we fine-condition our model on utterance-level prosodic factors and word-level prominence generated by the PFG model from emotion soft labels, which provides intra-emotion diversity. Due to this two-stage control design, we can increase intra-emotion diversity at both the utterance and word levels, and also preserve inter-emotion controllability. The experiments achieved 1) 51% emotion-distinguishable accuracy on average when conditioning on soft labels of three emotions, 2) average linear controllability scores of 0.95 when fine-conditioning on prosodic factors and prominence, respectively, and 3) comparable audio quality to conventional models. [ABSTRACT FROM AUTHOR]

The main purpose of this paper is to present a chatbot developed for international students. It provides guidance on a range of challenges, including safety, culture shock, housing, language, and academic notes. For this purpose, a dataset of 72 question-and-answer intents was developed from scratch, designed to assist students. The training phase uses the dataset to create the neural network model. The chatbot makes use of established rules and techniques for natural language processing to predict the appropriate responses to user’s demand. Two Python-based versions of the chatbot are developed: one with a user interface, designed using a Flask server connected to a front-end section, and one with a compiler-based method that can handle speech input and output. Both versions achieved high accuracy during the training phase, suggesting good forecasting for the purpose upon which the chatbot was designed. [ABSTRACT FROM AUTHOR]

The eye is an essential sensory organ that allows us to perceive our surroundings at a glance. Losing this sense can result in numerous challenges in daily life. However, society is designed for the majority, which can create even more difficulties for visually impaired individuals. Therefore, empowering them and promoting self-reliance are crucial. To address this need, we propose a new Android application called “The Eye” that utilizes Machine Learning (ML)-based object detection techniques to recognize objects in real-time using a smartphone camera or a camera attached to a stick. The article proposed an improved YOLOv5l algorithm to improve object detection in visual applications. YOLOv5l has a larger model size and captures more complex features and details, leading to enhanced object detection accuracy compared to smaller variants like YOLOv5s and YOLOv5m. The primary enhancement in the improved YOLOv5l algorithm is integrating L1 and L2 regularization techniques. These techniques prevent overfitting and improve generalization by adding a regularization term to the loss function during training. Our approach combines image processing and text-to-speech conversion modules to produce reliable results. The Android text-to-speech module is then used to convert the object recognition results into an audio output. According to the experimental results, the improved YOLOv5l has higher detection accuracy than the original YOLOv5 and can detect small, multiple, and overlapped targets with higher accuracy. This study contributes to the advancement of technology to help visually impaired individuals become more self-sufficient and confident. Doi: 10.28991/ESJ-2023-07-05-011 Full Text: PDF

Enhancing the naturalness and rhythmicity of generated audio in end-to-end speech synthesis is crucial. The current state-of-the-art (SOTA) model, VITS, utilizes a conditional variational autoencoder architecture. However, it faces challenges, such as limited robustness, due to training solely on text and spectrum data from the training set. Particularly, the posterior encoder struggles with mid- and high-frequency feature extraction, impacting waveform reconstruction. Existing efforts mainly focus on prior encoder enhancements or alignment algorithms, neglecting improvements to spectrum feature extraction. In response, we propose BERTIVITS, a novel model integrating BERT into VITS. Our model features a redesigned posterior encoder with residual connections and utilizes pre-trained models to enhance spectrum feature extraction. Compared to VITS, BERTIVITS shows significant subjective MOS score improvements (0.16 in English, 0.36 in Chinese) and objective Mel-Cepstral coefficient reductions (0.52 in English, 0.49 in Chinese). BERTIVITS is tailored for single-speaker scenarios, improving speech synthesis technology for applications like post-class tutoring or telephone customer service.

Text, as one of humanity's most influential innovations, has played an important role in shaping our lives. Reading a text is a difficult task due to several reasons factors, such as luminosity, text orientation, writing style, and very light colors. However, the visually impaired, on the other hand, have difficulty reading a text in all of these situations. In particular, a handwritten text is more difficult to read than a digital text due to the different forms and styles of the handwriting of different writers or, sometimes, of the same writer. Therefore, they would benefit from a device or a system to help them to solve this problem and improve their quality of life. Arabic language recognition and identification is a very difficult task because of diacritics such as consonant score, tashkil, and others. In this context, we propose a recognition and identification system for Arabic Handwritten Texts with Diacritics (AHTD) based on deep learning by using the convolutional neural network. Text images are trained, tested, and validated with our Arabic Handwritten Texts with a Diacritical Dataset (AHT2D). Then, the recognized text is enhanced with augmented reality technology and produced as a 2D image. Finally, the recognized text is converted into an audio output using AR technology. Voice output and visual output are given to the visually impaired user. The experimental results show that the proposed system is robust, with an accuracy rate of 95%. [ABSTRACT FROM AUTHOR]

Based on the research "The text-to-speech resource beyond assistance", it is suggested that reading through voice synthesis applications entails a double relationship with the text: hearing and visual. This article proposes the term audiotextual reading, which is a textual format different from the traditional audiobook and it must be integrated into the category of tertiary orality. Finally, some research findings based on the perceptions of the audiences are considered. The term audioreaders is also suggested in this article. These insights contribute to the understanding of the conditions in which text-to-speech is used as an alternative form of reading experience. [ABSTRACT FROM AUTHOR]

The "Smart Glasses" are made to make it easier for blind persons to read and decipher written English-language content. A blind people find it exceedingly challenging to travel alone, and they run the danger of getting lost and regular sticks won't allow the person to go around independently in public without things growing worse. The objective of our work is to help blind people communicate more easily by developing a smart assistive glasses using artificial intelligence. The function of the glasses is to read out any text picture as audio text, which can then be heard through a headset attached to the spectacles. OpenCV, optical character recognition and efficient and accurate scene text (EAST) detector were used to identify the text in the image; ultrasonic sensor in the glasses is used to calculate the distance to snap a clear picture. The motion sensor directs the blind to the lecture halls, classrooms, and laboratory locations using an radio frequency identification reader. The results shows that combination of optical character recognizer and EAST detector produced a fairly accurate result, demonstrating the potential of the glasses to recognize the text. Currently, the language supported by the glasses is English, and the distance covered is 40 to 150 cm. [ABSTRACT FROM AUTHOR]

In 2023, global data revealed that approximately 2.2 billion people are affected by some form of vision impairment. By addressing the specific challenges faced by individuals who are blind and visually impaired, this research paper introduces a novel system that integrates Google Speech input and text-to-speech technology. This innovative approach enables users to perform a variety of tasks, such as reading, detecting weather conditions, and finding locations, using simple voice commands. The user-friendly application is designed to significantly improve social interaction and daily activities for individuals who are visually impaired, underscoring the critical role of accessibility in technological advancements. This research effectively demonstrates the potential of this system to enhance the quality of life for those with visual impairments.

Individuals with visual impairments frequently confront substantial difficulties in interacting with their environment, a problem that is often exacerbated by the cost and accessibility of existing assistive technologies. This study introduces a prototype for a costeffective and accessible assistive device that employs deep learning techniques for object recognition. The proposed system utilizes the YOLO-V7 model, a deep learning algorithm trained on a comprehensive dataset encompassing various everyday objects, including US dollar denominations. In conjunction with two transfer learning-based cascade models, the system offers detection across 86 object categories. Upon object identification, the name of the item is converted into a Braille-readable format using the Python Braille library. Comprehensive experiments and analyses were undertaken to assess the efficacy of the proposed system. The results corroborate the system's effectiveness in achieving its intended purpose, demonstrating its potential to significantly aid visually impaired individuals in recognizing and interacting with objects in their environment. With a processing and Braille code generation time of 188.5 ms per frame, the model achieved recall, precision, and mAP scores of 0.81, 0.92, and 0.96, respectively. The integration of deep learning technology with high-performance platform boards has facilitated the development of a promising solution to the challenges faced by visually impaired individuals in environmental interaction. Overall, the proposed prototype represents an accessible and cost-effective assistive device, potentially revolutionizing the manner in which visually impaired individuals interact with their surroundings. [ABSTRACT FROM AUTHOR]

This study aims to determine the role of text to speech assistive technology on the reading ability of ADHD students in learning using e-learning. The ability of ADHD students who have impulsive and hyperactive behavior shows that students who are diagnosed will have difficulty focusing on one thing as a result of coordination functions and emotional responses that affect learning abilities, especially in reading. To improve reading skills, the use of assistive text to speech technology is needed by students during e-learning. The purpose of this study was to determine the role of text to speech assistive technology to improve the reading ability of ADHD students in reading spelling of words, with a clear voice and straightforward word emphasis. The method used in this study is Single Subject Research (SSR) using the ABA technique. Instruments to determine the level of reading ability of ADHD students are observation and documentation. The results showed that the role of text to speech assistive technology to improve reading skills in e-learning for ADHD students made progress in spelling words and with clear vowels and straightforward word emphasis. It can be concluded that the role of text to speech assistive technology to improve reading skills in e-learning for ADHD students is highly recommended to be applied in every e-learning lesson. It can be concluded that the role of text to speech assistive technology to improve reading skills in e-learning for ADHD students is highly recommended to be applied in every e-learning lesson.

Detecting and recognizing the objects and generating speech about the objects helps visually impaired in a great way in understanding their surroundings. Required a mechanism to assist the visually impaired person to travel independently with the ability to identify objects in their path, and the ability to generate speech describing the objects detected in the scene. This can be achieved with the help of YOLOv5 image detection model and text to speech converters such as gTTS and pyttsx3 modules in python. The proposed method called Third Eye, giving better accuracy in detection and speech generation to help the visually impaired people Compared to existing approaches. YOLO v5 is trained on custom dataset of 15 objects along with MS COCO 2017 Dataset of 80 objects (95 objects overall). The output labels of the model are transformed to text and later converted to audio format and are presented to the visually impaired, through a speaker. We compared two python libraries for audio conversion, one is pyttsx3, and the other is gTTS. [ABSTRACT FROM AUTHOR]

This case study situates an early text-to-speech computer developed for blind persons, the Kurzweil Reading Machine (KRM), within a broader history of speech synthesis technologies. Though typically no more than a footnote in the technical history of speech synthesis, the KRM was still a powerful symbol of innovation that reveals how disability can be used as a pretext for funding technology development. I argue that various boosters held the KRM up as a symbol of technological solutionism that promised to fully enroll blind people into the US political economy. However, the success of the KRM as a symbol belies its technical flaws, the federal subsidies needed to bring it to fruition, and the structural barriers to its use that were elided by its utopian promise. [ABSTRACT FROM AUTHOR]

Radio broadcasting is a source of information for the public. In delivering information, the broadcaster must read the text of the news before it is transmitted through a radio transmitter. Implementing a text-to-speech conversion can help solve this problem. The purpose of this study is to develop a Broadcasting Service Application by implementing text to speech on a Web-Based Radio Broadcasting service application that can convert data/information in text form into information in the form of voice. This paper does not discuss text-to-speech conversion techniques but emphasizes how to apply text-to-speech conversion techniques in web-based broadcasting service applications, especially news and music data management. This application utilizes web hosting services and Google voice services to convert text data into voice. The method used is Analyzing the application system of Broadcasting Radio services, Application of Text to speech (TTS), which functions to convert text into sound before being transmitted to broadcasting stations and to test the response of the conversion results and data management to the quality of the design. The process carried out in this system, namely retrieval of text data from the news database, converting text data into sound, adjusting the volume of music and sound information, and updating the status of the text data that has been broadcast. Based on application testing carried out with the Black Box Testing Method, all functional applications have been running well and are now successfully operationalized.

Abstract Deep learning techniques are currently being applied in automated text-to-speech (TTS) systems, resulting in significant improvements in performance. However, these methods require large amounts of text-speech paired data for model training, and collecting this data is costly. Therefore, in this paper, we propose a single-speaker TTS system containing both a spectrogram prediction network and a neural vocoder for the target language, using only 30 min of target language text-speech paired data for training. We evaluate three approaches for training the spectrogram prediction models of our TTS system, which produce mel-spectrograms from the input phoneme sequence: (1) cross-lingual transfer learning, (2) data augmentation, and (3) a combination of the previous two methods. In the cross-lingual transfer learning method, we used two high-resource language datasets, English (24 h) and Japanese (10 h). We also used 30 min of target language data for training in all three approaches, and for generating the augmented data used for training in methods 2 and 3. We found that using both cross-lingual transfer learning and augmented data during training resulted in the most natural synthesized target speech output. We also compare single-speaker and multi-speaker training methods, using sequential and simultaneous training, respectively. The multi-speaker models were found to be more effective for constructing a single-speaker, low-resource TTS model. In addition, we trained two Parallel WaveGAN (PWG) neural vocoders, one using 13 h of our augmented data with 30 min of target language data and one using the entire 12 h of the original target language dataset. Our subjective AB preference test indicated that the neural vocoder trained with augmented data achieved almost the same perceived speech quality as the vocoder trained with the entire target language dataset. Overall, we found that our proposed TTS system consisting of a spectrogram prediction network and a PWG neural vocoder was able to achieve reasonable performance using only 30 min of target language training data. We also found that by using 3 h of target language data, for training the model and for generating augmented data, our proposed TTS model was able to achieve performance very similar to that of the baseline model, which was trained with 12 h of target language data.

This multimodal conversation analysis study is part of a larger video ethnographic project that explores the media literacy practices that children develop as they use digital and mobile technologies. The study investigates how Swedish students in grades 3–4 make use of text to speech (TTS) technology as an interactional resource during collaborative writing on iPads in the classroom. The results show that students routinely make use of synthetic voicings to display and claim knowledge about the voiced written units and negotiate writing roles with differing epistemic rights and obligations to assess voicings and practice repair. [ABSTRACT FROM AUTHOR]