Search

Technological advancements and changes in online survey participation pave the way for new data collection methods. Particularly, the increasing smartphone rate in online surveys facilitates a re-consideration of prevailing communication channels to, for instance, naturalize the communication process between researchers and respondents and to collect more in-depth and high-quality data. However, so far, there is a lack of information on whether respondents are willing to undergo a change in communication channels. In this study, I therefore investigate respondents’ willingness to participate in online surveys with a smartphone to have the survey questions read out loud (audio channel) and to give oral answers via voice input (voice channel). For this purpose, I employed two willingness questions – one on audio and one on voice channels – in the probability-based German Internet Panel (N = 4,426). The results reveal that a substantial minority of respondents is willing to participate in online surveys with a smartphone to have the survey questions read out loud and to give oral answers via voice input. They also show that the device used for survey participation and personality traits, such as conscientiousness and extraversion, play a role when it comes to respondents’ willingness.

People with different languages need to be assisted by translator to establish the communication between them. The technology development which exists to fulfill communication needs is digital dictionary as translator tool. The capability of digital dictionary to translate the languages yet has a weakness in putting the input. Through this research, Optical Character Recognition using Tesseract library and Voice Recognition technologies using Google Speech-To-Text are used to replace the previous input system. Based on the implementation and testing, the OCR and Voice Recognition have been successfully recognizing the text and voice input with the amount of similarity of 92,72% for OCR and 95,46% for Voice Recognition. The result of the implementation is expected to help a group of people with different language to communicate easily.

Voice interactions and voice messages on mobile phones are rapidly growing in popularity. However, the user experience of these services is still worse than desired in noisy environments, especially in multi-talker scenarios, where the phone can only provide low-quality voice recordings. Speech enhancement using only audio as the input remains a grand challenge in these scenarios. In this paper, we handle this with the help of the emerging acoustic sensing technology. The key insight is that the inaudible acoustic signals emitted by speakers of phones can capture the subtle lip movements when people speak. Instead of enabling lip reading for the classification of limited voice commands, we further unlock the potential of acoustic sensing and leverage the captured lip information to improve the voice recording quality. We propose WaveVoice, a joint audio-sensory deep learning method for end-to-end speech enhancement on mobile phones. The model of WaveVoice is structured as an encoder-decoder network, in which audio and acoustic sensing data are processed through two individual CNN branches, respectively, and then fused into a joint network to generate enhanced speech. In addition, to improve the performance on new users, a self-supervised learning methodology is developed to adapt the model to extract speaker-specific features. We construct a dataset to train and evaluate WaveVoice. We also perform online tests under various noisy conditions to show the applicability of our system in real-world scenarios. Experimental results show that WaveVoice can effectively reconstruct the target clean speech from the noisy audio signals, and yield notably superior performance compared with the audio-only encoder-decoder model and the state-of-the-art speech enhancement methods. Given its promising performance, we believe that WaveVoice has made a substantial contribution to the advancement of mobile voice input.

This study aims to examine the use of E-learning based on LMS for college students with special needs during the Covid-19 pandemic. The method used in this research is a literature study, where the researcher collects relevant data to support the researcher’s ideas. Based on the literature review results, the use of E-Learning based on LMS Edmodo, Schoology, and Learnboost has the same effectiveness despite having different levels of ease of use. Meanwhile, the most frequently used E-Learning based on LMS at the university level is Moodle. In its utilization, E-learning based on LMS has five standard features. So that these five features can be used optimally by students with special needs during the Covid-19 pandemic, things that need to be considered are: (1) particular guidelines for the use of E-Learning based on LMS are available for college students with special needs; (2) audio is available that can be used by college students with visual impairments, (3) sign language is available as an interface so that college students can access it with hearing impairments, (4) available text on lecturer materials in the form of video or audio so that it can be accessed by college students with hearing impairments, (5) there is a virtual keyboard feature available in E-Learning based on LMS which allows college students with physical and motor disabilities to use the mouse to type or do other activities on the computer, (6) the features available in E-Learning based on LMS are designed to accept voice input.

As a natural and convenient interaction modality, voice input has now become indispensable to smart devices (e.g. mobile phones and smart appliances). However, voice input is strongly constrained by surroundings and may raise privacy leakage in public areas. In this paper, we present SoundLip, an end-to-end interaction system enabling users to interact with smart devices via silent voice input. The key insight is to use inaudible acoustic signals to capture the lip movements of users when they issue commands. Previous works have considered lip reading as a naive classification task and thus can only recognize individual words. In contrast, our proposed system enables lip reading at both word and sentence levels, which are more suitable for daily-life use. We exploit the built-in speakers and microphones of smart devices to emit acoustic signals and listen to their reflections, respectively. In order to better abstract representations from multi-frequency and multi-modality acoustic signals, we elaborate a hierarchical convolutional neural network (HCNN) to serve as the front-end as well as recognize individual word commands. Then, for the sentence-level recognition, we exploit a multi-task encoder-decoder network to get around temporal segmentation and output sentences in an end-to-end way. We evaluate SoundLip on 20 individual words and 70 sentences from 12 participants. Our system achieves an accuracy of 91.2% at word-level and a word error rate of 7.1% at sentence-level in both user-independent and environment-independent settings. Given its innovative solution and promising performance, we believe that SoundLip has made a significant contribution to the advancement of silent voice input technology.

We conduct an examination of the preferences of drivers and passengers alike for in-vehicle interactions with a multistudy approach consisting of (1) a targeted literature survey of applications an...

In this digital world integrating smart city concepts, there is a tremendous scope and need for e-governance applications. Now people analyze the opinion of others before purchasing any product, hotel booking, stepping onto restaurants etc. and the respective user share their experience as a feedback towards the service. But there is no e-governance platform to obtain public opinion grievances towards covid19, government new laws, policies etc. With the growing availability and emergence of opinion rich information's, new opportunities and challenges might arise in developing a technology for mining the huge set of public messages, opinions and alert the respective departments to take necessary actions and also nearby ambulances if its related to covid-19. To overcome this pandemic situation a natural language processing based efficient e-governance platform is demandful to detect the corona positive patients and provide transparency on the covid count and also alert the respective health ministry and nearby ambulance based on the user voice inputs. To convert the public voice messages into text, we used Hidden Markov Models (HMMs). To identify respective government department responsible for the respective user voice input, we perform pre-processing, part of speech, unigram, bigram, trigram analysis and fuzzy logic (machine learning technique). After identifying the responsible department, we perform 2 methods, (1) Automatic alert e-mail and message to the government departmental officials and nearby ambulance or covid camp if the user input is related to covis19. (2) Ticketing system for public and government officials monitoring. For experimental results, we used Java based web and mobile application to execute the proposed methodology. Integration of HMM, Fuzzy logic provides promising results.

This demonstration enables participants to explore large data sets via voice interfaces. The focus of the demonstration is on methods generating concise speech descriptions of query results, specified by users via voice input. The technical novelty of the demonstrated system lies in the fact that processing overheads are mostly moved into a pre-processing phase, generating speeches for batches of queries defined via templates. Visitors can access the demo via smart speakers on-site or via their own smart phones. They will be able to customize the generated voice descriptions and to tune voice output methods.

The emergence, development, problems, prospects and statistics of the use of voice assistants in interactive marketing communications have been considered. The use of voice assistants as an interface of interaction with electronic equipment is a new, dynamically developing trend, covering all spheres of human activity. The relevance of the studied problem lies in the fact that the integration of advertising in voice assistants in the near future will affect all types of business in one way to another and become an integral part of a successful advertising campaign: these are the most effective interactive marketing communications of the future. The author’s recommendations on the use of voice assistants in interactive marketing communications have been given.

Recent advances in speech technologies have produced new tools that can be used to improve the performance and flexibility of speaker recognition While there are few degrees of freedom or alternative methods when using fingerprint or iris identification techniques, speech offers much more flexibility and different levels for performing recognition: the system can force the user to speak in a particular manner, different for each attempt to enter. Also with voice input the system has other degrees of freedom, such as the use of knowledge/codes that only the user knows, or dialectical/semantical traits that are difficult to forge. This paper offers and overview of the state of the art in speaker recognition, with special emphasis on the pros and contras, and the current research lines. The current research lines include improved classification systems, and the use of high level information by means of probabilistic grammars. In conclusion, speaker recognition is far away from being a technology where all the possibilities have already been explored., 7 pages. arXiv admin note: text overlap with arXiv:2202.11459

Smart devices and Internet of Things (IoT) technologies are replacing or being incorporated into traditional devices at a growing pace. The use of digital interfaces to interact with these devices has become a common occurrence in homes, work spaces, and various industries around the world. The most common interfaces for these connected devices focus on mobile apps or voice control via intelligent virtual assistants. However, with augmented reality (AR) becoming more popular and accessible among consumers, there are new opportunities for spatial user interfaces to seamlessly bridge the gap between digital and physical affordances. In this paper, we present a human-subject study evaluating and comparing four user interfaces for smart connected environments: gaze input, hand gestures, voice input, and a mobile app. We assessed participants’ user experience, usability, task load, completion time, and preferences. Our results show multiple trade-offs between these interfaces across these measures. In particular, we found that gaze input shows great potential for future use cases, while both gaze input and hand gestures suffer from limited familiarity among users, compared to voice input and mobile apps.

Current infrastructure design, discouragement by parents, and lack of internal motivation act as barriers for people with visual impairments (PVIs) to perform physical activities at par with sighted individuals. This has triggered accessible exercise technologies to be an emerging area of research. However, most current solutions have either safety concerns and/or are expensive, hence limiting their mass adoption. In our work, we propose VStroll, a smartphone app to promote walking among PVIs, by enabling them to virtually explore real-world locations, while physically walking in the safety and comfort of their homes. Walking is a cheap, accessible, and a common physical activity for people with blindness. VStroll has several added features, such as places-of-interest (POI) announcement using spatial audio and voice input for route selection at every intersection, which helps the user to gain spatial awareness while walking. To understand the usability of VStroll, 16 participants used our app for five days, followed by a semi-structured interview. Overall, our participants took 253 trips, walked for 50.8 hours covering 121.6 kms. We uncovered novel insights, such as discovering new POIs and fitness-related updates acted as key motivators, route selection boosted their confidence in navigation, and spatial audio resulted in an immersive experience. We conclude the paper with key lessons learned to promote accessible exercise technologies.

Voice input has been used interactively in games for a long time. Existing voice games showed the potential of voice input games by utilizing various features of voice such as voice volume detection, pitch and rhythm detection, speech recognition, and conversation. However, most of these conventional voice games showed that the main game characteristics depended only on the voice input. In this context, we propose Harmonionz, a two-player collaborative game that goes beyond the limits of the existing voice input game genre. Our game has a novelty in which the voice plays a functional and story-wise core role, but the fun of the game is not limited to the voice. In our game, players can win the game when they explore a colorful map, avoid enemies, and eventually create a pitch required by the target with their voice. Each player has a different role. One defeats the approaching enemies and the other creates the pitch. The target pitch is mapped to the color to indicate players the color of the notes that they need to make. Players must find the required pitch by looking at the color on the target. The fun part of conventional voice input games was the voice itself only. However, voice input in our game can easily interact with overall game design including shooting and team-work. The worldview of our game, colorizing the pitch in order to interact with the colorful map, interacting with the background music are our attempts to make our game contextually and logically persuasive.

Entering text precisely by voice, users might encounter colloquial inserts, inappropriate wording, and recognition errors, which brings difficulties to voice editing. Users need to locate the errors and then correct them. In eyes-free scenarios, this select-modify mode brings a cognitive burden and a risk of error. This paper introduces neural networks and pre-trained models to understand users’ revision intention based on semantics, reducing the need for the information from users’ statements. We present two strategies. One is to remove the colloquial inserts automatically. The other is to allow users to edit by just speaking out the target words without having to say the context and the incorrect text. Accordingly, our approach can predict whether to insert or replace, the incorrect text to replace, and the position to insert. We implement these strategies in SmartEdit, an eyes-free voice input agent controlled with earphone buttons. The evaluation shows that our techniques reduce the cognitive load and decrease the average failure rate by 54.1% compared to descriptive command or re-speaking.

We present RePlay, an interaction substitution method designed to support people with upper extremity motor impairments while interacting with mobile apps. RePlay overcomes key limitations of existing approaches, allowing users to quickly access all app interface elements, even those not visible to the OS (a common situation especially with games). This is achieved through personalized mapping of interface elements to alternative inputs, such as external switches or non-verbal vocal sounds, hence adapting to users with diverse abilities. RePlay was implemented as an Android Accessibility Service, running without any changes at OS level. An evaluation conducted with ten participants with upper extremity motor impairments shows that RePlay can provide access to mobile games that would otherwise be inaccessible. Still, reaction time using RePlay is slower than direct touchscreen interaction by users without motor impairments, and therefore speed compensation might be needed. Both external switches and non-verbal voice input achieve comparable reaction times, while their combination is slower and more cognitively demanding. Prolonged voice input may also cause fatigue and should be used parsimoniously.

Hand-tracking allows users to engage with a virtual environment with their own hands, rather than the more traditional method of using accompanying controllers in order to operate the device they are using and interact with the virtual world. We seek to explore the range of low-level interaction actions and high-level interaction tasks and domains can be associated with the multimodal hand-tracking and voice input in VR. Thus, we created Let's Go There, which explores this joint-input method. So far, we have identified four low-level interaction actions which are exemplified by this demo: positioning oneself, positioning others, selection, and information assignment. We anticipate potential high-level interaction tasks and domains to include customer service training, social skills training, and cultural competency training (e.g. when interacting with older adults). Let's Go There, the system described in this paper, had been previously demonstrated at CUI 2020 and MobileHCI 2021. We have since updated our approach to its development to separate it into low- and high-level interactions. Thus, we believe there is value in bringing it to MobileHCI again to highlight these different types of interactions for further showcase and discussion.

Smart Mirror for Home Automation This research describes the design, construction and working of the smart mirror. Every morning our day begins by watching ourselves at least once in mirror before leaving our homes. We interact with it psychologically to find out how we look and how our attire is. Smart Mirror or Magic Mirror is one of the applications of Raspberry Pi. A computer screen embedded in mirror looks very futuristic. The Raspberry Pi stays at back scenes and controls the data displayed on mirror. While looking at mirror, the user can look at various notifications from social sites as well news, weather forecast and more things. Such mirrors can be programmed to work as control home appliances by voice input. The Raspberry Pi is connected to monitor via HDMI as well as it also has built-in Wi-Fi and Bluetooth interfaces so we can just swipe music and videos to mirror.

Alternatif mempelajari bahasa Jepang adalah dengan menggunakan mobile learning. Berdasarkan perbandingan yang dilakukan terhadap 7 aplikasi pembelajaran bahasa Jepang yang populer di playstore, belum terdapat aplikasi pembelajaran bahasa Jepang yang memiliki fitur speech recognition untuk pengucapan kalimat Jepang. Pada penelitian ini dirancang aplikasi pembelajaran hiragana Jepang untuk kalimat menggunakan speech recognition. Aplikasi ini menggunakan google cloud speech-to-text untuk fitur speech recognition. Algoritma yang diterapkan pada aplikasi ini adalah levenshtein distance untuk membandingkan inputan suara pengguna dengan teks pada database. Aplikasi berbasis android untuk pengguna dan web untuk admin. Berdasarkan hasil pengujian akurasi dan keberhasilan deteksi, jarak ideal aplikasi menerima inputan suara pengguna adalah 50 cm pada lingkungan tanpa noise dengan keberhasilan deteksi sebesar 100%. Pada pengujian sebelum dan sesudah menggunakan aplikasi terjadi peningkatan pengetahuan kosa kata sebesar 44.48% setelah pengguna menggunakan aplikasi. Berdasarkan pengujian usabilitas menggunakan SUS diperoleh hasil 73,01 yang menunjukkan bahwa aplikasi termasuk kedalam kategori acceptable.

Speech input, such as voice assistant and voice message, is an attractive interaction option for mobile users today. However, despite its popularity, there is a use limitation for smartphone speech input: users need to press a button or say a wake word to activate it before use, which is not very convenient. To address it, we match the motion that brings the phone to mouth with the user's intention to use voice input. In this paper, we present ProxiTalk, an interaction technique that allows users to enable smartphone speech input by simply moving it close to their mouths. We study how users use ProxiTalk and systematically investigate the recognition abilities of various data sources (e.g., using a front camera to detect facial features, using two microphones to estimate the distance between phone and mouth). Results show that it is feasible to utilize the smartphone's built-in sensors and instruments to detect ProxiTalk use and classify gestures. An evaluation study shows that users can quickly acquire ProxiTalk and are willing to use it. In conclusion, our work provides the empirical support that ProxiTalk is a practical and promising option to enable smartphone speech input, which coexists with current trigger mechanisms.

This study proposed a method—automatic lock of cursor movement (ALCM)—that locks a cursor at the center of a target at the instant the cursor enters the target. The method is intended to suppress irritating subtle cursor movements that occur when an eye-gaze input system transforms involuntary eye movement (e.g., drift) into cursor coordinates. The effectiveness of the proposed ALCM was verified using pointing performance (speed and accuracy) in two types of HCI tasks. In a drag task, we compared mouse input versus eye-gaze input with use of a backspace (BS) key or voice input. The key or voice facilitates target selection once the eye gaze was aligned with a target. In a menu selection task, we also compared mouse input with eye-gaze and the use of two voice input conditions. This task required gaze alignment with a menu and menu item by use of voice input for selection. Whether the ALCM function was added to the eye-gaze input system or not was a within-subject factor. The input method and target sizes were within-subject factors. The study concluded that the ALCM improved pointing accuracy for all eye-gaze input methods and all two tasks.

Muroja’ah is a method of repeating new memorization and old memorization that be heard to other people. This method is very popular in Indonesia. In order to make the learning process isn’t boring, it is required a learning media that can be used anytime, one of them are using Android application. The research describes how to build a Qur’an recitation correction application based on Android and integrate it with Google Speech so that application built more interesting and interactive with users. Google Speech be integrate as voice input media, users requested recite Qur’an verses then Google Speech will convert their voice to text. Text result obtained then matched with text in source code. Result of testing the accuracy of Google Speech implementation on this application it’s 100%, that means it’s been as expected.

Spoken language understanding (SLU) systems translate voice input commands to semantics which are encoded as an intent and pairs of slot tags and values. Most current SLU systems deploy a cascade of two neural models where the first one maps the input audio to a transcript (ASR) and the second predicts the intent and slots from the transcript (NLU). In this paper, we introduce FANS, a new end-to-end SLU model that fuses an ASR audio encoder to a multi-task NLU decoder to infer the intent, slot tags, and slot values directly from a given input audio, obviating the need for transcription. FANS consists of a shared audio encoder and three decoders, two of which are seq-to-seq decoders that predict non null slot tags and slot values in parallel and in an auto-regressive manner. FANS neural encoder and decoders architectures are flexible which allows us to leverage different combinations of LSTM, self-attention, and attenders. Our experiments show compared to the state-of-the-art end-to-end SLU models, FANS reduces ICER and IRER errors relatively by 30 % and 7 %, respectively, when tested on an in-house SLU dataset and by 0.86 % and 2 % absolute when tested on a public SLU dataset., Comment: Published in Interspeech 2021

Ecological momentary assessment (EMA) is used to evaluate subjects’ behaviors and moods in their natural environments, yet collecting real-time and self-report data with EMA is challenging due to user burden. Integrating voice into EMA data collection platforms through today’s intelligent virtual assistants (IVAs) is promising due to hands-free and eye-free nature. However, efficiently managing conversations and EMAs is non-trivial and time consuming due to the ambiguity of the voice input. We approach this problem by rethinking the data modeling of EMA questions and what is needed to deploy them on voice-first user interfaces. We propose a unified metadata schema that models EMA questions and the necessary attributes to effectively and efficiently integrate voice as a new EMA modality. Our schema allows user experience researchers to write simple rules that can be rendered at run-time, instead of having to edit the source code. We showcase an example EMA survey implemented with our schema, which can run on multiple voice-only and voice-first devices. We believe that our work will accelerate the iterative prototyping and design process of real-world voice-based EMA data collection platforms.

With the increasing popularity of remote sensing technology applications, some emergency scenarios require rapid retrieval of remote sensing images, such as earthquake rescue, etc. Due to the high efficiency of voice input, researchers have focused on cross-modal remote sensing image-voice retrieval methods. However, these methods have two major drawbacks: speech input lacks discrimination and the intra-modal semantic information is under used. To address these drawbacks, we propose a novel cross-modal feature fusion retrieval model. Our model provides a more optimized cross-modal common feature space than previous models and thus optimizes the retrieval performance. First, our model adds the extra textual keyword information to the audio feature for remote sensing image retrieval. Second, it introduces inter-modality adversarial learning and intra-modality semantic discrimination into the remote sensing image-voice retrieval task. We conducted experiments on two datasets modified from the UCM-Captions dataset and the Remote Sensing Image Caption Dataset. The experimental results show that our model outperforms state-of-the-art models in this task.

In view of the fact that current web questionnaire systems are not suitable for use by specific groups such as those at multi-ethnic inhabited multilingual area, and the status quo that the questionnaire systems based on monolith architecture are difficult to expand and maintain, we designed and implemented HeTong based on Spring Cloud, a frame of microservice architecture that is easy to extend and upgrade. HeTong is a multi-language questionnaire system that enables voice input for subjective investigation questions apart from general functions of conventional questionnaire systems. In this paper, the microservice system design for HeTong is introduced. Firstly, the overall framework of HeTong is described. Secondly, the business logic module is designed in detail where different types of microservices are divided according to different functions. Then, the business logic module is implemented based on Spring Cloud. Finally, the interactive interface between the user and the system is designed. The software tests show that the system can well realize the functions of language switching, user management, questionnaire management, data statistics and voice answering.

Sketch and speech are intuitive interaction methods that convey complementary information and have been independently used for 3D model retrieval in virtual environments. While sketch has been shown to be an effective retrieval method, not all collections are easily navigable using this modality alone. We design a new challenging database for sketch comprised of 3D chairs where each of the components (arms, legs, seat, back) are independently colored. To overcome this, we implement a multimodal interface for querying 3D model databases within a virtual environment. We base the sketch on the state-of-the-art for 3D Sketch Retrieval, and use a Wizard-of-Oz style experiment to process the voice input. In this way, we avoid the complexities of natural language processing which frequently requires fine-tuning to be robust. We conduct two user studies and show that hybrid search strategies emerge from the combination of interactions, fostering the advantages provided by both modalities., Published at IMX 2021

Wake-up-free techniques (e.g., Raise-to-Speak) are important for improving the voice input experience. We present ProxiMic, a close-to-mic (within 5 cm) speech sensing technique using only one microphone. With ProxiMic, a user keeps a microphone-embedded device close to the mouth and speaks directly to the device without wake-up phrases or button presses. To detect close-to-mic speech, we use the feature from pop noise observed when a user speaks and blows air onto the microphone. Sound input is first passed through a low-pass adaptive threshold filter, then analyzed by a CNN which detects subtle close-to-mic features (mainly pop noise). Our two-stage algorithm can achieve 94.1% activation recall, 12.3 False Accepts per Week per User (FAWU) with 68 KB memory size, which can run at 352 fps on the smartphone. The user study shows that ProxiMic is efficient, user-friendly, and practical.

In recent years, with the increase in the amount of multimedia data, the data cleaning technology in the traditional system has been unable to meet the current data cleaning requirements. For this reason, people need to upgrade the high-speed data transmission interface in the system hardware framework, so as to ensure the optimized operation of the central control host. This article summarizes the hardware design content of the entire multimedia database online system based on previous work experience. The author discusses the test situation of the system from five aspects: test platform, test indicators, text input test results, voice input test results, and main body performance analysis.

Navigation of visually impaired people is one of the important disputes that requires significant research consideration. The visually impaired users generally use white canes for obstacle detection by remembering all the familiar locations. In a new-fangled and unacquainted environment, they totally depend on individuals passing by to enquire for certain places. In this contemporary world along with various sensors, there should be a system with the most basic invention to make their life a bit tranquil. A contactless, hands free, LVU (Lidars and Vibrotactile Units), discrete wearable device was designed in this proposed work that allows blind people to detect obstacles. To provide a safe mobility for the impaired people, a suitable mobile assistance device is necessary. This paper propose a safe wearable device with audio output for benign local navigation in both inside and outdoor environment which help in assisting the user to discriminate free space from obstacles. The device presented is composed of wearable strap with sensors. By using TOF sensor attached in the front of the belt worn by the users, the pulses from the LiDAR provide a reliable and correct measurements of the distances between the handler and obstacles. The image captured by the camera is processed and classified by the convolution neural network algorithm. The identified image is given as an audio input to the audio jockey. The vibratory motor and voice intimation by audio jockey provides the haptic feedback when the disabled person reach the obstacle. The vibration motor is placed with a pretension point-loaded applicator to transmit the isolated vibrations to incapacitate person. Distance between the obstacle and the disabled person is measured by using LiDAR sensor and it will be given as a feedback to the visually impaired person with the voice input. Thus, this wearable device helps in assisting the visually impaired people in a more comfortable way than white canes.

In this paper, we investigate the reasons for the embarrassment that Japanese people feel when using voice inputs in public and propose a way to reduce it. Recently, multiple devices have added a voice input function. However, few people use voice input even in situations where using voice input would be better. We first investigate the reasons why Japanese people do not use voice inputs even when they can, and it becomes clear that many people feel embarrassment at speaking out. After the investigation, we conduct an experiment to compare the embarrassment caused by using voice input when the target of voice input is an inorganic machine compared to when it is a biological character. In the experiment, we conclude that a user's emotion can be altered by using an animal-like character as the target of voice commands.

As China’s population ages increasingly serious, the proportion of the elderly population is increasing. The safety of the elderly living alone has become the focus of social concern. The system designed in this paper runs on Raspberry PI 4B, connects the camera module and the voice input module. Based on the OpenCV framework, Python language programming is utilized to realize a remote monitoring system that can identify falls of elderly people living alone in the room and give an intelligent alarm.

This paper proposes a system that can be installed on a shopping cart that be used to assist visually impaired people. This method uses the concept of ROI, text localization and recognition by optical character recognition software and a webcam controlled by Raspberry Pi. Two micro controllers are used to program both the shopping cart movement and OCR. The user is required to give a voice input of the required product and the ROBOKART guides the user to that product.

We are living in the era of the fourth industrial revolution, which also treated as 4IR or Industry 4.0. Generally, 4IR considered as the mixture of robotics, artificial intelligence (AI), quantum computing, the Internet of Things (IoT) and other frontier technologies. It is obvious that nowadays a plethora of smart devices is providing services to make the daily life of human easier. However, in the morning most people around the globe use a traditional mirror while preparing themselves for daily task. The aim is to build a low-cost intelligent mirror system that can display a variety of details based on user recommendations. Therefore, in this article, Internet of Things (IoT) and AI-based smart mirror is introduced that will support the users to receive the necessary daily update of weather information, date, time, calendar, to-do list, updated news headlines, traffic updates, COVID-19 cases status and so on. Moreover, a face detection method also implemented with the smart mirror to construct the architecture more secure. Our proposed MirrorME application provides a success rate of nearly 87% in interacting with the features of face recognition and voice input. The mirror is capable of delivering multimedia facilities while maintaining high levels of security within the device., Comment: 17 pages, 7 figures, 5 tables

In recent years, the number of people with physical disabilities has been on the rise, with about half of them being physically disabled limb. A physically disabled person is a person with a physical disability that interferes with daily life, such as missing or malfunctioning fingers. This study focused on the use of computers in daily life. A person with a limb disability used a PC.Earlier studies that made "eye contact", "voice input", "facial expressions using a 3D camera". Those problems are the need to keep the distance between the face and the camera constant, the inability to protect privacy when people are around, and that 3D cameras are difficult and expensive to install. In this study, by using a webcam as input and machine learning, image processing techniques. Developing a system that allows a handicapped person to operate a computer with only facial movements without the use of arms or hands. We also evaluated the operability, improving the problems of previous studies.

Object detection has been extensively used and is considered as one of the prerequisites for various vision-based applications like object recognition, instance segmentation, pose estimation, and so on. This field has attracted much research attention in recent years due to its close relationship with scene analysis and image understanding. Traditional methods extract features and use shallow machine learning architectures to detect objects. These methods faced many difficulties in combining the extracted low-level features with the high-level context of the detector and classifier. Though the recent developments in deep learning architectures helped visual scene analysis and detection methods to perform remarkably, a lot more focus is required in performing voice-based scene analysis and object detection methods. In this paper, a voice-based scene analysis and object detection methods are proposed which aims in detecting a specific object through voice input in inventory storage. An automatic speech recognizer based on deep learning architecture is used to perform speech-to-text conversion. The converted text is used to identify which object is to be located in the scene. A dictionary-based search algorithm is used for reducing the search time of the class of interest. Object detection is performed using Faster RCNN architecture. Experimental results are tested on the retail product checkout dataset for evaluating the system performance. Experimental results show that this approach makes it possible for the model to segment only the specific object while ignoring all the other classes.

The automatic speech recognition has achieved quite good performance in the medical domain in the past several years. However, it is still lacking of enough practical solutions with considering the characteristics of real applications. In this work, we develop an approach to automatically generate semantic-coherent ultrasound reports with voice input. The solution includes key algorithms based on a proposed semantic tree structure. The radiologists do not need to follow the fixed templates. They just need to speak their specific observations for individual patients. We have carried out a set of experiments against a real world thyroid ultrasound dataset with more than 40,000 reports from a reputable hospital in Shanghai, China. The experimental results show that our proposed solution can generate concise and accurate reports.

This paper presents key aspects and trade-offs that designers and Human-Computer Interaction practitioners might encounter when designing multimodal interaction for older adults. The paper gathers literature on multimodal interaction and assistive technology, and describes a set of design challenges specific for older users. Building on these main design challenges, four trade-offs in the design of multimodal technology for this target group are presented and discussed. To highlight the relevance of the trade-offs in the design process of multimodal technology for older adults, two of the four reported trade-offs are illustrated with two user studies that explored mid-air and speech-based interaction with a tablet device. The first study investigates the design trade-offs related to redundant multimodal commands in older, middle-aged and younger adults, whereas the second one investigates the design choices related to the definition of a set of mid-air one-hand gestures and voice input commands. Further reflections highlight the design trade-offs that such considerations bring in the process, presenting an overview of the design choices involved and of their potential consequences.

The Information Retrieval user experience has remained largely unchanged since its inception for computers and mobile devices alike. However, recent developments in Virtual Reality hardware (pioneered by Oculus Rift in 2013) could introduce a new environment for Information Retrieval. This paper reports the results of a Scoping Literature Review (PRISMA-ScR) by rigorously examining the entire body of relevant literature with reproducible methods. The following research questions are answered: “What prototypes and concepts of Virtual Reality Information Retrieval systems with current generation hardware exist?”, “How are user interaction and especially user input realised in these systems?”, “What Retrieval features are used in these systems?”, “How are search results displayed in these systems?” and “Can these VR IR systems compare to traditional (non-VR) IR systems?”. After querying Google Scholar, Scopus and Web of Science, 1042 documents were reviewed in depth. Key features and attributes of the systems are summarised and discussed. Sketches of the user interfaces are included as well. The 30 documents that were relevant to the research questions include 16 distinct systems or theories. They discuss and utilise several user input technologies, ranging from controllers, voice input or hand tracking. Although conventional retrieval features are less common, systems enable retrieval of literature, 3D objects, images, books and texts and arrange them in a virtual space (e.g. as grids, arcs or maps). Finally, many of these systems were compared to conventional counterparts through user evaluation (n = 10). Most found user task times to be shorter or equal (n = 5, n = 3). In the seven papers that measured user performance (rate of correct solutions), three reported better performance (one equal). Notably, users always were more satisfied with the Virtual Reality systems compared to conventional ones. Possible limitations of these evaluations are demographic selection and the quality of baseline systems (control).

Assembly activities are a very important part of the production process of large and complex equipment. In order to solve the huge resource problem of large and complex equipment assembly process, this paper proposes a virtual assembly for satellite. Microsoft’s latest holographic glasses HoloLens2 is used as a display facility, and Unity3D is applied to develop a satellite part assembly scene based on augmented reality (AR) technology. Voice input and gaze in the Mixed Reality Toolkit (MRTK), provided by Microsoft, are adopted in the development scenario to enable users to interact with virtual models in a more natural way. Finally, the development scenario has been tested in a small space. The results show its validity and good performance.

Since the presentation is a one-sided communication, it is difficult to realize an interactive communication while grasping the behavior of the audience. Therefore, a method that conveys audience’s actions and affects such as interest and concern during the presentation is required. In this study, we focused on the dilated pupils which make favorable impression and developed a presentation support system using group pupil response interfaces. This system generates a dilated pupil response and a nodding motion based on voice input in each interface. Then, a communication experiment was conducted to evaluate the developed system under the condition that the system acts like audience. The effectiveness of the system was demonstrated by mean of sensory evaluations.

We present an end-to-end system for creating talking head video from text and a single image automatically. A two-dimensional clone of an individual made entirely of a still image of a person, that conveys the given text message to the target audience through replicated facial expressions. It can send any message that is delivered in a text to it with a voice that is cloned from a few seconds of their actual voice input. Our technique yields videos of natural facial movements including blinks and brow movements, as well as mouth movements that are in synchronization with the audio. The proposed method only uses a single base model to create a personalized video, avoiding the hassle of training it every time.

Traffic accidents worldwide have caused ever-increasing death numbers. By observing the significant number of casualties caused by traffic accidents, we can see worldwide road safety standards dropping significantly. In many cases, acquaintances or else urgent situation services are not notified within time. This can result in emergency services being delayed, which can, in turn, lead to serious harmor even death. To address the issue, in this paper we developa SOS Rescue mobile applicationbased on artificial intelligence and voice-recognition as well as Google STT (speech-to-text). The. STT system is based on methods and technologies that enable computers and computerized equipment to recognize as well as translate spoken language into text. Via Google cloud services, users of our system will find the application vastly efficient in recognizing the voice of the wounded, and it can automatically convert voice into text-form. Also, it can detect the real-time position of an accident, and automatically send a message to the responder. It will collect relevant information in text form via voice recognition in addition to a link of the audio file and real-time positioning. Just taping once to trigger our SOS Rescue System, it starts working automatically.

This paper presents a next generation health platform incorporating android application for verbal interaction and ICT (information and communications technology). The architecture of the platform extends the Internet of Things (IoT) to a web-centric paradigm. This system incorporates smart phone assistance to support meticulous care for the treatment of chronic diseases. A smart phone application will act as human assistance. The application records the patient's details and day to day activities by verbal interaction, just like a caretaker. This is achieved by taking the voice input from the patient and providing voice output. It is connected to a medical kit (wirelessly), through which it sends all the test details periodically to the diseases management hub (DMH). By accumulated data stored in the patient's electronic medical record, it creates reminders, warnings messages, and appropriate health advices when self management outcome deviates from pre- specified targets. E-health platform is a boon for the people enduring from chronic diseases.

Due to the open nature of voice input, voice assistant (VA) systems (e.g., Google Home and Amazon Alexa) are vulnerable to various security and privacy leakages (e.g., credit card numbers, passwords), especially when issuing critical user commands involving large purchases, critical calls, etc. Though the existing VA systems may employ voice features to identify users, they are still vulnerable to various acoustic-based attacks (e.g., impersonation, replay, and hidden command attacks). In this work, we propose a training-free voice authentication system, WearID, leveraging the cross-domain speech similarity between the audio domain and the vibration domain to provide enhanced security to the ever-growing deployment of VA systems. In particular, when a user gives a critical command, WearID exploits motion sensors on the user’s wearable device to capture the aerial speech in the vibration domain and verify it with the speech captured in the audio domain via the VA device’s microphone. Compared to existing approaches, our solution is low-effort and privacy-preserving, as it neither requires users’ active inputs (e.g., replying messages/calls) nor to store users’ privacy-sensitive voice samples for training. In addition, our solution exploits the distinct vibration sensing interface and its short sensing range to sound (e.g., 25cm) to verify voice commands. Examining the similarity of the two domains’ data is not trivial. The huge sampling rate gap (e.g., 8000Hz vs. 200Hz) between the audio and vibration domains makes it hard to compare the two domains’ data directly, and even tiny data noises could be magnified and cause authentication failures. To address the challenges, we investigate the complex relationship between the two sensing domains and develop a spectrogram-based algorithm to convert the microphone data into the lower-frequency “ motion sensor data” to facilitate cross-domain comparisons. We further develop a user authentication scheme to verify that the received voice command originates from the legitimate user based on the cross-domain speech similarity of the received voice commands. We report on extensive experiments to evaluate the WearID under various audible and inaudible attacks. The results show WearID can verify voice commands with 99.8% accuracy in the normal situation and detect 97.2% fake voice commands from various attacks, including impersonation/replay attacks and hidden voice/ultrasound attacks.

SIPENILAI is an automatic oral exam and grading system that has been developed by the Electrical Engineering Department Universitas Indonesia. The system developed in this thesis uses the google speech recognition API and LSA methods for assessment. The Google speech recognition API processes voice input which is then converted as text. LSA method looks for similarities between two documents and the Frobenius norm for assessment scores. Based on testing that has been done the average value of SIPENILAI accuracy is 83.64% for fluent users and 76.89% for non-fluent users.

Tabletop interactions can support various social activities such as collaborative review of dynamic sales information in the business setting or route planning with an interactive map among others. Sharing control between users of interactive surfaces can be difficult to ensure participation and efficiency for co-located contexts. We explore the differences between two modalities for interacting with a tabletop map including interactions with a voice assistant and tangible interactions with a shape-changing controller. To simulate a shared goal-oriented situation, we use the context of shared autonomous mobility and compare the tangible controller with voice input through simulated tasks of route planning and detours. The findings suggest that the most suitable interaction modality depends on user preferences, scenarios, and contexts. Tangible controls work well for communicating the state of the system and facilitating discussion, while voice can be more pragmatic and task oriented.

In this study, we developed a character control interface for a virtual character using fuzzy voice input in Japanese. Previous voice input interfaces have used pre-registered words in the system to control characters. The system uses the Japanese WordNet, a Japanese semantic dictionary, to associate words including synonyms with the character's commands so that even fuzzy voice input can be easily controlled. This system is considered to be effective as a voice input interface for systems that require instantaneous operations, such as action games.

Engaging in financial transactions has remained a hassle for the visually impaired due to the lack of technological products to facilitate their financial independence and inclusion. Automated teller machines (ATMs) and online banking applications do not provide any means through which the blind can engage in transactions without the need of a third party in managing their finances. This study aims at building a voice chatbot device that can be used for payment using Amazon Lex Service with Eyowo as the payment platform. The chatbot is built by leveraging on Artificial Intelligence (AI) technologies in the form of a service called Amazon Lex for configuring the bot with utterances and responses and Lambda Functions to validate the responses while carrying out the transactions by calling the Eyowo Application Programming Interface (API). A Raspberry pi single board computer is utilised as the medium of communication between the user and the chatbot. The Raspberry pi runs a script that collects voice input through a Universal Serial Bus (USB) microphone connected to it, which is sent to the Amazon Lex to be processed using Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). Then the chatbot sends back a suitable response to the user through the speaker connected to the Raspberry pi. The device brings comfort and security to the visually impaired by providing balance and cash transfers.

As mobile applications continue to offer more features, tackling the complexity of mobile interfaces can become challenging for older adults. Owing to a small screen and frequent updates that modify the visual layouts of menus and buttons, older adults can find it challenging to locate a function on a mobile interface quickly—even when familiar with the application. To address this issue, we present a system that supports older adults to quickly locate an on-screen feature on a mobile interface using speech queries. Our system allows users to ask for a function related to the current mobile screen using voice input. When that function is available, it provides visual guidance for users to engage with the pertinent user interface (UI) widget. The label and location of all UI components on the current screen are acquired via the Android’s Assist API. We discuss four scenarios of use.