Search

The current generation of Industry 4.0 involves the integration of automation and data in manufacturing, while the upcoming generation, Industry 5.0, aims to enable seamless interaction between humans, robots, and machines for sustainable and personalized product and service creation. This necessitates manufacturing components according to customer wishes, leading to frequent changes in the production line. In this context, automation proves unsustainable and could be addressed through Human–Robot Cooperation. In this paper, an Extended Reality (XR)-based Human–Robot Interaction with haptic feedback has been proposed to assist users in remotely performing tasks. The haptic feedback, generated using the Artificial Potential Field method, aids users in controlling robots without collisions. This paper introduces a multimodal XR-based framework for effective communication between humans and robots, utilizing vision models to provide information about the remote side to local users. This seamless communication is facilitated through multiple modalities such as haptic, speech, visual, and text. The system's effectiveness through pick-and-place activities and adaptation to it for complex industry use cases such as peg in the hole and welding are further evaluated. The results demonstrate the system's efficacy in two specific human–robot collaborative scenarios in industries: arc welding and assembly tasks. [ABSTRACT FROM AUTHOR]

This paper investigates how manual know-how is manifested in instructed action in a technologized setting characterised by distributed spaces and bodies. Drawing on data from a temporary game lab where co-present participants try out virtual reality games, it analyses how an experienced VR user instructs novice participants in accomplishing the steps required to set up the game, in interacting with virtual objects and resolving troubles with tasks in the virtual environment. We show how instructions are fitted to and serve to advance the practical activity, and how they are recipient designed to teach novice users the manual and bodily techniques for interacting with the virtual game and its features. • Instructing novice users of virtual reality shows orientation to manual skills. • Practices of instruction involve tactile interventions and manual actions targeting correct techniques for using the controllers. • Instructional practices include use of touch to guide users' understanding of relevant features of the tactile interface. • Experienced player guides novice's perception by drawing attention to system instructions. [ABSTRACT FROM AUTHOR]

This scoping review examines the broad applications, risks, and ethical challenges associated with Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), within the context of Metaverse. XR is revolutionizing fields such as immersive learning in education, medical and professional training, neuropsychological assessment, therapeutic interventions, arts, entertainment, retail, e-commerce, remote work, sports, architecture, urban planning, and cultural heritage preservation. The integration of multimodal technologies—haptics, eye-, face-, and body tracking, and brain–computer interfaces—enhances user engagement and interactivity, playing a key role in shaping the immersive experiences in the Metaverse. However, XR's expansion raises serious concerns, including data privacy risks, cybersecurity vulnerabilities, cybersickness, addiction, dissociation, harassment, bullying, and misinformation. These psychological, social, and security challenges are further complicated by intense advertising, manipulation of public opinion, and social inequality, which could disproportionately affect vulnerable individuals and social groups. This review emphasizes the urgent need for robust ethical frameworks and regulatory guidelines to address these risks while promoting equitable access, privacy, autonomy, and mental well-being. As XR technologies increasingly integrate with artificial intelligence, responsible governance is essential to ensure the safe and beneficial development of the Metaverse and the broader application of XR in enhancing human development. [ABSTRACT FROM AUTHOR]

In the construction industry, where work environments are complex, unstructured and often dangerous, the implementation of human–robot collaboration (HRC) is emerging as a promising advancement. This underlines the critical need for intuitive communication interfaces that enable construction workers to collaborate seamlessly with robotic assistants. This study introduces a conversational virtual reality (VR) interface integrating multimodal interaction to enhance intuitive communication between construction workers and robots. By integrating voice and controller inputs with the robot operating system (ROS), building information modeling (BIM), and a game engine featuring a chat interface powered by a large language model (LLM), the proposed system enables intuitive and precise interaction within a VR setting. Evaluated by 12 construction workers through a drywall installation case study, the proposed system demonstrated its low workload and high intuitiveness and ease of use with succinct command inputs. The proposed multimodal interaction system suggests that such technological integration can substantially advance the integration of robotic assistants in the construction industry. [ABSTRACT FROM AUTHOR]

Text‐based hair design (TBHD) represents an innovative approach that utilizes text instructions for crafting hairstyle and colour, renowned for its flexibility and scalability. However, enhancing TBHD algorithms to improve generation quality and editing accuracy remains a current research difficulty. One important reason is that existing models fall short in alignment and fusion designs. Therefore, we propose a new layered multimodal fusion network called ETBHD‐HMF, which decouples the input image and hair text information into layered hair colour and hairstyle representations. Within this network, the channel enhancement separation (CES) module is proposed to enhance important signals and suppress noise for text representation obtained from CLIP, thus improving generation quality. Based on this, we develop the weighted mapping fusion (WMF) sub‐networks for hair colour and hairstyle. This sub‐network applies the mapper operations to input image and text representations, acquiring joint information. The WMF then selectively merges image representation and joint information from various style layers using weighted operations, ultimately achieving fine‐grained hairstyle designs. Additionally, to enhance editing accuracy and quality, we design a modality alignment loss to refine and optimize the information transmission and integration of the network. The experimental results of applying the network to the CelebA‐HQ dataset demonstrate that our proposed model exhibits superior overall performance in terms of generation quality, visual realism, and editing accuracy. ETBHD‐HMF (27.8 PSNR, 0.864 IDS) outperformed HairCLIP (26.9 PSNR, 0.828 IDS), with a 3% higher PSNR and a 4% higher IDS. [ABSTRACT FROM AUTHOR]

Immersive virtual learning environments (iVLEs), particularly serious games and virtual simulations, typically ignore psychomotor skills development, partly due to the difficulty and cost associated with accurately replicating touch. Simulating touch, also known as haptics, requires specialized haptic devices that are not widely accessible at the consumer-level. Using visual (and/or auditory) cues, pseudo-haptics aims to mimic touch sensations without haptic devices. Although pseudo-haptics has predominantly focused on visual cues, a 2019 review by Collins and Kapralos on pseudo-haptics emphasized the role of auditory cues and cross-modal interactions. Since then, great advancements, notably during the COVID-19 pandemic's shift to remote learning, have been made. Here, we build upon the work of Collins and Kapralos with a narrative review on audio-based pseudo-haptics. This narrative review explores 17 articles obtained from the Google Scholar, RefSeek, Scopus, and PubMed databases, with the aim of providing a comprehensive summary of the progress in this field since 2019. Pseudo-haptics presents a viable alternative to simulate various aspects of touch, including weight, stiffness, roughness, dampness, force, and glossiness, when haptic devices are unavailable, enhancing immersion and providing the potential to improve psychomotor skill training within iVLEs. [ABSTRACT FROM AUTHOR]

Emergent states are temporal group phenomena that arise from collective affective, behavioral, and cognitive processes shared among the group's members during their interactions. Cohesion is one such state, mainly conceptualized by scholars as affective in nature, and frequently distinguished into the two dimensions social and task cohesion. Whereas social cohesion is related to the need of belonging to a group, task cohesion is related to the group's goals and tasks. In this article, we emphasize the importance of behavioral interaction dynamics to predict cohesion's dynamics. Drawing from Social Science insights, we investigate the interplay between social and task cohesion to predict their dynamics across group tasks from nonverbal behavioral features. Three computational architectures exploiting transfer learning are presented. Transfer learning capitalizes on information learnt by a model for a specific dimension to predict the dynamics of the other dimension. Results show that integrating the influence of social cohesion to predict dynamics of task cohesion outperforms state-of-the-art. To predict dynamics of social cohesion, a model integrating the reciprocal impact of social and task cohesion significantly improves performance with respect to the state-of-the-art and a model only integrating the impact of task cohesion on dynamics of social cohesion. [ABSTRACT FROM AUTHOR]

Abstract The concept of an innovative human–machine interface and interaction modes based on virtual and augmented reality technologies for airport control towers has been developed with the aim of increasing the human performances and situational awareness of air traffic control operators. By presenting digital information through see-through head-mounted displays superimposed over the out-of-the-tower view, the proposed interface should stimulate controllers to operate in a head-up position and, therefore, reduce the number of switches between a head-up and a head-down position even in low visibility conditions. This paper introduces the developed interface and describes the exercises conducted to validate the technical solutions developed, focusing on the simulation platform and exploited technologies, to demonstrate how virtual and augmented reality, along with additional features such as adaptive human–machine interface, multimodal interaction and attention guidance, enable a more natural and effective interaction in the control tower. The results of the human-in-the-loop real-time validation exercises show that the prototype concept is feasible from both an operational and technical perspective, the solution proves to support the air traffic controllers in working in a head-up position more than head-down even with low-visibility operational scenarios, and to lower the time to react in critical or alerting situations with a positive impact on the human performances of the user. While showcasing promising results, this study also identifies certain limitations and opportunities for refinement, aimed at further optimising the efficacy and usability of the proposed interface.

Virtual reality (VR) collaborative manipulation requires a proper understanding of the user's self, other collaborators, and shared objects in the virtual environment. Users must be aware of the movements of themselves and others to coordinate their actions when they hand over an object or move it together. The awareness of collaboration plays an essential role in virtual environments. However, previous studies have focused on providing awareness only for cooperative manipulation. This study proposes a novel framework for dual awareness that comprehensively manages the state of interactions between multiple users and shared virtual objects and provides distinct sensory cues to distinguish themselves from collaborators. The experimental results showed that participants could easily and quickly collaborate on manipulation using our feedback, and they also answered positively in the subjective evaluation. [ABSTRACT FROM AUTHOR]

Robots are becoming an increasingly important part of our society and have started to be used in tasks that require communicating with humans. Communication can be decoupled in two dimensions: symbolic (information aimed to achieve a particular goal) and spontaneous (displaying the speaker's emotional and motivational state) communication. Thus, to enhance human–robot interactions, the expressions that are used have to convey both dimensions. This paper presents a method for modelling a robot's expressiveness as a combination of these two dimensions, where each of them can be generated independently. This is the first contribution of our work. The second contribution is the development of an expressiveness architecture that uses predefined multimodal expressions to convey the symbolic dimension and integrates a series of modulation strategies for conveying the robot's mood and emotions. In order to validate the performance of the proposed architecture, the last contribution is a series of experiments that aim to study the effect that the addition of the spontaneous dimension of communication and its fusion with the symbolic dimension has on how people perceive a social robot. Our results show that the modulation strategies improve the users' perception and can convey a recognizable affective state. [ABSTRACT FROM AUTHOR]

Art-based interventions are increasingly employed in workplace settings for the purposes of organizational training and development. In this article, we examine the implementation of a workshop where hand drumming is used in two client organizations. Through the detailed analysis of the trainers' actions, the article shows how the trainers use the combination of verbal and embodied means in order to instruct and encourage participation in joint drumming. In particular, the article demonstrates how the trainers' instructive practice change across the instructional phases of the workshop and form a trajectory from pre-planned rhythm patterns to free collective improvisation in the pre-designed service product. [ABSTRACT FROM AUTHOR]

Virtual Reality (VR)‐enabled multi‐user collaboration has been gradually applied in academic research and industrial applications, but it still has key problems. First, it is often difficult for users to select or manipulate objects in complex three‐dimesnional spaces, which greatly affects their operational efficiency. Second, supporting natural communication cues is crucial for cooperation in VR, especially in collaborative tasks, where ambiguous verbal communication cannot effectively assign partners the task of selecting or manipulating objects. To address the above issues, in this paper, we propose a new interaction method, Eye‐Gesture Combination Interaction in VR, to enhance the execution of collaborative tasks by sharing the visualization of eye movement and gesture data among partners. We conducted user experiments and showed that using dots to represent eye gaze and virtual hands to represent gestures can help users complete tasks faster than other visualization methods. Finally, we developed a VR multi‐user collaborative assembly system. The results of the user study show that sharing gaze points and gestures among users can significantly improve the productivity of collaborating users. Our work can effectively improve the efficiency of multi‐user collaborative systems in VR and provide new design guidelines for collaborative systems in VR. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Data Acquisition & Processing / Shu Ju Cai Ji Yu Chu Li is the property of Editorial Department of Journal of Nanjing University of Aeronautics & Astronautics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

This study reviews empirical studies on applying Multimodal Interactions in Virtual Reality (MIVR) environments in education. We selected 25 papers and reviewed them based on (1) educational research domains, (2) MIVR modalities, (3) MIVR devices/tools, and (4) the methodology and major findings. The results suggest that MIVR focused on language and mathematical science. The common modalities were Visual + Haptic and Visual + Auditory. The mature haptic feedback devices and Virtual Reality (VR) app have advantages in devices. In methodology and key findings, the participants were primarily from middle school and university; the duration of the studies commonly less than one day, and the number of participants were mostly less than 50; the research design more used quasi-experimental; the research methods were quantitative and mixed; the common data collection approaches were tests and questionnaires; scholars evaluated learners’ knowledge, skills development, learning experience, learning attitude, and opinion about the MIVR system. The advantages of MIVR in education include natural and real learning experiences, flexibility and efficiency in obtaining information etc., disadvantages were availability issues, cognitive load and personalized insufficiency. [ABSTRACT FROM AUTHOR]

With the development of digital technology, tourists have higher expectations for their experiences with digital cultural heritage (CH). However, there is limited research and design guidelines on how to effectively convey expert knowledge and transfer the value and connotations of digital CH to users in a virtual environment. Based on existing literature, we proposed a systematic design strategy for immersive virtual reality (IVR) systems, incorporating multimodal interaction, gamification, and storytelling for CH presentation, development, and promotion to the public. Accordingly, taking the virtual restoration of the Dunhuang murals as an example, we developed an IVR system that provided users with learning, interactive, and entertaining experiences, facilitating their transition from mere visitors to active learners while engaging with digital CH. We conducted a between-subjects user study involving 42 participants. The results demonstrated that our IVR system motivated users interests in CH, promoted CH preservation awareness, and could be applied in the domain of digital CH experiences. [ABSTRACT FROM AUTHOR]

Named entity recognition (NER) is a fundamental task in natural language processing that involves identifying and classifying entities in sentences into pre-defined types. It plays a crucial role in various research fields, including entity linking, question answering, and online product recommendation. Recent studies have shown that incorporating multilingual and multimodal datasets can enhance the effectiveness of NER. This is due to language transfer learning and the presence of shared implicit features across different modalities. However, the lack of a dataset that combines multilingualism and multimodality has hindered research exploring the combination of these two aspects, as multimodality can help NER in multiple languages simultaneously. In this paper, we aim to address a more challenging task: multilingual and multimodal named entity recognition (MMNER), considering its potential value and influence. Specifically, we construct a large-scale MMNER dataset with four languages (English, French, German and Spanish) and two modalities (text and image). To tackle this challenging MMNER task on the dataset, we introduce a new model called 2M-NER, which aligns the text and image representations using contrastive learning and integrates a multimodal collaboration module to effectively depict the interactions between the two modalities. Extensive experimental results demonstrate that our model achieves the highest F1 score in multilingual and multimodal NER tasks compared to some comparative and representative baselines. Additionally, in a challenging analysis, we discovered that sentence-level alignment interferes a lot with NER models, indicating the higher level of difficulty in our dataset. [ABSTRACT FROM AUTHOR]

In this paper, we present an approach to improve pointing methods and target selection on tactile human–machine interfaces. This approach defines a two-level highlighting technique (TLH) based on the direction of gaze for target selection on a touch screen. The technique uses the orientation of the user's head to approximate the direction of his gaze and uses this information to preselect the potential targets. An experimental system with a multimodal interface has been prototyped to assess the impact of TLH on target selection on a touch screen and compare its performance with that of traditional methods (mouse and touch). We conducted an experiment to assess the effectiveness of our proposition in terms of the rate of selection errors made and time for completion of the task. We also made a subjective estimation of ease of use, suitability for selection, confidence brought by the TLH, and contribution of TLH to improving the selection of targets. Statistical results show that the proposed TLH significantly reduces the selection error rate and the time to complete tasks. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer Engineering & Applications is the property of Beijing Journal of Computer Engineering & Applications Journal Co Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Exposure to varied noisy environments impairs the recognition performance of artificial intelligence-based speech recognition technologies. Degradedperformance services can be utilized as limited systems that assure good performance in certain environments, but impair the general quality of speech recognition services. This study introduces an audiovisual speech recognition (AVSR) model robust to various noise settings, mimicking human dialogue recognition elements. The model converts word embeddings and log-Mel spectrograms into feature vectors for audio recognition. A dense spatial–temporal convolutional neural network model extracts features from log-Mel spectrograms, transformed for visual-based recognition. This approach exhibits improved aural and visual recognition capabilities. We assess the signalto-noise ratio in nine synthesized noise environments, with the proposed model exhibiting lower average error rates. The error rate for the AVSR model using a three-feature multi-fusion method is 1.711%, compared to the general 3.939% rate. This model is applicable in noise-affected environments owing to its enhanced stability and recognition rate.

Real-time RGB-D semantic segmentation is crucial for tasks like dynamic environment analysis of mobile robots or autonomous vehicles, real-time and efficient segmentation results can enhance the accuracy of subsequent tasks such as free space detection, mapping and navigation. It requires models to balance computational cost and performance by employing more efficient mechanisms to effectively recognize differences in RGB-D multimodal information, retain complementary information, and reduce redundancies. Currently, although convolutional neural network (CNN) methods are less accurate than Transformer-based methods, they offer stronger real-time performance under the same computational load. Therefore, in this study, we proposed the Efficient Feature Interaction Network (EFINet), a real-time RGB-D segmentation method that uses a lightweight CNN encoder and incorporates encoder blocks with a lightweight upsampling method Dysample and the carefully optimized number of ConvNeXt V2 blocks, to redesign the decoder and minimize redundant computations. Additionally, we propose a robust cross-modal interaction mechanism that facilitates the exchange of useful information and the elimination of noise between modalities with minimal computational overhead. By incorporating lightweight convolutions, the proposed method can acquire multi-scale global information with minimal additional computation, thus providing the model with rich contextual information. The proposed EFINet has 38.0 M parameters and 40.0 GFlops, achieving 55.5% and 50.6% mIoU on the challenging NYUDv2 and SUN RGB-D datasets, respectively, and real-time inference at 31.2 FPS. EFINet outperforms several state-of-the-art methods in the real-time RGB-D segmentation task while achieving a better balance between segmentation performance and computational efficiency.

This scoping review examines the broad applications, risks, and ethical challenges associated with Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), within the context of Metaverse. XR is revolutionizing fields such as immersive learning in education, medical and professional training, neuropsychological assessment, therapeutic interventions, arts, entertainment, retail, e-commerce, remote work, sports, architecture, urban planning, and cultural heritage preservation. The integration of multimodal technologies—haptics, eye-, face-, and body tracking, and brain–computer interfaces—enhances user engagement and interactivity, playing a key role in shaping the immersive experiences in the Metaverse. However, XR’s expansion raises serious concerns, including data privacy risks, cybersecurity vulnerabilities, cybersickness, addiction, dissociation, harassment, bullying, and misinformation. These psychological, social, and security challenges are further complicated by intense advertising, manipulation of public opinion, and social inequality, which could disproportionately affect vulnerable individuals and social groups. This review emphasizes the urgent need for robust ethical frameworks and regulatory guidelines to address these risks while promoting equitable access, privacy, autonomy, and mental well-being. As XR technologies increasingly integrate with artificial intelligence, responsible governance is essential to ensure the safe and beneficial development of the Metaverse and the broader application of XR in enhancing human development.

This case study explores the trajectory of a reading teacher's self-directed development when implementing drama-based pedagogy in an under-resourced College in China. Through qualitative analyses of multiple data sources collected over one semester, the study demonstrates that the teacher's self-directed use of drama-based pedagogy following teacher student joint learning of reading texts was contextually driven with his use of drama-based pedagogy being partially motivated by its potential to engage his students' interest in reading learning. The teacher was able to dynamically adjust and refine his implementation of drama-based pedagogy in relation to his students' post-reading reading reflections, their drama planning and dramatic presentations, their post-performance reflections, as well as their in-class learning. The students' increased interest for reading and improved literacy development in the latter part of the semester augmented the teacher's confidence in drama-based pedagogy, and feelings of self-empowerment resulting from his self-directed development. The study concludes that, along with teachers' efforts to deal with the dynamic complexities of better meeting students' needs, the self-directed use of drama-based pedagogy can be a helpful and accessible tool to support reading teachers to re-conceptualize their self-directed development in an under-resourced educational context. [ABSTRACT FROM AUTHOR]

Over the last decade, several projects have demonstrated how interactive tactile graphics and tangible interfaces can improve and enrich access to information for people with vision impairments. While the former can be used to display a relatively large amount of information, they cannot be physically updated, which constrains the type of tasks that they can support. On the other hand, tangible interfaces are particularly suited for the (re)construction and manipulation of graphics, but the use of physical objects also restricts the type and amount of information that they can convey. We propose to bridge the gap between these two approaches by investigating the potential of tactile and tangible graphics for people with vision impairments. Working closely with special education teachers, we designed and developed the FlexiBoard, an affordable and portable system that enhances traditional tactile graphics with tangible interaction. In this paper, we report on the successive design steps that enabled us to identify and consider technical and design requirements. We thereafter explore two domains of application for the FlexiBoard: education and board games. Firstly, we report on one brainstorming session that we organized with four teachers in order to explore the application space of tangible and tactile graphics for educational activities. Secondly, we describe how the FlexiBoard enabled the successful adaptation of one visual board game into a multimodal accessible game that supports collaboration between sighted, low-vision and blind players. [ABSTRACT FROM AUTHOR]

In Industry 4.0, manufacturing entails a rapid change in customer demands which leads to mass customization. The variation in customer requirements leads to small batch sizes and several process variations. Assembly task is one of most important steps in any manufacturing process. A factory floor worker often needs a guidance system due to variations in product or process, to assist them in assembly task. Existing Augmented Reality (AR) based systems use markers for each assembly component for detection which is time consuming and laborious. This paper proposed utilizing state-of-the-art deep learning based object detection technique and employed a regression based mapping technique to obtain the 3D locations of assembly components. Automatic detection of machine parts was followed by a multimodal interface involving both eye gaze and hand tracking to guide the manual assembly process. We proposed eye cursor to guide the user through the task and utilized fingertip distances along with object sizes to detect any error committed during the task. We analyzed the proposed mapping method and found that the mean mapping error was 1.842 cm. We also investigated the effectiveness of the proposed multimodal user interface by conducting two user studies. The first study indicated that the current interface design with eye cursor enabled participants to perform the task significantly faster compared to the interface without eye cursor. The shop floor workers during the second user study reported that the proposed guidance system was comprehendible and easy to use to complete the assembly task. Results showed that the proposed guidance system enabled 11 end users to finish the assembly of one pneumatic cylinder within 55 s with average TLX score less than 25 in a scale of 100 and Cronbach alpha score of 0.8 indicating convergence of learning experience. [ABSTRACT FROM AUTHOR]

With the advancement of interactive technology, alternative input modalities are often used, instead of conventional ones, to create intuitive, efficient, and user-friendly avenues of controlling and collaborating with robots. Researchers have examined the efficacy of natural interaction modalities such as gesture or voice in single and dual-task scenarios. These investigations have aimed to address the potential of the modalities on diverse applications encompassing activities like online shopping, precision agriculture, and mechanical component assembly, which involve tasks like object pointing and selection. This article aims to address the impact on user performance in a practical human–robot interaction application where a fixed-base robot is controlled through the utilization of natural alternative modalities. We explored this by investigating the impact of single-task and dual-task conditions on user performance for object picking and dropping. We undertook two user studies—one focusing on single-task scenarios, employing a fixed-base robot for object picking and dropping and the other encompassing dual-task conditions, utilizing a mobile robot for a driving scenario. We measured task completion times and estimated cognitive workload through the NASA Task Load Index (TLX), which offers a subjective, multidimensional scale measuring the perceived cognitive workload of a user. The studies revealed that the ranking of completion times for the alternative modalities remained consistent across both single-task and dual-task scenarios. However, the ranking based on perceived cognitive load was different. In the single-task study, the gesture-based modality resulted the highest TLX score, contrasting with the dual-task study, where the highest TLX score was associated with the eye gaze-based modality. Likewise, the speech-based modality achieved a lower TLX score compared to eye gaze and gesture in the single-task study, but its TLX score in the dual-task study was between gesture and eye gaze. These outcomes suggest that the efficacy of alternative modalities is contingent not only on user preferences but also on the specific situational context. [ABSTRACT FROM AUTHOR]

Exposure to varied noisy environments impairs the recognition performance of artificial intelligence‐based speech recognition technologies. Degraded‐performance services can be utilized as limited systems that assure good performance in certain environments, but impair the general quality of speech recognition services. This study introduces an audiovisual speech recognition (AVSR) model robust to various noise settings, mimicking human dialogue recognition elements. The model converts word embeddings and log‐Mel spectrograms into feature vectors for audio recognition. A dense spatial–temporal convolutional neural network model extracts features from log‐Mel spectrograms, transformed for visual‐based recognition. This approach exhibits improved aural and visual recognition capabilities. We assess the signal‐to‐noise ratio in nine synthesized noise environments, with the proposed model exhibiting lower average error rates. The error rate for the AVSR model using a three‐feature multi‐fusion method is 1.711%, compared to the general 3.939% rate. This model is applicable in noise‐affected environments owing to its enhanced stability and recognition rate. [ABSTRACT FROM AUTHOR]

This article examines requests for verification across varieties of Spanish. It focuses on the role of gaze behaviour in the design of the request and shows how gaze is interrelated with other resources, such as facial expressions, epistemic markers, prosody and tag questions. By comparing speakers from Argentina, Bolivia, Colombia and Ecuador, the article finds similarities but also differences between the groups of speakers. The speakers from Bolivia and Ecuador rarely establish mutual gaze or use facial expressions during the verification sequence. Instead, more frequently, they deploy other resources for obtaining verifications, such as tag questions, hand gestures and rising intonation. The article considers differences in gaze behaviour within the larger machinery of response mobilisation and shows how gaze is intertwined with other aspects of conversational organisation. Results are based on qualitative and quantitative analysis. The data stems from face-to-face conversations recorded in similar side-by-side settings. [ABSTRACT FROM AUTHOR]

Featured Application: This paper introduces a flexible parallel multimodal integration framework that outlines a development paradigm and encourages multimodal application across various scenarios. Multimodal interaction systems can provide users with natural and compelling interactive experiences. Despite the availability of various sensing devices, only some commercial multimodal applications are available. One reason may be the need for a more efficient framework for fusing heterogeneous data and addressing resource pressure. This paper presents a parallel multimodal integration framework that ensures that the errors and external damages of integrated devices remain uncorrelated. The proposed relative weighted fusion method and modality delay strategy process the heterogeneous data at the decision level. The parallel modality operation flow allows each device to operate across multiple terminals, reducing resource demands on a single computer. The universal fusion methods and independent devices further remove constraints on the integrated modality number, providing the framework with extensibility. Based on the framework, we develop a multimodal virtual shopping system, integrating five input modalities and three output modalities. The objective experiments show that the system can accurately fuse heterogeneous data and understand interaction intent. User studies indicate the immersive and entertaining of multimodal shopping. Our framework proposes a development paradigm for multimodal systems, fostering multimodal applications across various domains. [ABSTRACT FROM AUTHOR]

Immersive virtual learning environments (iVLEs), particularly serious games and virtual simulations, typically ignore psychomotor skills development, partly due to the difficulty and cost associated with accurately replicating touch. Simulating touch, also known as haptics, requires specialized haptic devices that are not widely accessible at the consumer-level. Using visual (and/or auditory) cues, pseudo-haptics aims to mimic touch sensations without haptic devices. Although pseudo-haptics has predominantly focused on visual cues, a 2019 review by Collins and Kapralos on pseudo-haptics emphasized the role of auditory cues and cross-modal interactions. Since then, great advancements, notably during the COVID-19 pandemic’s shift to remote learning, have been made. Here, we build upon the work of Collins and Kapralos with a narrative review on audio-based pseudo-haptics. This narrative review explores 17 articles obtained from the Google Scholar, RefSeek, Scopus, and PubMed databases, with the aim of providing a comprehensive summary of the progress in this field since 2019. Pseudo-haptics presents a viable alternative to simulate various aspects of touch, including weight, stiffness, roughness, dampness, force, and glossiness, when haptic devices are unavailable, enhancing immersion and providing the potential to improve psychomotor skill training within iVLEs.