Search

Robotic avatar technology has the potential to impact the future of human connectivity, transporting the sense of a human’s presence to a remote location anywhere and in real-time. In this regard, the recent ANA Avatar XPRIZE challenge fostered the development of the ultimate generation of non-autonomous robotic avatar designs. This paper is devoted to introducing our system proposal, allowing intuitive motion-based control and multi-modal feedback. The teleoperated robot endorses an anthropomorphic upper body with dual arms and dexterous hands for fine manipulation of even small objects, as well as an omnidirectional platform for improved and safe locomotion. Special focus is pointed at the crucial challenge of providing self-body perception to the operator, in particular regarding control of the arms and rendering of haptic sensations. To this end, we propose an upper-limb exoskeleton and a teleoperation architecture allowing retargeting of the operator’s skills and receiving tactile and kinesthetic force feedback with realistic and informative perception of the own’s arms. Lastly, the platform was validated during laboratory and challenge missions mimicking several social, cooperative, and cultural scenarios from which we report on the lesson learned and future improvements.

Quality evaluation of kindergarten classrooms is important for promoting teachers’ competencies and ensuring high-quality education. Currently, human observers physically visit kindergartens and provide assessments by observing classroom activities. However, rural areas suffer from a lack of observers, despite the high demand for quality evaluation. In this study, we proposed the use of telepresence robots in early childhood education and care, specifically for the evaluation of kindergarten classroom quality. A telepresence robot is a contemporary tool that provides both clear visibility of the teacher and the ability to hear interactions with children as a part of the classroom, while reducing the observer’s transportation cost and time. We conducted a field study in which observers evaluated the classroom both in-person and via telepresence robots. A total of 12 observation and rating scores were collected for both the quantitative and qualitative analyses. The overall quality evaluation score using telepresence observation showed no statistically significant difference from that obtained using in-person observation. In addition, the participants in this experiment showed a positive attitude toward the robot observation. However, we identified some weaknesses in current telepresence robots; for example, some observers were unable to observe instantaneous behaviors and the particularity of scenes. Telepresence robots should be improved to provide quick movement with high-quality video and sound for quality evaluation of kindergarten classrooms.

Avatar-mediated symmetric Augmented Reality (AR) telepresence has emerged with the ability to empower users located in different remote spaces to interact with each other in 3D through avatars. However, different spaces have heterogeneous structures and features, which bring difficulties in synchronizing avatar motions with real user motions and adapting avatar motions to local scenes. To overcome these issues, existing methods generate mutual movable spaces or retarget the placement of avatars. However, these methods limit the telepresence experience in a small sub-area space, fix the positions of users and avatars, or adjust the beginning/ending positions of avatars without presenting smooth transitions. Moreover, the delay between the avatar retargeting and users' real transitions can break the semantic synchronization between users' verbal conversation and perceived avatar motion. In this paper, we first examine the impact of the aforementioned transition delay and explore the preferred transition style with the existence of such delay through user studies. With the results showing a significant negative effect of avatar transition delay and providing the design choice of the transition style, we propose a Predict-and-Drive controller to diminish the delay and present the smooth transition of the telepresence avatar. We also introduce a grouping component as an upgrade to immediately calculate a coarse virtual target once the user initiates a transition, which could further eliminate the avatar transition delay. Once having the coarse virtual target or an exactly predicted target, we find the corresponding target for the avatar according to the pre-constructed mapping of objects of interest between two spaces. The avatar control component maintains an artificial potential field of the space and drives the avatar towards the target while respecting the obstacles in the physical environment. We further conduct ablation studies to evaluate the effectiveness of our proposed components. [ABSTRACT FROM AUTHOR]

Image-based geometric modeling and novel view synthesis based on sparse large-baseline samplings are challenging but important tasks for emerging multimedia applications such as virtual reality and immersive telepresence. Existing methods fail to produce satisfactory results due to the limitation on inferring reliable depth information over such challenging reference conditions. With the popularization of commercial light field (LF) cameras, capturing LF images (LFIs) is as convenient as taking regular photos, and geometry information can be reliably inferred. This inspires us to use a sparse set of LF captures to render high-quality novel views globally. However, the fusion of LF captures from multiple angles is challenging due to the scale inconsistency caused by various capture settings. To overcome this challenge, we propose a novel scale-consistent volume rescaling algorithm that robustly aligns the disparity probability volumes (DPV) among different captures for scale-consistent global geometry fusion. Based on the fused DPV projected to the target camera frustum, novel learning-based modules (i.e., the attention-guided multi-scale residual fusion module, and the disparity field-guided deep re-regularization module), which comprehensively regularize noisy observations from heterogeneous captures for high-quality rendering of novel LFIs, have been proposed. Both quantitative and qualitative experiments over the Stanford Lytro Multi-view LF dataset show that the proposed method outperforms state-of-the-art methods significantly under different experiment settings for disparity inference and LF synthesis. [ABSTRACT FROM AUTHOR]

In this article, we propose a system design and implementation for output-sensitive reconstruction, transmission and rendering of 3D video avatars in distributed virtual environments. In our immersive telepresence system, users are captured by multiple RGBD sensors connected to a server that performs geometry reconstruction based on viewing feedback from remote telepresence parties. This feedback and reconstruction loop enables visibility-aware level-of-detail reconstruction of video avatars regarding geometry and texture data, and considers individual and groups of collocated users. Our evaluation reveals that our approach leads to a significant reduction of reconstruction times, network bandwidth requirements and round-trip times as well as rendering costs in many situations. [ABSTRACT FROM AUTHOR]

We propose augmenting immersive telepresence by adding a virtual body, representing the user's own arm motions, as realized through a head-mounted display and a 360-degree camera. Previous research has shown the effectiveness of having a virtual body in simulated environments; however, research on whether seeing one's own virtual arms increases presence or preference for the user in an immersive telepresence setup is limited. We conducted a study where a host introduced a research lab while participants wore a head-mounted display which allowed them to be telepresent at the host's physical location via a 360-degree camera, either with or without a virtual body. We first conducted a pilot study of 20 participants, followed by a pre-registered 62 participant confirmatory study. Whereas the pilot study showed greater presence and preference when the virtual body was present, the confirmatory study failed to replicate these results, with only behavioral measures suggesting an increase in presence. After analyzing the qualitative data and modeling interactions, we suspect that the quality and style of the virtual arms, and the contrast between animation and video, led to individual differences in reactions to the virtual body which subsequently moderated feelings of presence. [ABSTRACT FROM AUTHOR]

Flying in virtual reality (VR) using standard handheld controllers can be cumbersome and contribute to unwanted side effects such as motion sickness and disorientation. This article investigates a novel hands-free flying interface—HeadJoystick, where the user moves their head similar to a joystick handle toward the target direction to control virtual translation velocity. The user sits on a regular office swivel chair and rotates it physically to control virtual rotation using 1:1 mapping. We evaluated short-term (Study 1) and extended usage effects through repeated usage (Study 2) of the HeadJoystick versus handheld interfaces in two within-subject studies, where participants flew through a sequence of increasingly difficult tunnels in the sky. Using the HeadJoystick instead of handheld interfaces improved both user experience and performance, in terms of accuracy, precision, ease of learning, ease of use, usability, long-term use, presence, immersion, sensation of self-motion, workload, and enjoyment in both studies. These findings demonstrate the benefits of using leaning-based interfaces for VR flying and potentially similar telepresence applications such as remote flight with quadcopter drones. From a theoretical perspective, we also show how leaning-based motion cueing interacts with full physical rotation to improve user experience and performance compared to the gamepad. [ABSTRACT FROM AUTHOR]

Developing, evaluating, and disseminating IT research prototypes for and with indigenous partners is both challenging and rewarding. In conjunction with our domain expert collaborators, Te Rau Aroha Marae [Bluff, Aotearoa/New Zealand (NZ)] and our academic colleagues at the Universities of Waikato and Canterbury, we are implementing a mixed reality telepresence system to connect a diasporic Māori community to their historical, cultural, and geographic mātauranga (knowledge). In this article, we describe our project, tea Presence, which is guided by the principles of partnership, participation, and protection. We describe the design and evaluation of the system developed, the collaborative process we undertook with Te Rau Aroha Marae, and our Māori academic colleagues and report on lessons learned along the way. [ABSTRACT FROM AUTHOR]

In this article, we present a novel system architecture for teleoperation of the networked robotic system (NRS), involving several slave robots (i.e., followers) interconnected through event-triggered local communication, based on one/several long-distance communication links between a human-operated master robot (i.e., leader) and one/multiple followers. The control objective of such human-aided teleoperation problem is to force the master robot to track the slave ones in a fixed time under the human supervision, and meanwhile to drive all the slave ones to track the master in a fixed time, thus allowing the human operator to flexibly and remotely control the NRS. Another important concern in this article is to improve the telepresence of such system, i.e., to reconstruct the interaction force between followers and the task environment in the side of the human operator. To this end, a novel force-reflecting hierarchical control framework, involving the (local) dissipativity layer, (estimate) event-triggered layer, fixed-time force observer and neuroadaptive observer, is proposed. Based on the tools of hierarchical control analysis and Lyapunov stability, we derive several sufficient conditions for guaranteeing the fixed-time stability of the closed-loop dynamics. Finally, several experiments are performed to illustrate the performance of the proposed results. [ABSTRACT FROM AUTHOR]

Rapidly developing technologies are realizing a 3D telepresence, in which geographically separated users can interact with each other through their virtual avatars. In this article, we present novel methods to determine the avatar’s position in an indoor space to preserve the semantics of the user’s position in a dissimilar indoor space with different space configurations and furniture layouts. To this end, we first perform a user survey on the preferred avatar placements for various indoor configurations and user placements, and identify a set of related attributes, including interpersonal relation, visual attention, pose, and spatial characteristics, and quantify these attributes with a set of features. By using the obtained dataset and identified features, we train a neural network that predicts the similarity between two placements. Next, we develop an avatar placement method that preserves the semantics of the placement of the remote user in a different space as much as possible. We show the effectiveness of our methods by implementing a prototype AR-based telepresence system and user evaluations. [ABSTRACT FROM AUTHOR]

A 3D Telepresence system allows users to interact with each other in a virtual, mixed, or augmented reality (VR, MR, AR) environment, creating a shared space for collaboration and communication. There are two main methods for representing users within these 3D environments. Users can be represented either as point cloud reconstruction-based avatars that resemble a physical user or as virtual character-based avatars controlled by tracking the users' body motion. This work compares both techniques to identify the differences between user representations and their fit in the reconstructed environments regarding the perceived presence, uncanny valley factors, and behavior impression. Our study uses an asymmetric VR/AR teleconsultation system that allows a remote user to join a local scene using VR. The local user observes the remote user with an AR head-mounted display, leading to facial occlusions in the 3D reconstruction. Participants perform a warm-up interaction task followed by a goal-directed collaborative puzzle task, pursuing a common goal. The local user was represented either as a point cloud reconstruction or as a virtual character-based avatar, in which case the point cloud reconstruction of the local user was masked. Our results show that the point cloud reconstruction-based avatar was superior to the virtual character avatar regarding perceived co-presence, social presence, behavioral impression, and humanness. Further, we found that the task type partly affected the perception. The point cloud reconstruction-based approach led to higher usability ratings, while objective performance measures showed no significant difference. We conclude that despite partly missing facial information, the point cloud-based reconstruction resulted in better conveyance of the user behavior and a more coherent fit into the simulation context. [ABSTRACT FROM AUTHOR]

The COVID-19 pandemic is affecting the mental and emotional health of patients, family members, health-care workers, and the community. Patients are affected by symptoms produced by prolonged hospitalization and isolation, and health-care workers are affected by the stress produced by the possibility of infection and rapid changes in the kinds of care they provide. Robotics technologies can contribute to addressing these stresses. This article describes the development and main characteristics of Pudu, a telepresence robot designed to provide safe communication between COVID-19 patients and health-care workers, particularly neurologists and intensive care psychologists. The article identifies the key aspects that enabled the development and deployment of the robot in just eight weeks during an intensive quarantine and describes how the robot is assisting health-care workers in providing mental and psychological health care. At present, two Pudu robots are in use in two Chilean hospitals. Although the city of Santiago is moving toward less-intensive levels of quarantine, our robots are still needed to help provide the required emotional support and mental/psychological health care to confined patients. [ABSTRACT FROM AUTHOR]

Developing countries represent about three-quarters of the world's population and have become the main driver for the global population and economic growth. Technology entrepreneurship plays an important role in sustainable development of vulnerable social groups and element. These efforts can channel technological benefits to underdeveloped regions. We discuss technology entrepreneurship in developing countries, with a focus on mobile health (mHealth) and telepresence robots. Specifically, we discuss how mHealth can help aged and disadvantaged developing country populations cope with normal life difficulties and challenges—but also in the extreme case and threat of a pandemic. Using past practices and lessons in telepresence robots, we propose FLEXTRA, a flexible robot mechanism to support instant video calls, smart medicine dispensation, and remote control. We also illustrate the applications of FLEXTRA in healthcare, e.g., fighting against COVID-19. [ABSTRACT FROM AUTHOR]

Improving the life quality of people with severe motor paralysis has a significant impact on restoring their functional independence to perform activities of daily living (ADL). Telepresence is a subfield of the robotic-assisted route, where human plays the role of an operator, sending high-level instructions to an assistive robot while receiving sensory feedback. However, for severely motor-impaired people, conventional interaction modalities may not be suitable due to their complete paralysis. Thus, designing alternative ways of interaction such as Brain-Computer Interfaces (BCI) is essential for a telepresence capability. We propose a novel framework that integrates a BCI system and a humanoid robot to develop a brain-controlled telepresence system with multimodal control features. In particular, the low-level control is executed by Programming by Demonstration (PbD) models, and the higher-level cognitive commands are produced by a BCI system to perform vital ADLs. The presented system is based on real-time decoding of attention-modulated neural responses elicited in the brain electroencephalographic signals and generating multiple control commands. As a result, the system allows a user to interact with a humanoid robot while receiving auditory and visual feedback from the robot's sensors. We validated our system across ten subjects in a realistic scenario. The experimental results show the feasibility of the approach in the design of a telepresence robot with high BCI decoding performances.

Spatial presence encompasses the user's ability to experience a sense of “being there”. While particular attention was given to assess spatial presence in real and virtual environments, few have been interested in measuring it in telepresence situations. To bridge this gap, the present work introduces a study that compares the execution of a task in three conditions: a real physical environment, a remote environment via a telepresence system, and a virtual simulation of the real environment. Following a within-subject design, 27 participants performed a navigation task consisting in following a route while avoiding obstacles. Spatial presence and five related factors (affordance, enjoyment, attention allocation, reality, and cybersickness) were evaluated using a presence questionnaire. In addition, performance measures were gathered regarding environment recollection and task execution. The evaluation also included a behavioral metric measured by obstacle avoidance distance extracted from participants' trajectories. Results indicated a higher presence in the real environment, along with the best performance measures. No difference was found in spatial presence between the remote and the virtual conditions, although a higher degree of affordance and enjoyment was attributed to the virtual environment, and a higher degree of reality was attributed to the remote environment. The number of collisions was found to be lower in the remote condition compared to the virtual condition. Similarly, the avoidance distance was also bigger (and almost similar) in the real and the remote environments compared to the virtual environment indicating a greater caution of participants. These cues highlight that the behavior of participants in the remote condition was closer to their behavior in the real situation than it was in the virtual condition. Furthermore, positive correlations were found between the reality factor and two of the three performance measures, as well as with the behavioral metric. This suggests that the degree of physical existence of the space in which participants operate can influence their performance and behavior. [ABSTRACT FROM AUTHOR]

Tactile sensors have been incorporated into robots to help in many tasks, such as preventing damage to humans in collaborative works or assistance or in dexterous manipulation. Proximity sensors are also common in robotics and are proposed to implement pretouch in multimodal realizations. Many tactile sensors are based on off-the-shelf force-sensing resistors (FSRs). The parasitic capacitance associated with the electrodes of the FSR can be exploited to measure proximity. Moreover, it is possible to implement both force and proximity signal acquisition with a single chip. This article presents a multimodal proximity and tactile patch based on commercial FSR plus a programmable system on chip (PSoC). The ability to measure proximity is achieved without adding any extra element. Discussions about key design issues as well as results that show the performance of the sensor patch are presented. As proximity sensing array, the range is in the order of the size of the electrodes, and it depends on the properties of the object nearby, as expected. The best performance is observed for conductive objects. Therefore, this approach is especially interesting in human–robot interaction tasks, such as shaping robotic hands or grippers around the human body. Furthermore, dielectric objects are also detected, so it can also be used to implement pretouch in manipulation tasks. [ABSTRACT FROM AUTHOR]

Telecollaboration involves the teleportation of a remote collaborator to another real-world environment where their partner is located. The fidelity of the environment plays an important role for allowing corresponding spatial references in remote collaboration. We present a novel asymmetric platform, Augmented Virtual Teleportation (AVT), which provides high-fidelity telepresence of a remote VR user (VR-Traveler) into a real-world collaboration space to interact with a local AR user (AR-Host). AVT uses a 360° video camera (360-camera) that captures and live-streams the omni-directional scenes over a network. The remote VR-Traveler watching the video in a VR headset experiences live presence and co-presence in the real-world collaboration space. The VR-Traveler's movements are captured and transmitted to a 3D avatar overlaid onto the 360-camera which can be seen in the AR-Host's display. The visual and audio cues for each collaborator are synchronized in the Mixed Reality Collaboration space (MRC-space), where they can interactively edit virtual objects and collaborate in the real environment using the real objects as a reference. High fidelity, real-time rendering of virtual objects and seamless blending into the real scene allows for unique mixed reality use-case scenarios. Our working prototype has been tested with a user study to evaluate spatial presence, co-presence, and user satisfaction during telecollaboration. Possible applications of AVT are identified and proposed to guide future usage. [ABSTRACT FROM AUTHOR]

Historically, robots first found application in factories and plants. Until recently, the most noticeable examples of robot systems directly sold to the consumer were limited to edutainment systems (e.g., NAO [1]), automated chore robots [26], and social telepresence platforms [27]. Initially, telepresence robots consisted of a mobile base with an interactive screen. Today, following a trend of anthropomorphization of technology, human-like upper bodies have begun to replace those simple screens (e.g., Pepper [2] and R1 [3]) and share the same social communication modalities of humans, e.g., body posture, gestures, gaze direction, and facial expressions. Unfortunately, social robots are mostly designed to speak and make gestures and have limited capabilities when it comes to physically interacting with people and their surrounding environments. [ABSTRACT FROM AUTHOR]

Mobile manipulation robots have great potential for roles in support of rescuers on disaster-response missions. Robots can operate in places too dangerous for humans and therefore can assist in accomplishing hazardous tasks while their human operators work at a safe distance. We developed a disaster-response system that consists of the highly flexible Centauro robot and suitable control interfaces, including an immersive telepresence suit and support-operator controls offering different levels of autonomy. [ABSTRACT FROM AUTHOR]

We propose an algorithm for generating an unstructured lumigraph in real-time from an image stream. This problem has important applications in mixed reality, such as telepresence, interior design or as-built documentation. Unlike conventional texture optimization in structure from motion, our method must choose views from the input stream in a strictly incremental manner, since only a small number of views can be stored or transmitted. This requires formulating an online variant of the well-known view-planning problem, which must take into account what parts of the scene have already been seen and how the lumigraph sample distribution could improve in the future. We address this highly unconstrained problem by regularizing the scene structure using a regular grid structure. Upon the grid structure, we define a coverage metric describing how well the lumigraph samples cover the grid in terms of spatial and angular resolution, and we greedily keep incoming views if they improve the coverage. We evaluate the performance of our algorithm quantitatively and qualitatively on a variety of synthetic and real scenes, and demonstrate visually appealing results obtained at real-time frame rates (in the range of 3Hz-100Hz per incoming image, depending on configuration). [ABSTRACT FROM AUTHOR]

Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages. [ABSTRACT FROM AUTHOR]

The mobility and ubiquity of mobile head-mounted displays make them a promising platform for telepresence research as they allow for spontaneous and remote use cases not possible with stationary hardware. In this work we present a system that provides immersive telepresence and remote collaboration on mobile and wearable devices by building a live spherical panoramic representation of a user's environment that can be viewed in real time by a remote user who can independently choose the viewing direction. The remote user can then interact with this environment as if they were actually there through intuitive gesture-based interaction. Each user can obtain independent views within this environment by rotating their device, and their current field of view is shared to allow for simple coordination of viewpoints. We present several different approaches to create this shared live environment and discuss their implementation details, individual challenges, and performance on modern mobile hardware; by doing so we provide key insights into the design and implementation of next generation mobile telepresence systems, guiding future research in this domain. The results of a preliminary user study confirm the ability of our system to induce the desired sense of presence in its users. [ABSTRACT FROM AUTHOR]

In this paper, we discuss and speculate about the concept of the Tactile Robot connected with human operators via smart wearables as an essential multimodal embodiment of the coming Tactile Internet. The Tactile Robot, succeeding the recently introduced kinesthetic soft robot, is the upcoming next step in the evolution of rapidly developing robotic platforms that are capable of sensitive physical interaction with their environment. From the combination of rich tactile feedback with state-of-the-art robotics, technology, and algorithms emerge the potential of a meaningful and immersive connection to human operators via the vastly progressing smart wearables and virtual reality/augmented reality devices, effectively creating real-world avatars. Moreover, the Tactile Internet is believed to make it possible to create avatar collectives spanning different application domains and, therefore, cover heterogeneous robotic platforms. We hypothesize that this development will enable us to seamlessly interact with heterogeneous systems such as industrial assembly lines, service robots, automated medical units, or even deep sea and space exploration units. This new paradigm of an immersive coexistence between humans and robots builds on numerous technological advances in robotics, multimodal teleoperation, wearable technology, distributed computing, or network technology, for example. However, such a vision obviously poses major challenges in multiple areas that are still to be overcome. In this paper, we discuss the potentials and enabling technologies together with foreseeable application domains in the framework of the Tactile Internet. Furthermore, we address major challenges and hypothesize about potential solutions. [ABSTRACT FROM AUTHOR]

The Tactile Internet will enable users to physically explore remote environments and to make their skills available across distances. An important technological aspect in this context is the acquisition, compression, transmission, and display of haptic information. In this paper, we present the fundamentals and state of the art in haptic codec design for the Tactile Internet. The discussion covers both kinesthetic data reduction and tactile signal compression approaches. We put a special focus on how limitations of the human haptic perception system can be exploited for efficient perceptual coding of kinesthetic and tactile information. Further aspects addressed in this paper are the multiplexing of audio and video with haptic information and the quality evaluation of haptic communication solutions. Finally, we describe the current status of the ongoing IEEE standardization activity P1918.1.1 which has the ambition to standardize the first set of codecs for kinesthetic and tactile information exchange across communication networks. [ABSTRACT FROM AUTHOR]

This paper describes a novel web-based platform promoting real-time advanced teleconsultation services on medical imaging. Principles of heterogeneous workflow management systems and state-of-the-art technologies such as the microservices architectural pattern, peer-to-peer networking, and the single-page application concept are combined to build a scalable and extensible platform to aid collaboration among geographically distributed healthcare professionals. The real-time communication capabilities are based on the webRTC protocol to enable direct communication among clients. This paper discusses the conceptual and technical details of the system, emphasizing on its innovative elements. [ABSTRACT FROM AUTHOR]

The advancements in Mixed Reality (MR), Unmanned Aerial Vehicle, and multi-scale collaborative virtual environments have led to new interface opportunities for remote collaboration. This paper explores a novel concept of flying telepresence for multi-scale mixed reality remote collaboration. This work could enable remote collaboration at a larger scale such as building construction. We conducted a user study with three experiments. The first experiment compared two interfaces, static and dynamic IPD, on simulator sickness and body size perception. The second experiment tested the user perception of a virtual object size under three levels of IPD and movement gain manipulation with a fixed eye height in a virtual environment having reduced or rich visual cues. Our last experiment investigated the participant's body size perception for two levels of manipulation of the IPDs and heights using stereo video footage to simulate a flying telepresence experience. The studies found that manipulating IPDs and eye height influenced the user's size perception. We present our findings and share the recommendations for designing a multi-scale MR flying telepresence interface. [ABSTRACT FROM AUTHOR]

We propose a new approach for 3D reconstruction of dynamic indoor and outdoor scenes in everyday environments, leveraging only cameras worn by a user. This approach allows 3D reconstruction of experiences at any location and virtual tours from anywhere. The key innovation of the proposed ego-centric reconstruction system is to capture the wearer's body pose and facial expression from near-body views, e.g. cameras on the user's glasses, and to capture the surrounding environment using outward-facing views. The main challenge of the ego-centric reconstruction, however, is the poor coverage of the near-body views – that is, the user's body and face are observed from vantage points that are convenient for wear but inconvenient for capture. To overcome these challenges, we propose a parametric-model-based approach to user motion estimation. This approach utilizes convolutional neural networks (CNNs) for near-view body pose estimation, and we introduce a CNN-based approach for facial expression estimation that combines audio and video. For each time-point during capture, the intermediate model-based reconstructions from these systems are used to re-target a high-fidelity pre-scanned model of the user. We demonstrate that the proposed self-sufficient, head-worn capture system is capable of reconstructing the wearer's movements and their surrounding environment in both indoor and outdoor situations without any additional views. As a proof of concept, we show how the resulting 3D-plus-time reconstruction can be immersively experienced within a virtual reality system (e.g., the HTC Vive). We expect that the size of the proposed egocentric capture-and-reconstruction system will eventually be reduced to fit within future AR glasses, and will be widely useful for immersive 3D telepresence, virtual tours, and general use-anywhere 3D content creation. [ABSTRACT FROM AUTHOR]

Tele-immersive systems development is always driven as well as restricted by the available immersive technology. Hence, existing such systems are described mainly from a technological point of view; their conceptual description is usually limited to the description of a scenario that is implementable with or circumvents the limitations of the chosen technology. This focus on technology makes it difficult to compare systems’ concepts; moreover, it has led to different views on tele-immersion in different fields, such as remotely controlled robots, immersive video conferencing, and tele-collaboration. In this work, we give a general, structured principle to describe the conceptual part of any tele-immersion system. This principle naturally unifies the different views on tele-immersion. Our idea is based on the insight that, in order to be general, immersion must be described separately for each direction of communication. We characterize communication between locations using a graph; for each directed edge of this graph, we describe immersion as operations on volumes. Using this principle, we define a typology, which enables the comparison and enumeration of tele-immersion concepts. We apply this typology to survey the concepts of existing tele-immersion systems and thereby demonstrate how three well-known tele-immersive scenarios—Marvin Minsky’s tele-operated robot, the Office of the Future, and the asymmetric Beaming scenario—integrate naturally. We show how the general principle can be utilized conveniently to grasp conceptual ideas in tele-immersion, such as direct interaction, locational presence, spatial consistency, symmetries, and self-inclusion. [ABSTRACT FROM AUTHOR]

Haptic perception is an important modality in reinforcing the presence of virtual or remote targets, but providing haptic feedback of dynamic environments still remains a challenging task. We address the issue of haptic telepresence systems by improving and integrating two independent approaches: real-time haptic rendering of unstructured spatial data and collaborative interaction with a remote partner. Contact with physical constraints is directly estimated from streaming point-cloud data without surface reconstruction, and haptic guidance cues that restrict or promote the users motion can be provided by both predefined triggers and gesture-based input from a helper. Through a user study with a proof-of-concept prototype, the authors show that the proposed approach significantly improves the performance of remote exploration tasks while enabling stable haptic interaction with real-world spatial data. [ABSTRACT FROM AUTHOR]

In this letter, we introduce several algorithms for geometry inpainting of three-dimensional (3-D) point clouds with large holes. The algorithms are exemplar based. Hole filling is performed iteratively using templates near the hole boundary to find the best matching regions elsewhere in the cloud, from where existing points are transferred to the hole. We propose two improvements over the previous work on exemplar-based hole filling. The first one is adaptive template size selection in each iteration, which simultaneously leads to higher accuracy and lower execution time. The second improvement is a nonrigid transformation to better align the candidate set of points with the template before the point transfer, which leads to even higher accuracy. We demonstrate the algorithm's ability to fill holes that are difficult or impossible to fill by existing methods. [ABSTRACT FROM AUTHOR]

Telepresence systems have the potential to overcome limits and distance constraints of the real-world by enabling people to remotely visit and interact with each other. However, current telepresence systems usually lack natural ways of supporting interaction and exploration of remote environments (REs). In particular, single webcams for capturing the RE provide only a limited illusion of spatial presence, and movement control of mobile platforms in today's telepresence systems are often restricted to simple interaction devices. One of the main challenges of telepresence systems is to allow users to explore a RE in an immersive, intuitive and natural way, e.g., by real walking in the user's local environment (LE), and thus controlling motions of the robot platform in the RE. However, the LE in which the user's motions are tracked usually provides a much smaller interaction space than the RE. In this context, redirected walking (RDW) is a very suitable approach to solve this problem. However, so far there is no previous work, which explored if and how RDW can be used in video-based 360° telepresence systems. In this article, we conducted two psychophysical experiments in which we have quantified how much humans can be unknowingly redirected on virtual paths in the RE, which are different from the physical paths that they actually walk in the LE. Experiment 1 introduces a discrimination task between local and remote translations, and in Experiment 2 we analyzed the discrimination between local and remote rotations. In Experiment 1 participants performed straightforward translations in the LE that were mapped to straightforward translations in the RE shown as 360° videos, which were manipulated by different gains. Then, participants had to estimate if the remotely perceived translation was faster or slower than the actual physically performed translation. Similarly, in Experiment 2 participants performed rotations in the LE that were mapped to the virtual rotations in a 360° video-based RE to which we applied different gains. Again, participants had to estimate whether the remotely perceived rotation was smaller or larger than the actual physically performed rotation. Our results show that participants are not able to reliably discriminate the difference between physical motion in the LE and the virtual motion from the 360° video RE when virtual translations are down-scaled by 5.8% and up-scaled by 9.7%, and virtual rotations are about 12.3% less or 9.2% more than the corresponding physical rotations in the LE. [ABSTRACT FROM PUBLISHER]

We show how a combination of virtual reality and robotics can be used to beam a physical representation of a person to a distant location, and describe an application of this system in the context of journalism. Full body motion capture data of a person is streamed and mapped in real time, onto the limbs of a humanoid robot present at the remote location. A pair of cameras in the robots eyes stream stereoscopic video back to the HMD worn by the visitor, and a two-way audio connection allows the visitor to talk to people in the remote destination. By fusing the multisensory data of the visitor with the robot, the visitors consciousness is transformed to the robots body. This system was used by a journalist to interview a neuroscientist and a chef 900 miles distant, about food for the brain, resulting in an article published in the popular press. [ABSTRACT FROM PUBLISHER]

In this paper, a novel accumulate-repeat-accumulate-check-accumulate (ARACA) code is proposed as a subclass of protograph-based low-density parity-check (LDPC) codes. The key feature of the proposed ARACA code is represented by the outer connection doping in the protograph. This feature can provide the linear minimum distance growth (LMDG) property at a good iterative decoding threshold while maintaining an efficient encoder structure. The effect of the outer connection doping on the typical minimum distance, the iterative decoding threshold, and the LMDG property is discussed and analyzed by comparing case examples and using the asymptotic protograph ensemble weight enumerator. Some good ARACA code protographs are provided for a wide range of code rates. In addition, an efficient and universal encoding procedure and the corresponding encoder structure are provided for them. The performance of the proposed ARACA code is evaluated and compared with well-known good LDPC codes. The simulation results confirm the superiority of the proposed ARACA codes in terms of encoding complexity and frame error rate performance, especially at low-rates in an ultra-reliable regime. [ABSTRACT FROM AUTHOR]

Susanne Boll recounts her experience as a telepresence robot attending this year’s ACM Conference on Human Factors in Computing Systems (CHI), illustrating the fusion of the state of the art in robotics, long-distance interaction, and shared remote-audio-visual experiences. As a researcher in human computer interaction, yet with an international team facing restricted travel owing to the newly imposed US travel ban, she opted for remote participation at CHI 2017. Furthermore, as a member of the multimedia community, she wanted to explore first-hand how multimedia is at work in the field of telepresence. [ABSTRACT FROM PUBLISHER]

In this paper, a novel soft inflatable arm is proposed for telepresence robots. The new proposed structure of the arm is achieved by a very common and low-cost inflatable material and it is very light, weighing only about 50 g. However, it can realize agile movement by driving six tiny cables installed in the shoulder and elbow joints. The soft inflatable arm can work by pumping air at a very low pressure ($7.32 \pm 3.45$ kPa) and allows direct and soft human contact without any external force sensors. This paper proposes joint compressed models, joint kinematic models, and kinematic models for the whole inflatable robot arm. These models can be easily applied to multijoint arms. In addition, a new redundancy resolution method is also developed for the inflatable arm, which makes it easier to control resolution and is less complex than other traditional approaches. Numerous experiments have been conducted, including performances of accuracy, repeatability, motion trajectories, human-safe interaction, and remote interaction. Results are satisfactory and validate the expected performance of the proposed robotic arm. [ABSTRACT FROM PUBLISHER]

The idea of being present in a remote location has inspired researchers to develop robotic devices, that make humans to experience the feeling of telepresence. These devices need of multiple sensory feedback to provide a more realistic telepresence experience. In this paper, we develop a wearable interface for immersion and telepresence that provides to human with the capability of both to receive multisensory feedback from vision, touch, and audio, and to remotely control a robot platform. Multimodal feedback from a remote environment is based on the integration of sensor technologies coupled to the sensory system of the robot platform. Remote control of the robot is achieved by a modularised architecture, which allows to visually exploring the remote environment. We validated our paper with multiple experiments where participants, located at different venues, were able to successfully control the robot platform while visually exploring, touching, and listening a remote environment. In our experiments, we used two different robotic platforms: 1) the iCub humanoid robot and 2) the Pioneer LX mobile robot. These experiments show that our wearable interface is comfortable, easy to use, and adaptable to different robotic platforms. Furthermore, we observed that our approach allows humans to experience a vivid feeling of being present in a remote environment. [ABSTRACT FROM PUBLISHER]

This paper presents a mixed reality (MR) system that results from the integration of a telepresence system and an application to improve collaborative space exploration. The system combines free viewpoint video with immersive projection technology to support nonverbal communication (NVC), including eye gaze, interpersonal distance, and facial expression. Importantly, these features can be interpreted together as people move around the simulation, maintaining a natural social distance. The application is a simulation of Mars, within which the collaborators must come to agreement over; for example, where the Rover should land and go. The first contribution is the creation of an MR system supporting contextualization of NVC. Two technological contributions are prototyping a technique to subtract a person from a background that may contain physical objects and/or moving images and a lightweight texturing method for multiview rendering, which provides balance in terms of visual and temporal quality. A practical contribution is the demonstration of pragmatic approaches to sharing space between display systems of distinct levels of immersion. A research tool contribution is a system that allows comparison of conventional authored and video-based reconstructed avatars, within an environment that encourages exploration and social interaction. Aspects of system quality, including the communication of facial expression and end-to-end latency are reported. [ABSTRACT FROM PUBLISHER]

Sharing one's own immersive experience over the Internet is one of the ultimate goals of telepresence technology. In this paper, we present JackIn Head, a visual telepresence system featuring an omnidirectional wearable camera with image motion stabilization. Spherical omnidirectional video footage taken around the head of a local user is stabilized and then broadcast to others, allowing remote users to explore the immersive visual environment independently of the local user's head direction. We describe the system design of JackIn Head and report the evaluation results of real-time image stabilization and alleviation of cybersickness. Then, through an exploratory observation study, we investigate how individuals can remotely interact, communicate with, and assist each other with our system. We report our observation and analysis of inter-personal communication, demonstrating the effectiveness of our system in augmenting remote collaboration. [ABSTRACT FROM PUBLISHER]

This paper develops an event-related potential (ERP)-based adaptive model for the control of humanoid robot movements in an environment cluttered with obstacles based on live video feedback. This model adaptively determines the repetition number according to an individual’s mental state to speed up the robot control cycle. N200 and P300 potential features increase in the frontal and occipital areas when using robot images as visual stimuli, so it is able to effectively recognize target visual stimuli by processing Fisher’s linear discriminant analysis (FLDA) and to identify a subject’s intention by using support vector machine (SVM), in parallel. The offline evaluations show that, compared with a nonadaptive model, the adaptive model increases the accuracy rate from 88.8% to 92.9%, a change of 4.1%, and the information transfer rate (ITR) from 41.3 to 46.3 bits/min, a change of 5.0 bits/min. Eight subjects participated in telepresence controlling a NAO humanoid robot to move in an office environment cluttered with obstacles. The successful maneuvers demonstrate that the brain-controlled humanoid robot can be applied for surveillance and exploration in unknown environments based on live video feedback, which are evaluated by using new metrics for the performance of the brain–robot interaction (BRI) system. [ABSTRACT FROM AUTHOR]

This paper develops a virtual platform of an underwater manipulator mounted on a submersible vehicle via the three-dimensional simulator “Webots” for teleoperation through a replica master arm. The graphical, kinematic, and dynamic models of the manipulator refer to a master–slave servo hydraulic manipulator with seven functions, consisting of six degrees of freedom and a parallel gripper, while the “Jiaolong” deep-manned submersible vehicle, operating below the sea surface down to 7000 m, is chosen as the underwater manipulator carrier. This study uses the virtual platform for training an operator to telepresence control the virtual manipulator to complete basic tasks in subsea environments. When training the operator, one has to consider uncertain external disturbances and the visual impacts that stem from subsea environments. In order to demonstrate the feasibility and effectiveness of the virtual platform, one designs two typical underwater operational tasks: grasping a marine organism sample and reaching at a given position. This paper presents the comparative studies: 1) the performances demonstrated by remotely controlling the virtual manipulator and the real manipulator; 2) the operating performances delivered by three operators before and after training when using the platform. [ABSTRACT FROM AUTHOR]

In augmented reality (AR) applications, a virtual avatar serves as a useful medium to represent a human in a different place. This paper deals with the problem of retargeting a human motion to an avatar. In particular, we present a novel method that retargets a human motion with respect to an object to that of an avatar with respect to a different object of a similar shape. To achieve this, we developed a spatial map that defines the correspondences between any points in the 3D spaces around the respective objects. The key advantage of the spatial map is that it identifies the desired locations of the avatar's body parts for any input motion of a human. Once the spatial map is created offline, the motion retargeting can be performed in real-time. The retargeted motion preserves important features of the original motion such as the human pose and the spatial relation with the object. We report the results of a number of experiments that demonstrate the effectiveness of the proposed method. [ABSTRACT FROM PUBLISHER]

Telecommunication and remote-controlled operations are becoming increasingly common in our daily lives. While performing these operations, ideally users would feel as if they were actually present at the remote sites. However, the present commercially available telecommunication and telepresence systems do not provide the sensation of self-presence or self-existence, and hence, users do not get the feeling of being spatially present or that they are directly performing spatial tasks, rather than simply controlling them remotely. This article describes the TELESAR V telexistence master-slave system that enables a human user to feel present in a remote environment. TELESAR V can transmit not only visual and auditory sensations, but also haptic sensations, which are conveyed using the principle of haptic primary colors. [ABSTRACT FROM AUTHOR]

Telepresence systems use 3D techniques to create a more natural human-centered communication over long distances. This work concentrates on the analysis of latency in telepresence systems where acquisition and rendering are distributed. Keeping latency low is important to immerse users in the virtual environment. To better understand latency problems and to identify the source of such latency, we focus on the decomposition of system latency into sub-latencies. We contribute a model of latency and show how it can be used to estimate latencies in a complex telepresence dataflow network. To compare the estimates with real latencies in our prototype, we modify two common latency measurement methods. This presented methodology enables the developer to optimize the design, find implementation issues and gain deeper knowledge about specific sources of latency. [ABSTRACT FROM AUTHOR]

This paper presents an important step forward towards increasing the independence of people with severe motor disabilities, by using brain–computer interfaces to harness the power of the Internet of Things. We analyze the stability of brain signals as end-users with motor disabilities progress from performing simple standard on-screen training tasks to interacting with real devices in the real world. Furthermore, we demonstrate how the concept of shared control—which interprets the user's commands in context—empowers users to perform rather complex tasks without a high workload. We present the results of nine end-users with motor disabilities who were able to complete navigation tasks with a telepresence robot successfully in a remote environment (in some cases in a different country) that they had never previously visited. Moreover, these end-users achieved similar levels of performance to a control group of 10 healthy users who were already familiar with the environment. [ABSTRACT FROM PUBLISHER]