Search

In avatar-mediated communication, the face-to-face interlocutor must sense the operator’s presence and emotions via an avatar. Although androids resembling humans have been developed to convey presence through appearance and movement, few studies have prioritized deepening the communication experience for operators and interlocutors using android robots as avatars. To address this gap, we introduced the “cybernetic avatar ‘Yui,’” featuring a human-like head unit with 28 actuation points, capable of expressing gaze, facial emotions, and speech-related mouth movements. Through an eye-tracking unit in a head-mounted display (HMD) and the degrees of freedom in both eyes of Yui, operators can naturally control the avatar’s gaze. Additionally, microphones embedded in Yui’s ears allowed operators to hear surrounding sounds in three dimensions, enabling them to discern the direction of calls based solely on auditory information. The HMD’s face-tracking unit synchronizes the facial movements of the avatar with those of the operator. This immersive interface, coupled with Yui’s human-like appearance, enabled real-time emotional transmission and communication, thus enhancing the sense of presence for both parties. Our experiments demonstrate Yui’s facial expression capabilities and validate the efficacy of the system through teleoperation trials, suggesting potential advancements in avatar technology.

The construction industry currently is confronted with significant challenges, stemming from a decreasing working population coupled with an aging workforce. Accordingly, the development of teleoperated hydraulic excavators is being investigated to optimize operations. The ability to switch between multiple machines during operations is expected to reduce transportation and labor costs, thereby increasing productivity. However, work efficiency may be severely affected when operators switch between different classes of machines, such as excavators with varying sizes and capabilities. Each machine class operates differently, and immediate adjustments to the dynamic characteristics may be challenging. Previous research has demonstrated that operators of teleoperated hydraulic excavators struggle to perceive load, as they rely on visual information from cameras instead of tactile feedback. Consequently, loading information, particularly related to the attachment’s maximum exertion force, is difficult to estimate. This study proposes a load index called “attachment joint-load ratio” to mitigate decreased work efficiency when switching machines. This index considers the attachment’s maximum exertion force and attempts to provide operators with intuitive feedback about load conditions. This study focuses on excavation work and compares the operating characteristics of a standard hydraulic excavator with a conventional load feedback method. The objective is to ascertain whether feedback regarding the attachment joint load ratio can mitigate the effects of transitioning between machine classes on work efficiency. Using a simulator that mimics a teleoperated hydraulic excavator based on the proposed feedback method, we conducted experiments and verified the effectiveness of the proposed method.

Robotic avatar technology has the potential to impact the future of human connectivity, transporting the sense of a human’s presence to a remote location anywhere and in real-time. In this regard, the recent ANA Avatar XPRIZE challenge fostered the development of the ultimate generation of non-autonomous robotic avatar designs. This paper is devoted to introducing our system proposal, allowing intuitive motion-based control and multi-modal feedback. The teleoperated robot endorses an anthropomorphic upper body with dual arms and dexterous hands for fine manipulation of even small objects, as well as an omnidirectional platform for improved and safe locomotion. Special focus is pointed at the crucial challenge of providing self-body perception to the operator, in particular regarding control of the arms and rendering of haptic sensations. To this end, we propose an upper-limb exoskeleton and a teleoperation architecture allowing retargeting of the operator’s skills and receiving tactile and kinesthetic force feedback with realistic and informative perception of the own’s arms. Lastly, the platform was validated during laboratory and challenge missions mimicking several social, cooperative, and cultural scenarios from which we report on the lesson learned and future improvements.

Pseudo-haptic techniques are becoming increasingly popular in human-computer interaction. They replicate haptic sensations by leveraging primarily visual feedback rather than mechanical actuators. These techniques bridge the gap between the real and virtual worlds by exploring the brain’s ability to integrate visual and haptic information. One of the many advantages of pseudo-haptic techniques is that they are cost-effective, portable, and flexible. They eliminate the need for direct attachment of haptic devices to the body, which can be heavy and large and require a lot of power and maintenance. Recent research has focused on applying these techniques to extended reality and mid-air interactions. To better understand the potential of pseudo-haptic techniques, the authors developed a novel taxonomy encompassing tactile feedback, kinesthetic feedback, and combined categories in multimodal approaches, ground not covered by previous surveys. This survey highlights multimodal strategies and potential avenues for future studies, particularly regarding integrating these techniques into extended reality and collaborative virtual environments.

Teleoperated robotics in recent years has proven to be valuable support in EOD tasks; a remarkable improvement in the systems that control these robots has been the Natural User Interfaces (NUI); however, the research that implements this type of system does not focus on the stability of the robotic arm movements, necessary for this type of applications due to the danger of working with explosives. In this paper, we propose the implementation of an Optimal Signal Processing for a NUI interface based on the Leap Motion (LM) controller. The main objective of this research is to correctly identify the intentional movements of the operator, achieve high stability of the robotic gripper and suppress the physiological tremors from the hand of the operator, considering not to increase the mental workload and not decrease the usability of the system. The signal processing proposed in this paper is composed of three filtering algorithms: Kalman, FIR, and moving average with a threshold. In addition, the obtained results are compared with the most representative processing of recent research using LM for robotic arm control. To evaluate and validate the proposed signal processing, a target path tracking test, a stability analysis of the robotic gripper, and a performance analysis in the execution of Pick and Place tasks, NASA-TLX and SUS questionnaires are developed. Finally, the proposed Optimal Signal processing is implemented in the DOBOT-MAGICIAN and tested by police officers of the EOD Unit-Arequipa (UDEX-AQP); the results indicate a reduction of the average Vibration of 31.61% and the Target Path Tracking error of 67.57%.

This paper presents a controller for step-by-step teleoperation of bipedal robots, in which the user commands the robot’s foot motions in a step-by-step manner through a pair of handheld 3-dimensional pointing devices. The proposed controller is intended to allow users to precisely manipulate the swing foot motions to traverse rough terrains by avoiding obstacles. It aims to realize quick response to user commands and stable automatic balancing even under erroneous user commands. The main components of the proposed controller are a COM (center-of-mass) shifter and a body rotator, which are simple sensory feedback controllers that do not involve first-in first-output (FIFO) buffers, time-series generation, or online optimization. The COM shifter is a controller to produce a COM motion according to a reference zero moment point (ZMP). The body rotator complements the COM shifter to produce an appropriate angular momentum rate to enhance the regulation of ZMP. The proposed controller is validated in our interactive/realtime simulation environment.

Remote teleoperation has shown significant advancements since the first teleoperation system was proposed by Goertz in the 1940s. In recent years, the research on shared control methodologies in which the robot assists the operators in accomplishing the desired tasks has gained extensive attention. One such important task in teleoperation is object grasping. In this paper, we propose a shared control framework to enhance the teleoperated grasping performance. The proposed framework is built upon a virtual reality device-based direct teleoperation system. In this framework, a template matching-based object point cloud compensation is introduced for multi-angle grasping pose generation. Then, the feasible grasping candidates are selected considering joint constraints-aware manipulability. Finally, the grasping assistance is achieved by trajectory blending with dynamic authority adjustment. To validate the performance of the proposed framework, we carried out experimental evaluations. The output results indicate improved grasping performance in terms of reduced task completion time, linear trajectory, and workload.

Biopsies are one of the most common ways to diagnose and make a prognosis of prostate cancer. Transrectal and Transperineal approaches are used to access the prostate for biopsy. Advancements in Magnetic Resonance Imaging (MRI) offer a clear visualization of soft tissues and improved volumetric clarity. Real time MRI guidance can significantly change the paradigm of soft tissue biopsy. Proposed is a device with the ability to collect the tissue samples taken by a biopsy needle during an MRI guided Prostate biopsy and store them temporarily. This could help avoid interruption during the process and make the entire process teleoperated. The proposed design consists of a double layer of paper, rolling over a set of pulleys. Paper collects and stores the core tissue sample. Experiments have been performed to optimize paper tension, speed, and size. Tissue collection performance improvement is noticed with optimum paper tension, speed, and size. A test has also been performed in an MRI environment to check MR compatibility of the device. The device is MRI conditional and demonstrates a maximum of 13.1% SNR reduction in MR images. The proposed device exhibits the potential to be integrated with an MRI safe or conditional robotic manipulator for real time MRI guided Prostate biopsy.

In this study, we developed a novel method to determine the optimal viewpoint from which an operator could realize faster and more accurate robot teleoperation using reinforcement learning. The reinforcement learning model was trained using images obtained from several candidate viewpoints from scratch, and the viewpoint at which the model achieved the highest rewards was considered the optimal viewpoint. The target robot, task, and environment were modeled using computer simulations and the candidate viewpoint images were obtained using those simulations. We employed the world model as our reinforcement learning model to maximize rewards in the reaching task of a robot arm. The reward function was designed to encourage the robot arm to reach the target position both quickly and accurately. The experimental results validated the choice of the world model as the reinforcement learning model. Moreover, subject experiments wherein subjects operated a robot arm remotely to reach the target position were conducted. The experiments produced results that strongly aligned with the performance obtained through computer simulations, indicating that the proposed method is capable of selecting the optimal viewpoint without handcrafted design and subject experiments.

Supernumerary effectors have been proposed to enable users to perform tasks alone that normally require assistance from a partner. While various supernumerary robotic limbs have been developed in the last decade, the capability of users to operate them effectively has not yet been proven. Here we tested whether users (i) can complete a task that requires simultaneous and fine control of three effectors, and (ii) can be trained to do so with similar or superior performance as through collaboration with a human partner. As in previous studies, initial augmented capability was less than that of working with a partner. However, one hour of dedicated solo trimanual training across three days significantly increased task performance, so that participants became able to perform trimanual control alone as well as or better than they could with a new partner. This shows the viability of augmentation systems for applications such as in robotic surgery or industrial assembly, which can be further validated on real tasks with physical systems.

Significant advancements in contemporary telemedicine applications enforce the demand for effective and intuitive telerehabilitation tools. Telerehabilitation can minimize the distance, travel burden, and costs between rehabilitative patients and therapists. This research introduces an interactive novel telerehabilitation system that integrates the Industrial Internet of Things (IIoT) platform with a robotic manipulator named xARm-5, aiming to deliver rehabilitation therapies to individuals with upper limb dysfunctions. With the proposed system, a therapist can provide upper limb rehab exercises remotely using an augmented reality (AR) user interface (UI) developed using Vuforia Studio, which transmits bidirectional data through the IIoT platform. The proposed system has a stable communication architecture and low teleoperation latency. Experimental results revealed that with the developed telerehabilitation framework, the xArm-5 could be teleoperated from the developed AR platform and/or use a joystick to provide standard upper limb rehab exercises. Besides, with the designed AR-based UI, a therapist can monitor rehab/robot trajectories along with the AR digital twin of the robot, ensuring that the robot is providing passive therapy for shoulder and elbow movements.

In recent years, the use of remote operation has been proposed as a bridge towards driverless mobility by providing human assistance remotely when an automated driving system finds a situation that is ambiguous and requires input from a remote operator. The remote operation of road vehicles has also been proposed as a way to enable drivers to operate vehicles from safer and more comfortable locations. While commercial solutions for remote operation exist, remaining challenges are being tackled by the research community, who is continuously testing and validating the feasibility of deploying remote operation of road vehicles on public roads. These tests range from the technological scope to social aspects such as acceptability and usability that affect human performance. This survey presents a compilation of works that approach the remote operation of road vehicles. We start by describing the basic architecture of remote operation systems and classify their modes of operation depending on the level of human intervention. We use this classification to organize and present recent and relevant work on the field from industry and academia. Finally, we identify the challenges in the deployment of remote operation systems in the technological, regulatory, and commercial scopes.

We present a new paradigm for performing remote haptic-robot interactive operations. The new paradigm is anchored on an architecture that combines local autonomy with a high-level exchange strategy of reference input. This represents a departure from the conventional reliance on direct exchange of low-level control signals in a global feedback control system. The new approach establishes two local autonomous controllers acting on the robot and the haptic device, interfaced at a higher level via a dual-proxy model. The dual proxy is a passive bridge between the local autonomous controllers. It generates appropriate motion and force reference inputs that are consistent with the task physical interactions and the levels of autonomy. Its model is adjusted online with respect to exchanged position, contact, and environment geometry information. A key component in this methodology is the perception algorithm on the robot side, the force-space particle filter, designed to reliably estimate in real time the environment contact geometry. The series of simulations and physical experimental validations of the approach demonstrate the transparency and high fidelity in haptic-robot interaction and its inherent robustness to communication delays. [ABSTRACT FROM AUTHOR]

In this article, subject to time-varying delay and uncertainties in dynamics, we propose a novel adaptive fixed-time control strategy for a class of nonlinear bilateral teleoperation systems. First, an adaptive control scheme is applied to estimate the upper bound of delay, which can resolve the predicament that delay has significant impacts on the stability of bilateral teleoperation systems. Then, radial basis function neural networks (RBFNNs) are utilized for estimating uncertainties in bilateral teleoperation systems, including dynamics, operator, and environmental models. Novel adaptation laws are introduced to address systems’ uncertainties in the fixed-time convergence settings. Next, a novel adaptive fixed-time neural network control scheme is proposed. Based on the Lyapunov stability theory, the bilateral teleoperation systems are proved to be stable in fixed time. Finally, simulations and experiments are presented to verify the validity of the control algorithm. [ABSTRACT FROM AUTHOR]

Ensuring the safety of a teleoperation system is crucial, and optimization-based control approaches are an effective solution to ensure that the robot satisfies the multiple safety constraints. However, the control input reshaping via optimization can make a system nonpassive. Therefore, this study proposes a new optimization method for ensuring both the safety and passivity of a teleoperation system by integrating it with the task scaling method. Additionally, the energy tank method is implemented to modulate the tradeoff between dissipativity and control performance. The experiments demonstrated that the proposed method ensures the safety and passivity of the teleoperated system for all robot configurations and any operator’s command inputs. [ABSTRACT FROM AUTHOR]

Haptic feedback is critical in a broad range of human-machine/computer-interaction applications. However, the high cost and low portability/wearability of haptic devices remain unresolved issues, severely limiting the adoption of this otherwise promising technology. Electrotactile interfaces have the advantage of being more portable and wearable due to their reduced actuators’ size, as well as their lower power consumption and manufacturing cost. The applications of electrotactile feedback have been explored in human-computer interaction and human-machine-interaction for facilitating hand-based interactions in applications, such as prosthetics, virtual reality, robotic teleoperation, surface haptics, portable devices, and rehabilitation. This article presents a technological overview of electrotactile feedback, as well a systematic review and meta-analysis of its applications for hand-based interactions. We discuss the different electrotactile systems according to the type of application. We also discuss over a quantitative congregation of the findings, to offer a high-level overview into the state-of-art and suggest future directions. Electrotactile feedback systems showed increased portability/wearability, and they were successful in rendering and/or augmenting most tactile sensations, eliciting perceptual processes, and improving performance in many scenarios. However, knowledge gaps (e.g., embodiment), technical (e.g., recurrent calibration, electrodes’ durability) and methodological (e.g., sample size) drawbacks were detected, which should be addressed in future studies. [ABSTRACT FROM AUTHOR]

The time-domain passivity approach (TDPA) has gained considerable attention (even beyond the fields of haptics and teleoperation) because it can ensure passivity while maintaining high system performance. Despite its simplicity and effectiveness, the practical applications of this method are limited by its high-frequency force modification, so-called chattering. Virtual mass-spring (VMS) filters have been proposed to overcome this limitation. However, such approaches also suffer from certain limitations attributable to the distorted forces. Herein, we propose a method to mitigate the chattering of the TDPA, while also retaining its merits. We introduce a non-zero velocity threshold, within which the adaptive damping of the TDPA is scaled down, to attenuate the chattering while maintaining a stable response significantly. The feasibility and effectiveness of the proposed method are evaluated using a time-delayed bilateral teleoperation system. We confirm that the proposed method can significantly outperform the conventional and VMS filter-based TDPAs. [ABSTRACT FROM AUTHOR]

The presence of unknown physical interaction between the patients’ body and surgical tool in laparoscopic surgery requires a secure end-effector positioning while assuring a reliable constraint motion. In this work, a task-space control approach based on fuzzy approximation is proposed for a teleoperated surgery scenario utilizing a serial redundant robot manipulator (7 degrees of freedom), the motions of which are constrained with respect to a point known as remote center of motion (RCM). The dynamical uncertainties due to the physical interaction are considered and estimated to maintain operational accuracy by introducing the decoupled adaptive fuzzy approximation technique. Experiments verify the presented approach’s effectiveness. Results indicate that with the proposed control method, the accuracy of the surgical operation is improved while the safety can be ensured for teleoperated surgery with RCM constraint. [ABSTRACT FROM AUTHOR]

Mapping human hand motion to robotic hands has great significance in a wide range of applications, such as teleoperation and imitation learning. The ultimate goal is to develop a device-independent control solution based on human hand synergies. Over the past twenty years, a considerable number of mapping methods have been proposed, but most of the mapping methods use intrusive devices, such as the CyberGlove data gloves, to capture human hand motion. Until recently, a very small number of mapping methods have been proposed based on vision-based human hand pose estimation. Traditionally, mapping methods and vision-based human hand pose estimation have been studied independently. To the best of our knowledge, no review has been conducted to summarize the achievements on haptic mapping methods or explore the feasibility of applying off-the-shelf human hand pose estimation algorithms to teleoperation. To address this literature gap, we present the first survey on mapping human hand motion to robotic hands from a kinematic and algorithmic perspective. We discuss the realistic challenges, intuitively summarize recent mapping methods, analyze the theoretical solutions, and provide a teleoperation-oriented human hand pose estimation overview. As a preliminary exploration, a vision-based human hand pose estimation algorithm is introduced for robotic hand teleoperation. [ABSTRACT FROM AUTHOR]

This study designs a high-precision bilateral teleoperation control for a dissimilar master–slave system. The proposed nonlinear control design takes advantage of a novel subsystem-dynamics-based control method that allows designing of individual (decentralized) model-based controllers for the manipulators locally at the subsystem level. Very importantly, a dynamic model of the human operator is incorporated into the control of the master manipulator. The individual controllers for the dissimilar master and slave manipulators are connected in a specific communication channel for the bilateral teleoperation to function. Stability of the overall control design is rigorously guaranteed with arbitrary time delays. Novel features of this study include the completely force-sensor-less design for the teleoperation system with a solution for a uniquely introduced computational algebraic loop, a method of estimating the exogenous operating force of an operator and the use of a commercial haptic manipulator. Most importantly, we conduct experiments on a dissimilar system in two degrees of freedom (DOFs). As an illustration of the performance of the proposed system, a force scaling factor of up to 800 and position scaling factor of up to 4 was used in the experiments. The experimental results show an exceptional tracking performance, verifying the real-world performance of the proposed concept. [ABSTRACT FROM AUTHOR]

Model-mediated teleoperation (MMT) employs an environment model at the master side to compute feedback output to the master at a faster rate. This approach improves system stability in the presence of time delay. MMT, however, does not generally perform well if the employed model is not accurate. The model mismatch is unavoidable when the environment is unknown in advance or varies. This paper proposes MMT employing an adaptive model. The proposed method adaptively moves the reference point of the employed model, whereas the previous MMTs used reference points fixed to the surface of objects in the environment. This can make system stability independent of the time delay. Experiments show that the proposed method improves stability compared to the previous MMTs when there are model mismatches. User studies are conducted to compare the operator’s performance in two tasks, control of force exerted to objects in the environment, and discrimination of object stiffness. The result shows that the error in the forces applied to objects in the environment significantly decreases in the proposed method. Errors in forces rendered to the master are also improved by at least 20.2%. The experiment result also shows that subjects can discriminate up to 40.9% smaller differences in the stiffness than the previous MMT under the same time delay.

Force feedback during teleoperation and in Virtual Reality (VR) environments is becoming increasingly common. We are interested in understanding the impact of motion on the directional accuracy of force perception, as observed in a VR environment. We used a custom force-feedback system that pulled a handle with a force of 1.87N at various angles in front of N=14 subjects. The virtual environment showed a curved wall, which corresponded to the locations from which the force could physically originate. Subjects selected where they perceived the force to originate from with a virtual laser pointer and by orienting their head. We compared several conditions: the subject held the handle still; the subject moved the handle back and forth toward the center of the wall; the subject moved the handle back and forth across their body; and the subject moved the handle back and forth toward where they thought the force was originating. Subjects were able to localize the force with an average accuracy of 1–10 degrees depending on the force’s location, which is better than previous studies. All conditions had similar accuracies. Subjects had the best precision when they followed the force as compared to either of the other conditions with movement. As a secondary objective, we compared the accuracy of a hand-held controller and the head-mounted display (HMD); the HMD was 2.4 degrees more precise than the hand-held controller.

In order to realize a vivid tactile transmission in a teleoperation system, it is necessary to send and receive a lot of motion information such as position, speed, and force with extreme high frequency. In addition, in the actual use, many systems are connected to each other through a network. Against this backdrop, the explosive increase in communication traffic becomes a serious problem. The difficulty of this problem is that there exist complicated exchanges of motion information among the systems and they affect each other. Therefore, this article proposes a novel configuration of a teleoperation system that consists of the conversion technique of transmitted and received information into a single physical quantity (equilibrium force) and the reconstruction technique of discrete data (ZOH: zero-order hold/FOH: first-order hold). The analyses and experiments revealed that 1) FOH is superior to ZOH in terms of operational force 2) FOH and ZOH are about the same performance in terms of reproducibility 3) FOH tends to be more unstable when the packet transmission period increases.

In recent years, floating debris in the sea has damaged the environment and harmed wildlife. Systems for collecting floating marine debris in the open ocean by installing a huge net or using a vacuum device have been proposed. However, to the best of our knowledge, there is no system for collecting marine debris in complex topographies, such as ports wherein marine debris tends to accumulate, and large vessels cannot enter. This study proposes a system to locate and collect floating marine debris by jointly using an unmanned aerial vehicle (UAV) and an unmanned underwater vehicle (UUV). The UAV discovers marine debris that cannot be observed by the UUV owing to its limited view, and the UUV is used for collection. However, for application of such a system, problems related to target detection, UUV navigation, and target collection must be solved. In the proposed system, the following components are developed: a graphical user interface to select the collection target, controller for the UUV to navigate to the target, and a collecting motion generator. We tested and evaluated the proposed cooperative system using a dynamic simulator and physical robots. As a result, we confirm that our system can achieve efficient debris collection in complex topographies, is easy to use, will reduced workload, and can be applied practically.

Teleoperated excavators have a risk of overturning or falling when digging because it is more difficult to receive information regarding the machine posture and the work object condition than in the case of actual machine boarding. In this study, the machine instability derived from the attachment posture and the digging reaction force has been proposed. In addition, for feedback regarding the machine instability to the operator, an intuitive visual presentation method using a meter is proposed. To verify the effect of the proposed machine instability feedback, an excavation operation simulator was constructed, and an experiment was conducted. As a result, by using the machine instability feedback, it was confirmed that digging could be performed without generating the tilting and forward movements of the machine body, and that the ratio of the time during which the machine instability exceeded 100% decreased, while the productivity remained constant. Additionally, by verifying the effect of the subject experiment, even in the actual teleoperated excavator, it was confirmed that the work could be conducted without generating a greater tilting of the machine body, while maintaining productivity. Therefore, the possibility of working safely in the teleoperated excavator using machine instability feedback was clarified.

Research on robotic manipulation of fragile, compliant objects, such as food items, is gaining traction due to its game-changing potential within the food production and retailing sectors, currently characterized by manually intensive and highly repetitive tasks. Food products exhibit high levels of frailness, biological variation, and complex 3-D shapes and textures. For these reasons, introducing greater levels of robotic automation in the food and agricultural sectors remains an important challenge. This article addresses this challenge by developing a human-centered, haptic-based, learning from demonstration (LfD) policy that enables pretrained autonomous grasping of food items using an anthropomorphic robotic system. The policy combines data from teleoperation and direct human manipulation of objects, embodying human intent and interaction areas of significance. We evaluated the proposed solution against a recent state-of-the-art LfD policy as well as against two standard impedance controller techniques. Results show that the proposed policy performs significantly better than the other considered techniques, leading to high grasping success rates while guaranteeing the integrity of the food at hand. [ABSTRACT FROM AUTHOR]

The communication time delay and uncertain models of robotic manipulators are the major problem in the teleoperation system, which can reduce the performance and stability of the system. This article proposed a novel finite-time adaptive control scheme for position and force tracking performances of the teleoperation system. First, a combined auxiliary error system with position and force tracking errors is designed. Second, a velocity feedback filter is introduced, and a new auxiliary variable function with finite-time structure is designed for controller design. The radial basis function neural network (RBFNN) is applied to estimate the uncertain parts. Then the finite-time adaptive control scheme and adaptive laws are given. Third, based on the Lyapunov method, stability and finite-time performance are demonstrated. And finally, the simulation and experimental studies (with Phantom Ommi devices) are performed and demonstrate the effectiveness of the proposed control scheme on teleoperation position/force tracking. [ABSTRACT FROM AUTHOR]

Operational devices used in the traditional robot teleoperation methods restrict the space for interaction. Furthermore, operational and feedback devices are often separated. Thus, the operator has to switch his attention between the operation area and the feedback area during the interaction, which is unnatural and inefficient. In this article, we propose a new gesture- and speech-guided teleoperation method, which unifies operation, visual feedback, and force feedback areas, to achieve a more immersive and natural interaction environment. First, a mobile interaction method combining mobile operation platform and augmented reality (AR) technology is proposed to enlarge the operational space of operators. Second, the gesture tracking method with AR allows the operators to directly interact with virtual objects using bare hands for natural interaction. Third, speech interaction is introduced to improve the precision of fine teleoperations and the naturalness of interactions. Fourth, we introduce unrestricted force feedback based on electromagnetic force technology to provide operators with force feedbacks without limiting their actions. Besides, a virtual guiding force based on the artificial potential field is presented to help the operators to complete high-precision tasks more efficiently. This approach allows the operators to focus on the task without switching attention and the experimental results show that the proposed method provides an efficient and natural interaction for robot teleoperation. [ABSTRACT FROM AUTHOR]

Achieving high-fidelity haptic feedback in teleoperated robotic systems is crucial, particularly in applications, such as medical, industrial, military, space, and underwater explorations. The quality of haptic feedback depends to a great extent on the controller design objectives, such as stability and transparency. In addition to the time delay, the most challenging issues that affect haptic feedback are model uncertainties, parametric variations, and external disturbances. In this article, such issues have been addressed by developing a novel multimodal adaptive robust control algorithm for teleoperated systems with time-varying delay. Here, parametric convergence is achieved swiftly with a milder condition than a stronger persistent excitation condition on the regressor signal, resulting in the enhancement of transient performance. The proposed method ensures the robustness of the system against bounded perturbations and high-frequency unmodeled dynamics using a modified terminal sliding-mode controller alongside the adaptive scheme. The proposed method promises high-fidelity haptic feedback to the operator by ensuring improved robust global asymptotic stability, synchronization, and transparency, addressing parametric variations and disturbances. The validity of the proposed adaptive robust control law is substantiated through simulations and comparisons with the existing methods for a 2-R teleoperation system. The experimental results for a 1-R teleoperation system are also presented. [ABSTRACT FROM AUTHOR]

Objective: Cardiovascular diseases are the most common cause of global death. Endovascular interventions, in combination with advanced imaging technologies, are promising approaches for minimally invasive diagnosis and therapy. More recently, teleoperated robotic platforms target improved manipulation accuracy, stabilisation of instruments in the vasculature, and reduction of patient recovery times. However, benefits of recent platforms are undermined by a lack of haptics and residual patient exposure to ionising radiation. The purpose of this research was to design, implement, and evaluate a novel endovascular robotic platform, which accommodates emerging non-ionising magnetic resonance imaging (MRI). Methods: We proposed a pneumatically actuated MR-safe teleoperation platform to manipulate endovascular instrumentation remotely and to provide operators with haptic feedback for endovascular tasks. The platform task performance was evaluated in an ex vivo cannulation study with clinical experts ($N = 7$) under fluoroscopic guidance and haptic assistance on abdominal and thoracic phantoms. Results: The study demonstrated that the robotic dexterity involving pneumatic actuation concepts enabled successful remote cannulation of different vascular anatomies with success rates of 90%–100%. Compared to manual cannulation, slightly lower interaction forces between instrumentation and phantoms were measured for specific tasks. The maximum robotic interaction forces did not exceed ${3}\rm {N}$. Conclusion: This research demonstrates a promising versatile robotic technology for remote manipulation of endovascular instrumentation in MR environments. Significance: The results pave the way for clinical translation with device deployment to endovascular interventions using non-ionising real-time 3D MR guidance. [ABSTRACT FROM AUTHOR]

A six-degree-of-freedom robotic manipulator inverse kinematics (IK) for position control is proposed for the bilateral teleoperation process that is implemented through a joystick for nuclear power plant dismantling operations. The control strategy of the manipulator includes the use of the joystick to generate the Cartesian space trajectory followed by the IK to yield the joint space trajectory for implementing position control. In this paper, a novel technique for the IK is proposed. It involves the use of the particle swarm optimization (PSO) algorithm with the inverse Jacobian (IJ). The special case of the dual PSO is based on dividing the PSO algorithm into two such that the trajectory position and orientation are separately optimized by the algorithms, resulting in a faster convergence. In contrast, the inverse Jacobian aids in generating a smooth joint trajectory. The integral sliding mode control (ISMC) is proposed for position control because it does not require information on system dynamics. The ISMC improves the system trajectory tracking performance by using a switching gain to compensate for system dynamics and perturbations (disturbance and unmatched uncertainties), ultimately reducing the time delay. The effectiveness of the PSO combined with the IJ and the robustness of the ISMC in the teleoperation process are confirmed by the experimental results.

As a senile chronic, progressive and currently incurable disease, dementia has an enormous impact on society and life quality of the elderly. The development of teleoperation technology has changed the traditional way of care delivery and brought a variety of novel applications for dementia care. In this paper, a telerobotic system is presented which gives the caregivers the capability of assisting dementia elderly remotely. The proposed system is composed of a dual-arm collaborative robot (YuMi) and a wearable motion capture device. The communication architecture is achieved by the robot operation system (ROS). The position-orientation data of the operator’s hand are obtained and used to control the YuMi robot. Besides, a path-constrained mapping method is designed for motion trajectory tracking between the robot and the operator in the progress of teleoperation. Meanwhile, corresponding experiments are conducted to verify the performance of the trajectory tracking using the path-constrained mapping method. Results show that the position tracking deviation between the trajectory of the operator and the robot measured by dynamic time warping distance is 1.05 mm at the sampling frequency of 7.5 Hz. Moreover, the practicability of the proposed system was verified by teleoperating the YuMi robot to pick up a medicine bottle and further demonstrated by assisting an elderly woman in picking up a cup remotely. The proposed telerobotic system has potential utility for improving the life quality of dementia elderly and the care effect of their caregivers.

In the human-in-the-loop robot-assisted telemanipulation application scenario, artificial intelligence and sensors technology were utilized to improve autonomous levels of operation. However, in some special application cases, human decision-making plays a crucial role, haptic display of interaction between the robot end-effector, and a virtual or real remote environment is of significant importance. To this end, this article proposed a new approach to expanding $Z$ -width by introducing magnetorheological fluids (MRFs) and the spring element hybrid actuation mechanism. Then, system stability and human decision-making based on perceiving transparency theory are further analyzed. Moreover, a hybrid actuation mechanism-based $Z$ -width expansion mathematical model is established while its control scheme-related parameters design is presented. Furthermore, $Z$ -width expansion experimental results demonstrated the effectiveness of this hybrid actuation mechanism by introducing MRFs and the spring element. A fabricated thumb haptic interface as a master controller can effectively reflect the $Z$ -width of the remote environment in a robot-assisted telemanipulation application scenario. [ABSTRACT FROM AUTHOR]

In this paper, two novel adaptive finite-time control schemes are proposed for position tracking of nonlinear teleoperation system, which dynamic uncertainties, actuator saturation, and time-varying communication delays are considered. First, a novel auxiliary variable is designed to provide more stable performance. The radial basis function (RBF) neural network is introduced to estimate dynamic uncertainties. Second, two adaptive finite-time control schemes are investigated. In control scheme I, the RBF neural network and the gain switching strategy are applied to compensate the actuator saturation. In control scheme II, an auxiliary compensation filter and the compensation adaptive update laws, which contain the finite-time structure, are developed for dealing with saturation. Third, the finite-time adaptive controller is designed in each of these two control schemes. Based on the multiple Lyapunov function method, the closed-loop teleoperation system with these two control methods is proved to be bounded and finite-time stability. Finally, the simulation experiments are performed and the comparisons with other control methods are shown. The effectiveness of the proposed control schemes is demonstrated. [ABSTRACT FROM AUTHOR]

Objective: We aimed to develop a system for people with amputation that non-invasively restores missing control and sensory information for an ankle-foot prosthesis. Methods: In our approach, a wrist exoskeleton allows people with amputation to control and receive feedback from their prosthetic ankle via teleoperation. We implemented two control schemes: position control with haptic feedback of ankle torque at the wrist; and torque control that allows the user to modify a baseline torque profile by moving their wrist against a virtual spring. We measured tracking error and frequency response for the ankle-foot prosthesis and the wrist exoskeleton. To demonstrate feasibility and evaluate system performance, we conducted an experiment in which one participant with a transtibial amputation tracked desired wrist trajectories during walking, while we measured wrist and ankle response. Results: Benchtop testing demonstrated that for relevant walking frequencies, system error was below human perceptual error. During the walking experiment, the participant was able to voluntarily follow different wrist trajectories with an average RMS error of 1.55° after training. The ankle was also able to track desired trajectories below human perceptual error for both position control (RMSE = 0.8°) and torque control (RMSE = 8.4%). Conclusion: We present a system that allows a user with amputation to control an ankle-foot prosthesis and receive feedback about its state using a wrist exoskeleton, with accuracy comparable to biological neuromotor control. Significance: This bilateral teleoperation system enables novel prosthesis control and feedback strategies that could improve prosthesis control and aid motor learning. [ABSTRACT FROM AUTHOR]

Robotic teleoperation with a bilateral control system has attracted attention owing to its haptic transmission performance. However, conventional bilateral control systems require broadband communication to transmit the vivid haptic sensation. This problem limits the application range of the bilateral control systems. The communication traffic can be reduced by predicting and extrapolating the incoming data. However, in the conventional prediction-based methods, only one type (e.g., position, velocity, or force) of data is transmitted per one direction because of the difficulty of predicting multiple independent data. The novelty of this article is the realization of the prediction-based traffic reduction in the four-channel bilateral control system that transmits accurate haptic sensation by communicating both position and force data. By equivalently transforming this control scheme in the structure of impedance control, the transmit data are summed up to one data, equilibrium force. The equilibrium force is not only transmitted but extrapolated on the receiver side. As a result, the communication frequency becomes low without degrading haptic transmission performance. The validity of the proposed method was confirmed through experiments and succeeded to reduce the communication data size to less than 3.0%. The proposed method helps to realize a high-performance bilateral control system on band-limited networks. [ABSTRACT FROM AUTHOR]

Haptic communication, the exchange of force and tactile information during dancing or moving a table together, has been shown to benefit the performance of human partners. Similarly, it could also be used to improve the performance of robots working in contact with a human operator. As we move to more robot integrated workspaces, how common network features such as delay or jitter impact haptic communication need to be better understood. Here using a human-like interactive robotic controller, that has been found to be indistinguishable by humans to human interaction, we evaluate how subjects' performance and perception is altered by varying levels of transmission delay. We find that subjects are able to recognise haptic delay at very small levels within haptic interaction. However, while they are consciously aware of the delay they can only compensate for it up until a certain point, after which they perceive it as the addition of noise/impedance into the system. [ABSTRACT FROM AUTHOR]

Teleoperated robots are an effective means to sense a remote environment. Impedance control is often preferred in teleoperated robots due to its capability to modulate the dynamic relationship between the position and force of the robot. One of the challenges in the practical implementation of impedance controllers is the determination of the extent to which the dynamic properties of the robot are to be modified. A methodology for quantitatively estimating the desired dynamic parameters of the robot in order to have stable interaction with the environment is proposed in this paper. The best set of scaling factors by which the environment parameters need to be scaled to obtain a corresponding desired parameter for the robot are formulated. In addition, novel impedance controllers for teleoperated robots, which achieve the desired dynamic performance even under time-varying delays, are formulated in this paper. The uniqueness of the proposed controllers lies in enabling the system to interact with a wide range of environments, unlike the traditional teleoperation controllers which allow interaction with only those environments whose dynamic properties are within a specific range. Passivity analysis is performed to prove the global asymptotic stability of the system. Experimental implementation demonstrates the successful sensing of a wide range of environments. [ABSTRACT FROM AUTHOR]

Teleoperated robots are traditionally used for extended human sensing and manipulation of remote environment. Though controllers have already been proposed in the literature to achieve stability and transparency in the system, their implementation becomes limited because of the complexities in tuning the gains. The objective of this paper is to propose novel controllers, which ensure the system stability and transparency irrespective of the controller gains. The proposed controllers are also shown to accommodate varying time-delays, during which the stability and transparency usually conflict each other. The non-linear tool of passivity analysis is used to demonstrate that the proposed controllers achieve the desired objectives. In addition to the proposed controllers, a torque based sensing is utilized to enhance the transparent sensing of the remote environment, which readily facilitates disturbance rejection for the proposed controllers. Therefore, the proposed methodology (combination of both the controller and sensing method) helps the user in accurately sensing the remote environment and concurrently maintaining global asymptotic stability for any arbitrary choice of controller gains. Experiments carried on a five-bar teleoperation system demonstrate global asymptotic stability and transparent sensing even under time varying delays, and with a much wider range of feedback gains. [ABSTRACT FROM AUTHOR]

In this paper, a neural network (NN) based adaptive controller has been successfully developed for the teleoperation of a DLR-HIT II robot hand using Leap Motion sensor. To achieve this, the coordinate positions of the human hand fingers are captured by a Leap Motion sensor. Moreover, inverse kinematics is used to transform the Cartesian position data of the fingertips into corresponding joint angles of all the five fingers, which are then directly sent to teleoperate the DLR-HIT II hand via User Datagram Protocol (UDP). In addition, a NN-based control system programmed by the MATLAB Simulink function has been investigated to enhance the overall control performance by compensating the dynamic uncertainties existing in the teleoperation system. The stability of the control system has been proved mathematically using Lyapunov stability equations. A series of experiments have been conducted to test the performance of the proposed control technique, which has been proved to be an effective teleoperation strategy for the DLR-HIT II robot hand.

Performance of teleoperation can be greatly influenced by time delay in the process of tele-manipulation with respect to accuracy and transparency. Wave variable is an effective algorithm to achieve a good stable capability. However, some traditional wave variable methods may decrease the performance of transparency and suffer the impacts of wave reflection. To deal with the problem of stability and transparency in teleoperation, in this paper, a novel wave variable method with four channel is presented to achieve stable tracking in position and force. In addition, the proposed method can achieve the distortion compensation and reduce the impacts of wave reflection. The simulation experimental results verified the tracking performance of the proposed method.

This paper proposes a novel teleoperation method that allows users to guide robots hand by hand along with speech. In this method, the virtual robot modeled according to the remote real robot is projected into the real local environment to form a 3D operation interface. In this case, users can directly interact with virtual objects by their hands. Furthermore, since the Leap Motion is attached to the augmented reality (AR) glasses, the operation space is greatly extended. Therefore, users can observe the virtual robot from an arbitrary angle without blind angle in such a mobile pattern, which enhances the users' interactive immersion and provides more natural human-machine interaction. To improve the accuracy of the measurement, an unscented Kalman filter (UKF) and an improved particle filter (IPF) are used to estimate the position and orientation of the hand, respectively. Furthermore, Term Frequency-Inverse Document Frequency (TF-IDF) and maximum entropy model are adopted to recognize the speech and gestures instructions of the user. The proposed method is compared with the three human-machine methods on various experiments. The results verified the effectiveness of the proposed method.

Semi-autonomous teleoperation based on Learning from Demonstration is an effective method of remote operation of space manipulators, especially in the scenarios with limited communication and repeated operation problems. However, the joint trajectories generated by those methods may not be suitable for space manipulator control. In this paper, we present a novel semi-autonomous teleoperation method, which has been used for the TianGong-2 manipulator system. The proposed method can not only reproduce trajectories for the tasks according to the current environment, but also generate the smoother and smaller joint control torques. To implement the method, we first collected kinesthetic demonstrations by the space manipulator teaching platform as prior knowledge. Then, based on these kinesthetic demonstrations, we designed the joint control commands with Dynamics Constraint Learning from Demonstration algorithm. We finally evaluated our method with the simulation and on-orbit experiment by locating the dexterous hand to the pre-screwing bolt. Our results show a significant reduction of joint control torque fluctuations and peak-to-peak values, and also can reduce energy consumption.

Objective: Intraoperative palpation is a surgical gesture jeopardized by the lack of haptic feedback which affects robotic minimally invasive surgery. Restoring the force reflection in teleoperated systems may improve both surgeons’ performance and procedures’ outcome. Methods: A force-based sensing approach was developed, based on a cable-driven parallel manipulator with anticipated seamless and low-cost integration capabilities in teleoperated robotic surgery. No force sensor on the end-effector is used, but tissue probing forces are estimated from measured cable tensions. A user study involving surgical trainees (n = 22) was conducted to experimentally evaluate the platform in two palpation-based test-cases on silicone phantoms. Two modalities were compared: visual feedback alone and both visual + haptic feedbacks available at the master site. Results: Surgical trainees’ preference for the modality providing both visual and haptic feedback is corroborated by both quantitative and qualitative metrics. Hard nodules detection sensitivity improves (94.35 ± 9.1% vs 76.09 ± 19.15% for visual feedback alone), while also exerting smaller forces (4.13 ± 1.02 N vs 4.82 ± 0.81 N for visual feedback alone) on the phantom tissues. At the same time, the subjective perceived workload decreases. Conclusion: Tissue-probe contact forces are estimated in a low cost and unique way, without the need of force sensors on the end-effector. Haptics demonstrated an improvement in the tumor detection rate, a reduction of the probing forces, and a decrease in the perceived workload for the trainees. Significance: Relevant benefits are demonstrated from the usage of combined cable-driven parallel manipulators and haptics during robotic minimally invasive procedures. The translation of robotic intraoperative palpation to clinical practice could improve the detection and dissection of cancer nodules. [ABSTRACT FROM AUTHOR]

In this paper, bilateral teleoperation of multiple slaves coupled to a single master under scheduling communication is investigated. The sampled-data transmission between the master and the multiple slaves is fulfilled over a delayed communication network, and at each sampling instant, only one slave is allowed to transmit its current information to the master side according to some scheduling protocols. To achieve the master–slave synchronization, round-robin (RR) scheduling protocol and try-once-discard (TOD) scheduling protocol are employed, respectively. By designing a scheduling-communication-based controller, some sufficient stability criteria related to the controller gain matrices, sampling intervals, and communication delays are obtained for the closed-loop teleoperation system under the RR and TOD scheduling protocols, respectively. Finally, simulation studies are given to validate the effectiveness of the proposed results. [ABSTRACT FROM AUTHOR]

Although robotic telemanipulation has always been a key technology for the nuclear industry, little advancement has been seen over the last decades. Despite complex remote handling requirements, simple mechanically linked master–slave manipulators still dominate the field. Nonetheless, there is a pressing need for more effective robotic solutions able to significantly speed up the decommissioning of legacy radioactive waste. This paper describes a novel haptic shared-control approach for assisting a human operator in the sort and segregation of different objects in a cluttered and unknown environment. A three-dimensional scan of the scene is used to generate a set of potential grasp candidates on the objects at hand. These grasp candidates are then used to generate guiding haptic cues, which assist the operator in approaching and grasping the objects. The haptic feedback is designed to be smooth and continuous as the user switches from a grasp candidate to the next one, or from one object to another one, avoiding any discontinuity or abrupt changes. To validate our approach, we carried out two human–subject studies, enrolling 15 participants. We registered an average improvement of 20.8%, 20.1%, and 32.5% in terms of completion time, linear trajectory, and perceived effectiveness, respectively, between the proposed approach and standard teleoperation. [ABSTRACT FROM AUTHOR]

The quality of robotic dexterous manipulation, in real or in virtual environments, relies on a fine control of the fingertips to perform stable grasps and inside-hand manipulation. In practice, teleoperating a robotic hand requires to capture the human hand configuration. If the user manipulates objects with fingertips, the acquisition of their motion must be accurate enough to produce realistic manipulation at the robot hand or its virtual avatar. In this context, one challenge is to accurately capture the motion of the human hand. The performances of three different hand-tracking devices are evaluated in this paper: two data gloves, the VRFree and the Manus VR, and a vision-based system, the Leap Motion Controller. To this end, the positions of the human hand joints and fingertips are captured while performing several tasks, with a high-precision motion capture system as reference, and with the tested devices. The accuracy of the measured joint angles and fingertips positions is compared for the different systems. Specific measurement configurations are considered by varying the hand orientation and the distances to the sensors. The strengths and weaknesses of these different systems are deduced from the experiments. This system review gives insights into the relevance of hand-tracking devices for remote robotic or virtual manipulation. [ABSTRACT FROM AUTHOR]

This paper proposes a modular teleoperation software platform based on the robot operating system (ROS), which is flexible and portable for different master and slave devices. The main issues for the modularity are to develop its control modules in MATLAB/Simulink and access ROS by the robotics system toolbox, which will facilitate the design, analysis, and adjustment of the control algorithm in teleoperation systems greatly. In addition, a case study for this kind of modular teleoperation system is carried out, which is composed of a Geomagic Touch as the master device and a robotic arm of the Rethink Baxter robot as the slave device. Due to the asymmetric structure property between the master and slave devices, the proper workspace mapping which can cover the whole workspace of the slave robot and simultaneously achieve an excellent mapping accuracy becomes a challenging issue. Different from the traditional methods, a hybrid workspace mapping algorithm is proposed, which uses the joint space mapping to cover the whole workspace of the slave device and the operating space mapping to conduct the elaborate manipulation. A smoothing switch law between the joint space mapping and the operating space mapping is also designed. Comparative experiments are carried out, and the results verify the effectiveness and excellent performance of the proposed design methodology.

Currently, diverse existing training environments help new users of electric powered wheelchairs (EPWs) learn how to drive, acquaint, and improve their abilities with these assistive devices. Several authors are developing such environments, and most of them use virtually simulated wheelchairs. Despite the similarities between virtual and real wheelchairs, it is easier to drive the real device, because representation of the wheelchair physical behavior is still a problem for virtual simulated environments. Concerning the driving methods, most of them are based on a joystick, which does not give the opportunity for users to test, practice, and acquaint themselves with new technology, such as driving through eye movements. This paper implements and tests a more realistic approach to a training environment dedicated to new users of EPW. The proposed system is based on a real EPW controlled by teleoperation, and it is flexible enough to attend to multiple driving methods. An architecture that allows a user to send command messages to control a real EPW through the Internet was implemented to test the system. The implemented driving methods used were conventional joystick, eye-tracker, and a generic human-machine interface. The experimental results suggest that new users can practice safely using a real EPW through the Internet, even in a situation with a communication delay of 130.2 ms (average). Furthermore, the proposed system have shown itself to be a potential tool for attending to new EPW users with different types of disabilities and to be a low-cost approach that could be applied in developing countries.

For a bimanual robot teleoperation system, the internal forces affecting the common deformable manipulation object are determined by both robot arms’ actions. F/T sensorless, uncertain dynamics, and varying time delays increase dramatically the control difficulty for system stability and transparency. To address these problems, an internal force control method based on relative impedance is proposed in this paper. First, we deduce the desired positions to decrease internal force tracking errors and abstract the internal force from the product of the relative distance and impedance. Second, two strategies are adopted for reducing internal force tracking errors and motion synchronization. One is designing an adaptive factor to switch authorities for position control and internal force control. The other is proposing a force estimating method to the coupled item of uncertain dynamics, disturbance, and internal forces. Different parts of the coupled item are divided and solved separately by nonlinear characters. An integral sliding surface based on the relative impedance item is designed to minimize the force tracking errors. Finally, the effectiveness of the method is verified by linear matrix inequalities based on Lyapunov–Krasovskii functional synthesis, and an experiment based on a physical robot system is implemented to validate the proposed method. [ABSTRACT FROM AUTHOR]