Search

Abstract In the industrial robots field, efficient and convenient programming methods have been a hot research topic. In recent years, immersive simulation technology has been developing rapidly in many fields, which provides new horizons for the development of industrial robots. This paper presents a HTC VIVE laser scan motion capture and Holohens augmented reality (AR) based interactive Programming by Demonstration (PbD) system for industrial robot. A portable Handheld Teaching Device (HTD) and its calibration algorithm are designed in the system. The portable HTD which is tracked by a laser motion capture system can be viewed as an AR robot end-effector to teach paths. Meanwhile, the AR robot can be simulated in real time during programing. In addition, the robot reproducing the operator’s actions at the same position in space is the focus of programming. So, Multi-system registration methods are proposed to determine the relationship between robot systems, motion capture systems and virtual robot systems. Meanwhile, a path planning algorithm is proposed to convert the captured raw path points into robot-executable code. For unskilled operators, they can easily perform complex programming using the HTD. For skilled senior workers, their skills can be quickly learned by robots using the system.

In the industrial robots field, efficient and convenient programming methods have been a hot research topic. In recent years, immersive simulation technology has been developing rapidly in many fields, which provides new horizons for the development of industrial robots. This paper presents a HTC VIVE laser scan motion capture and Holohens augmented reality (AR) based interactive Programming by Demonstration (PbD) system for industrial robot. A portable Handheld Teaching Device (HTD) and its calibration algorithm are designed in the system. The portable HTD which is tracked by a laser motion capture system can be viewed as an AR robot end-effector to teach paths. Meanwhile, the AR robot can be simulated in real time during programing. In addition, the robot reproducing the operator's actions at the same position in space is the focus of programming. So, Multi-system registration methods are proposed to determine the relationship between robot systems, motion capture systems and virtual robot systems. Meanwhile, a path planning algorithm is proposed to convert the captured raw path points into robot-executable code. For unskilled operators, they can easily perform complex programming using the HTD. For skilled senior workers, their skills can be quickly learned by robots using the system. [ABSTRACT FROM AUTHOR]

Copyright of Automatisierungstechnik is the property of De Gruyter and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Dynamic Movement Primitives (DMP) have found remarkable applicability and success in various robotic tasks, which can be mainly attributed to their generalization, modulation and robustness properties. However, the spatial generalization of DMP can be problematic in some cases, leading to excessive overscaling and in turn large velocities and accelerations. While other DMP variants have been proposed in the literature to tackle this issue, they can also exhibit excessive overscaling as we show in this work. Moreover, incorporating intermediate points (via-points) for adjusting the DMP trajectory to account for the geometry of objects related to the task, or to avoid or push aside objects that obstruct a specific task, is not addressed by the current DMP literature. In this work we tackle these unresolved so far issues by proposing an improved online spatial generalization, that remedies the shortcomings of the classical DMP generalization, and moreover allows the incorporation of dynamic via-points. This is achieved by designing an online adaptation scheme for the DMP weights which is proved to minimize the distance from the demonstrated acceleration profile to retain the shape of the demonstration, subject to dynamic via-point and initial/final state constraints. Extensive comparative simulations with the classical and other DMP variants are conducted, while experimental results validate the practical usefulness and efficiency of the proposed method. [ABSTRACT FROM AUTHOR]

The paper presents a novel offline programming (OLP) method based on programming by demonstration (PbD), which has been validated through user study. PbD is a programming method that involves physical interaction with robots, and kinesthetic teaching (KT) is a commonly used online programming method in industry. However, online programming methods consume significant robot resources, limiting the speed advantages of PbD and emphasizing the need for an offline approach. The method presented here, based on KT, uses a virtual representation instead of a physical robot, allowing independent programming regardless of the working environment. It employs haptic input devices to teach a simulated robot in augmented reality and uses automatic path planning. A benchmarking test was conducted to standardize equipment, procedures, and evaluation techniques to compare different PbD approaches. The results indicate a 47 % decrease in programming time when compared to traditional KT methods in established industrial systems. Although the accuracy is not yet at the level of industrial systems, users have shown rapid improvement, confirming the learnability of the system. User feedback on the perceived workload and the ease of use was positive. In conclusion, this method has potential for industrial use due to its learnability, reduction in robot downtime, and applicability across different robot sizes and types. [ABSTRACT FROM AUTHOR]

We propose a novel approach for robotic industrial insertion tasks using the Programming by Demonstration technique. Our method allows robots to learn a high-precision task by observing human demonstration once, without requiring any prior knowledge of the object. We introduce an Imitated-to-Finetuned approach that generates imitated approach trajectories by cloning the human hand's movements and then fine-tunes the goal position with a visual servoing approach. To identify features on the object used in visual servoing, we model object tracking as the moving object detection problem, separating each demonstration video frame into the moving foreground that includes the object and demonstrator's hand and the static background. Then a hand keypoints estimation function is used to remove the redundant features on the hand. The experiment shows that the proposed method can make robots learn precision industrial insertion tasks from a single human demonstration. [ABSTRACT FROM AUTHOR]

The paper presents a novel offline programming (OLP) method based on programming by demonstration (PbD), which has been validated through user study. PbD is a programming method that involves physical interaction with robots, and kinesthetic teaching (KT) is a commonly used online programming method in industry. However, online programming methods consume significant robot resources, limiting the speed advantages of PbD and emphasizing the need for an offline approach. The method presented here, based on KT, uses a virtual representation instead of a physical robot, allowing independent programming regardless of the working environment. It employs haptic input devices to teach a simulated robot in augmented reality and uses automatic path planning. A benchmarking test was conducted to standardize equipment, procedures, and evaluation techniques to compare different PbD approaches. The results indicate a 47% decrease in programming time when compared to traditional KT methods in established industrial systems. Although the accuracy is not yet at the level of industrial systems, users have shown rapid improvement, confirming the learnability of the system. User feedback on the perceived workload and the ease of use was positive. In conclusion, this method has potential for industrial use due to its learnability, reduction in robot downtime, and applicability across different robot sizes and types.

Programming by demonstration is reaching industrial applications, which allows non-experts to teach new tasks without manual code writing. However, a certain level of complexity, such as online decision making or the definition of recovery behaviors, still requires experts that use conventional programming methods. Even though, experts cannot foresee all possible faults in a robotic application. To encounter this, we present a framework where user and robot collaboratively program a task that involves online decision making and recovery behaviors. Hereby, a task-graph is created that represents a production task and possible alternative behaviors. Nodes represent start, end or decision states and links define actions for execution. This graph can be incrementally extended by autonomous anomaly detection, which requests the user to add knowledge for a specific recovery action. Besides our proposed approach, we introduce two alternative approaches that manage recovery behavior programming and compare all approaches extensively in a user study involving 21 subjects. This study revealed the strength of our framework and analyzed how users act to add knowledge to the robot. Our findings proclaim to use a framework with a task-graph based knowledge representation and autonomous anomaly detection not only for initiating recovery actions but particularly to transfer those to a robot. [ABSTRACT FROM AUTHOR]

Copyright of Automatisierungstechnik is the property of De Gruyter and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The great diversity of end-user tasks ranging from manufacturing environments to personal homes makes pre-programming robots for general purpose applications extremely challenging. In fact, teaching robots new actions from scratch that can be reused for previously unseen tasks remains a difficult challenge and is generally left up to robotics experts. In this work, we present iRoPro, an interactive Robot Programming framework that allows end-users with little to no technical background to teach a robot new reusable actions. We combine Programming by Demonstration and Automated Planning techniques to allow the user to construct the robot's knowledge base by teaching new actions by kinesthetic demonstration. The actions are generalised and reused with a task planner to solve previously unseen problems defined by the user. We implement iRoPro as an end-to-end system on a Baxter Research Robot to simultaneously teach low- and high-level actions by demonstration that the user can customise via a Graphical User Interface to adapt to their specific use case. To evaluate the feasibility of our approach, we first conducted pre-design experiments to better understand the user's adoption of involved concepts and the proposed robot programming process. We compare results with post-design experiments, where we conducted a user study to validate the usability of our approach with real end-users. Overall, we showed that users with different programming levels and educational backgrounds can easily learn and use iRoPro and its robot programming process. [ABSTRACT FROM AUTHOR]

Industrial robots used to assemble customized products in small batches require a lot of reprogramming. With this work we aim to reduce the programming complexity by autonomously finding the fastest assembly plans without any collisions with the environment. First, a digital twin of the robot uses a gym in simulation to learn which assembly skills (programmed by demonstration) are physically possible (i.e. no collisions with the environment). Only from this reduced solution space will the physical twin look for the fastest assembly plans. Experiments show that the system indeed converges to the fastest assembly plans. Moreover, pre-training in simulation drastically reduces the number of interactions before convergence compared to directly learning on the physical robot. This two-step procedure allows for the robot to autonomously find correct and fast assembly sequences, without any additional human input or mismanufactured products. [ABSTRACT FROM AUTHOR]

Programming robots often involves expert knowledge in both the robot itself and the task to execute. An alternative to direct programming is for a human to show examples of the task execution and have the robot perform the task based on these examples, in a scheme known as learning or programming from demonstration. We propose and study a generic and simple learning-from-demonstration framework. Our approach is to combine the demonstrated commands according to the similarity between the demonstrated sensory trajectories and the current replay trajectory. This tracking is solely performed based on sensor values and time and completely dispenses with the usually expensive step of precomputing an internal model of the task. We analyse the behaviour of the proposed model in several simulated conditions and test it on two different robotic platforms. We show that it can reproduce different capabilities with a limited number of meta parameters. [ABSTRACT FROM AUTHOR]

Augmented and virtual reality have been experiencing rapid growth in recent years, but there is still no deep knowledge regarding their capabilities and in what fields they could be explored. In that sense, this paper presents a study on the accuracy and repeatability of Microsoft's HoloLens 2 (augmented reality device) and HTC Vive (virtual reality device) using an OptiTrack system as ground truth. For the HoloLens 2, the method used was hand tracking, whereas, in HTC Vive, the object tracked was the system's hand controller. A series of tests in different scenarios and situations were performed to explore what could influence the measures. The HTC Vive obtained results in the millimeter range, while the HoloLens 2 revealed not very accurate measurements (around 2 cm). Although the difference can seem to be considerable, the fact that HoloLens 2 was tracking the user's hand and not the system's controller made a huge impact. The results are considered a significant step for the ongoing project of developing a human-robot interface by demonstrating an industrial robot using extended reality, which shows great potential to succeed based on our data. [ABSTRACT FROM AUTHOR]

Programming by demonstration has received much attention as it offers a general framework which allows robots to efficiently acquire novel motor skills from a human teacher. While traditional imitation learning that only focuses on either Cartesian or joint space might become inappropriate in situations where both spaces are equally important (e.g., writing or striking task), hybrid imitation learning of skills in both Cartesian and joint spaces simultaneously has been studied recently. However, an important issue which often arises in dynamical or unstructured environments is overlooked, namely how can a robot avoid obstacles? In this paper, we aim to address the problem of avoiding obstacles in the context of hybrid imitation learning. Specifically, we propose to tackle three subproblems: (i) designing a proper potential field so as to bypass obstacles, (ii) guaranteeing joint limits are respected when adjusting trajectories in the process of avoiding obstacles, and (iii) determining proper control commands for robots such that potential human-robot interaction is safe. By solving the aforementioned subproblems, the robot is capable of generalizing observed skills to new situations featuring obstacles in a feasible and safe manner. The effectiveness of the proposed method is validated through a toy example as well as a real transportation experiment on the iCub humanoid robot.

Improving the life quality of people with severe motor paralysis has a significant impact on restoring their functional independence to perform activities of daily living (ADL). Telepresence is a subfield of the robotic-assisted route, where human plays the role of an operator, sending high-level instructions to an assistive robot while receiving sensory feedback. However, for severely motor-impaired people, conventional interaction modalities may not be suitable due to their complete paralysis. Thus, designing alternative ways of interaction such as Brain-Computer Interfaces (BCI) is essential for a telepresence capability. We propose a novel framework that integrates a BCI system and a humanoid robot to develop a brain-controlled telepresence system with multimodal control features. In particular, the low-level control is executed by Programming by Demonstration (PbD) models, and the higher-level cognitive commands are produced by a BCI system to perform vital ADLs. The presented system is based on real-time decoding of attention-modulated neural responses elicited in the brain electroencephalographic signals and generating multiple control commands. As a result, the system allows a user to interact with a humanoid robot while receiving auditory and visual feedback from the robot's sensors. We validated our system across ten subjects in a realistic scenario. The experimental results show the feasibility of the approach in the design of a telepresence robot with high BCI decoding performances.

Programming by demonstration (PBD) is a technique which allows end users to create, modify, accommodate, and expand programs by demonstrating what the program is supposed to do. Although the ideal of common-purpose programming by demonstration or by examples has been rejected as practically unrealistic, this approach has found its application and shown potentials when limited to specific narrow domains and ranges of applications. In this paper, the original method of applying the principles of programming by demonstration in the area of process mining (PM) to interactive construction of block-structured parallel business processes models is presented. A technique and tool that enable interactive process mining and incremental discovery of process models have been described in this paper. The idea is based on the following principle: using a demonstrational user interface, a user demonstrates scenarios of execution of parallel business process activities, and the system gives a generalized model process specification. A modified process mining technique with the α|| algorithm applied on weakly complete event logs is used for creating parallel business process models using demonstration. [ABSTRACT FROM AUTHOR]

SUMMARY: The progressive automation framework allows the seamless transition of a robot from kinesthetic guidance to autonomous operation mode during programming by demonstration of discrete motion tasks. This is achieved by the synergetic action of dynamic movement primitives (DMPs), virtual fixtures, and variable impedance control. The proposed DMPs encode the demonstrated trajectory and synchronize with the current demonstration from the user so that the reference generated motion follows the human's demonstration. The proposed virtual fixtures assist the user in repeating the learned kinematic behavior but allow penetration so that the user can make modifications to the learned trajectory if needed. The tracking error in combination with the interaction forces and torques is used by a variable stiffness strategy to adjust the progressive automation level and transition the leading role between the human and the robot. An energy tank approach is utilized to apply the designed controller and to prove the passivity of the overall control method. An experimental evaluation of the proposed framework is presented for a pick and place task and results show that the transition to autonomous mode is achieved in few demonstrations. [ABSTRACT FROM AUTHOR]

Recently, programing by demonstration (PbD) received much attention for its capacity of fast programming with increasing demands in the robot manipulation area, especially in industrial applications. However, one of the biggest challenges of PbD is the recognition of demonstrator's finger high-fidelity motions especially in the environments with uncertainties, which limits the efficiency and accuracy of PbD. In this article, inspired by human dexterity, a novel PbD approach using the implicit muscular task planning strategy is presented to extract features from the arms’ giant movement and the hands’ fine motions during the demonstrator's operation. Furthermore, we integrate a deep reinforcement learning control method that further improves the manipulations’ adaptive ability in the unknown or dynamic environments. The experimental results show that our proposed approach can deal with relative complex assembly tasks with a success rate of more than 67% within a fit tolerance of 4.2 mm by one-shot demonstration. [ABSTRACT FROM AUTHOR]

The reproduction of hand movements by a robot remains difficult and conventional learning methods do not allow us to faithfully recreate these movements because it is very difficult when the number of crossing points is very large. Programming by Demonstration gives a better opportunity for solving this problem by tracking the user’s movements with a motion capture system and creating a robotic program to reproduce the performed tasks. This paper presents a Programming by Demonstration system in a trajectory level for the reproduction of hand/tool movement by a manipulator robot; this was realized by tracking the user’s movement with the ArToolkit and reconstructing the trajectories by using the constrained cubic spline. The results obtained with the constrained cubic spline were compared with cubic spline interpolation. Finally the obtained trajectories have been simulated in a virtual environment on the Puma 600 robot. [ABSTRACT FROM AUTHOR]

There is a growing need for adaptive robotic assembly systems that are fast to setup and reprogram when new products are introduced. The World Robot Challenge at World Robot Summit 2018 was centered around the challenge of setting up a flexible robotic assembly system aiming at changeover times below 1 day. This paper presents a method for programming robotic assembly tasks that was initiated in connection with the World Robot Challenge that enables fast and easy setup of robotic insertion tasks. We propose to program assembly tasks by demonstration, but instead of using the taught behavior directly, the demonstration is merged with assembly primitives to increase robustness. In contrast to other programming by demonstration approaches, we perform not only one demonstration but a sequence of four sub-demonstrations that are used to extract the desired robot trajectory in addition to parameters for the assembly primitive. The proposed assembly strategy is compared to a standard dynamic movement primitive and experiments show that the proposed assembly strategy increases the robustness towards pose uncertainties and significantly reduces the applied forces during the execution of the assembly task. [ABSTRACT FROM AUTHOR]

The world is living the fourth industrial revolution, marked by the increasing intelligence and automation of manufacturing systems. Nevertheless, there are types of tasks that are too complex or too expensive to be fully automated, it would be more efficient if the machines were able to work with the human, not only by sharing the same workspace but also as useful collaborators. A possible solution to that problem is on human–robot interaction systems, understanding the applications where they can be helpful to implement and what are the challenges they face. This work proposes the development of an industrial prototype of a human–machine interaction system through Augmented Reality, in which the objective is to enable an industrial operator without any programming experience to program a robot. The system itself is divided into two different parts: the tracking system, which records the operator’s hand movement, and the translator system, which writes the program to be sent to the robot that will execute the task. To demonstrate the concept, the user drew geometric figures, and the robot was able to replicate the operator’s path recorded.

Beating-time gestures are movement patterns of the hand swaying along with music, thereby indicating accented musical pulses. The spatiotemporal configuration of these patterns makes it diÿcult to analyse and model them. In this paper we present an innovative modelling approach that is based upon imitation learning or Programming by Demonstration (PbD). Our approach - based on Dirichlet Process Mixture Models, Hidden Markov Models, Dynamic Time Warping, and non-uniform cubic spline regression - is particularly innovative as it handles spatial and temporal variability by the generation of a generalised trajectory from a set of periodically repeated movements. Although not within the scope of our study, our procedures may be implemented for the sake of controlling movement behaviour of robots and avatar animations in response to music.

This paper proposes a method to construct a force control law for industrial robots by using the normalized cross-correlation (NCC) of human demonstration data. Conventionally there are two solutions where one is based on human demonstration data, and the other is based on numerical optimization. The former one gives the force control parameters efficiently, but the parameters may make the robot unstable. On the other hand, the latter can maximize the performance, but it requires an enormous number of trial and error. Taking into account the merit of each method, we consider combining the two approaches. In our proposed method, the force control laws are determined to maximize the NCC of the human demonstration data. The orientation of the coordinate system is also determined to maximize the NCC of the human demonstration data. Then, the parameters included in the force control law are optimized by the downhill simplex method. The proposed method was applied to a ring-shaped rubber packing assembly task, and it could realize an assembly with human-like performance. Moreover, some comparable experiments were also conducted, and it was confirmed that the proposed method can construct appropriate force controls.

Augmented and virtual reality have been experiencing rapid growth in recent years, but there is still no deep knowledge regarding their capabilities and in what fields they could be explored. In that sense, this paper presents a study on the accuracy and repeatability of Microsoft’s HoloLens 2 (augmented reality device) and HTC Vive (virtual reality device) using an OptiTrack system as ground truth. For the HoloLens 2, the method used was hand tracking, whereas, in HTC Vive, the object tracked was the system’s hand controller. A series of tests in different scenarios and situations were performed to explore what could influence the measures. The HTC Vive obtained results in the millimeter range, while the HoloLens 2 revealed not very accurate measurements (around 2 cm). Although the difference can seem to be considerable, the fact that HoloLens 2 was tracking the user’s hand and not the system’s controller made a huge impact. The results are considered a significant step for the ongoing project of developing a human–robot interface by demonstrating an industrial robot using extended reality, which shows great potential to succeed based on our data.

The evolution of production systems for smart factories foresees a tight relation between human operators and robots. Specifically, when robot task reconfiguration is needed, the operator must be provided with an easy and intuitive way to do it. A useful tool for robot task reconfiguration is Programming by Demonstration (PbD). PbD allows human operators to teach a robot new tasks by showing it a number of examples. The article presents two studies investigating the role of the robot in PbD. A preliminary study compares standard PbD with human–human teaching and suggests that a collaborative robot should actively participate in the teaching process as human practitioners typically do. The main study uses a wizard of oz approach to determine the effects of having a robot actively participating in the teaching process, specifically by controlling the end-effector. The results suggest that active behaviour inspired by humans can lead to a more intuitive PbD. [ABSTRACT FROM AUTHOR]

A handover strategy is proposed that aims at natural and fluent robot to human object handovers. For the approaching phase, a globally asymptotically stable dynamical system (DS) is utilized, trained from human demonstrations and exploiting the existence of mirroring in the human wrist motion. The DS operates in the robot task space thus achieving independence with respect to the robot platform, encapsulating the position and orientation of the human wrist within a single DS. It is proven that the motion generated by such a DS, having as target the current wrist pose of the receiver's hand, is bounded and converges to the previously unknown handover location. Haptic cues based on load estimates at the robot giver ensure full object load transfer before grip release. The proposed strategy is validated with simulations and experiments in real settings. [ABSTRACT FROM AUTHOR]