Search

The learning-from-observation (LfO) paradigm allows a robot to learn how to perform actions by observing human actions. Previous research in top-down learning-from-observation has mainly focused on the industrial domain, which consists only of the real physical constraints between a manipulated tool and the robot's working environment. To extend this paradigm to the household domain, which consists of imaginary constraints derived from human common sense, we introduce the idea of semantic constraints, which are represented similarly to the physical constraints by defining an imaginary contact with an imaginary environment. By studying the transitions between contact states under physical and semantic constraints, we derive a necessary and sufficient set of task representations that provides the upper bound of the possible task set. We then apply the task representations to analyze various actions in top-rated household YouTube videos and real home cooking recordings, classify frequently occurring constraint patterns into physical, semantic, and multi-step task groups, and determine a subset that covers standard household actions. Finally, we design and implement task models, corresponding to these task representations in the subset, with the necessary daemon functions to collect the necessary parameters to perform the corresponding household actions. Our results provide promising directions for incorporating common sense into the robot teaching literature. [ABSTRACT FROM AUTHOR]

With robotics applications playing an increasingly significant role in our daily lives, it is crucial to develop effective methods for teaching and understanding their behavior. However, limited access to physical robots in educational institutions and companies poses a significant obstacle for many individuals. To overcome this barrier, a novel framework that combines realistic robot simulation and intuitive control mechanisms within a virtual reality environment is presented. By accurately emulating the physical characteristics and behaviors of various robots, this framework offers an immersive and authentic learning experience. Through an intuitive control interface, users can interact naturally with virtual robots, facilitating the acquisition of practical robotics skills. In this study, a qualitative assessment to evaluate the effectiveness and user satisfaction with the framework is conducted. The results highlighted its usability, realism, and educational value. Specifically, the framework bridges the gap between theoretical knowledge and practical application in robotics, enabling users to gain hands-on experience and develop a deeper understanding of robot behavior and control strategies. Compared to existing approaches, the framework provides a more accessible and effective alternative for interacting with robots, particularly for individuals with limited physical access to such devices. In conclusion, the study presents a comprehensive framework that leverages virtual reality technology to enhance the learning and training process in robotics. By combining realistic simulations and intuitive controls, this framework represents a significant advancement in providing an immersive and effective learning environment. The positive user feedback obtained from the study reinforces the value and potential of the framework in facilitating the acquisition of essential robotics skills. Ultimately, this work contributes to flattening the robotics learning curve and promoting broader access to robotics education. [ABSTRACT FROM AUTHOR]

In robot teaching, the grasping strategies taught to robots by users are critical information, because these strategies contain the implicit knowledge necessary to successfully perform a series of manipulations; however, limited practical knowledge exists on how to utilize linguistic information for supporting grasp-type recognition in multimodal teaching. This study focused on the effects of text-driven object affordance—a prior distribution of grasp types for each object—on image-based grasp-type recognition. To this end, we created the datasets of first-person grasping-hand images labeled with grasp types and object names and tested if the object affordance enhanced the performance of image-based recognition. We evaluated two scenarios with real and illusory objects to be grasped, considering a teaching condition in mixed reality, where the lack of visual object information can make image-based recognition challenging. The results show that object affordance guided the image-based recognition in two scenarios, that is, increasing the recognition accuracy by (1) excluding the unlikely grasp types from the candidates and (2) enhancing the likely grasp types. Additionally, the “enhancing effect” was more pronounced with greater grasp-type bias for each object in a test dataset. These results indicate the effectiveness of object affordance for guiding grasp-type recognition in multimodal robot teaching applications. The contributions of this study are (1) demonstrating the effectiveness of object affordance in guiding grasp-type recognition both with and without the real objects in images, (2) demonstrating the conditions under which the merits of object affordance are pronounced, and (3) providing a dataset of first-person grasping images labeled with possible grasp types for each object. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Guangdong University of Technology is the property of Journal of Guangdong University of Technology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The concept of Industry 4.0 brings the change of industry manufacturing patterns that become more efficient and more flexible. In response to this tendency, an efficient robot teaching approach without complex programming has become a popular research direction. Therefore, we propose an interactive finger-touch based robot teaching schema using a multimodal 3D image (color (RGB), thermal (T) and point cloud (3D)) processing. Here, the resulting heat trace touching the object surface will be analyzed on multimodal data, in order to precisely identify the true hand/object contact points. These identified contact points are used to calculate the robot path directly. To optimize the identification of the contact points we propose a calculation scheme using a number of anchor points which are first predicted by hand/object point cloud segmentation. Subsequently a probability density function is defined to calculate the prior probability distribution of true finger trace. The temperature in the neighborhood of each anchor point is then dynamically analyzed to calculate the likelihood. Experiments show that the trajectories estimated by our multimodal method have significantly better accuracy and smoothness than only by analyzing point cloud and static temperature distribution.

With robotics applications playing an increasingly significant role in our daily lives, it is crucial to develop effective methods for teaching and understanding their behavior. However, limited access to physical robots in educational institutions and companies poses a significant obstacle for many individuals. To overcome this barrier, a novel framework that combines realistic robot simulation and intuitive control mechanisms within a virtual reality environment is presented. By accurately emulating the physical characteristics and behaviors of various robots, this framework offers an immersive and authentic learning experience. Through an intuitive control interface, users can interact naturally with virtual robots, facilitating the acquisition of practical robotics skills. In this study, a qualitative assessment to evaluate the effectiveness and user satisfaction with the framework is conducted. The results highlighted its usability, realism, and educational value. Specifically, the framework bridges the gap between theoretical knowledge and practical application in robotics, enabling users to gain hands-on experience and develop a deeper understanding of robot behavior and control strategies. Compared to existing approaches, the framework provides a more accessible and effective alternative for interacting with robots, particularly for individuals with limited physical access to such devices. In conclusion, the study presents a comprehensive framework that leverages virtual reality technology to enhance the learning and training process in robotics. By combining realistic simulations and intuitive controls, this framework represents a significant advancement in providing an immersive and effective learning environment. The positive user feedback obtained from the study reinforces the value and potential of the framework in facilitating the acquisition of essential robotics skills. Ultimately, this work contributes to flattening the robotics learning curve and promoting broader access to robotics education.

This paper proposes a robot teaching method using augmented and virtual reality technologies. Robot teaching is essential for robots to accomplish several tasks in industrial production. Although there are various approaches to perform motion planning for robot manipulation, robot teaching is still required for precision and reliability. Online teaching, in which a physical robot moves in the real space to obtain the desired motion, is widely performed because of its ease and reliability. However, actual robot movements are required. In contrast, offline teaching can be accomplished entirely in the computational space, and it requires constructing the robot's surroundings as computer graphic models. Additionally, planar displays do not provide sufficient information on 3D scenes. Our proposed method can be employed as offline teaching, but the operator can manipulate the robot intuitively using a head-mounted device and the specified controllers in the virtual 3D space. We demonstrate two approaches for robot teaching with augmented and virtual reality technologies and show some experimental results. [ABSTRACT FROM AUTHOR]

One of the key challenges in robotic remote laser 3D processing (RL3DP) is to achieve high accuracy for the laser’s working trajectory relative to the features of the workpiece. This paper presents a novel RL3DP system with an automatic 3D teaching functionality for a precise and rapid determination of the working trajectory which comprises a robot manipulator, 3D scanning head, fibre laser and an off-axis positioned camera. The 3D measurement is based on laser triangulation with laser-stripe illumination using the laser’s pilot beam and scanning head. The experimental results show that the system has a precision better than 70 μm and 120 μm along lateral and vertical direction respectively inside the measuring range of 100 × 100 mm. The teaching time is 30-times shorter compared to a visual teaching procedure. Therefore, such a system can lead to large cost reductions for modern production lines that have constant changes to the products’ geometries and functionalities.

Compared with traditional robot teaching methods, robots can learn various human-like skills in a more efficient and natural manner through teleoperation. In this paper, we propose a teleoperation method based on human-robot interaction (HRI), which mainly uses visual information. With only one teleoperation, the robot can reproduce a trajectory. There is a certain error between this trajectory and the optimal trajectory due to the cause of the human demonstrator or the robot. So we use an extreme learning machine (ELM) based algorithm to transfer the demonstrator’s motions to the robot. To verify the method, we use a Microsoft KinectV2 to capture the human body motion and the hand state, according to which a Baxter robot in Virtual Robot Experimentation Platform (V-REP) will be controlled by the command. Through learning and training by the ELM, the robot in V-REP can complete a certain task autonomously and the robot in reality can reproduce this trajectory well. The experimental results show that the developed method has achieved satisfactory performance. [ABSTRACT FROM AUTHOR]

Virtual offline robot teaching provides a flexible and more economic robot programming solution to belt lapping. Due to the elasticity of the components, the applied force directly influences the polishing result, and the corresponding reaction force sensed by the user affects his or her decision on next operation. However, most of the simulation works reported so far focus mainly on the machining effect of belt lapping but not the contacting force. Therefore, this paper proposes a novel force feedback model for virtual robot teaching of belt lapping. By analyzing the force conditions at different process stages, three kinds of forces, namely natural force, colliding force, and resistance force, are defined to facilitate users with continuous force feedback during a virtual lapping process. To validate the model, a comparative study is done between the actual force measured during the belt lapping process and that simulated by the proposed force model. Moreover, evaluators are involved to conduct the virtual lapping of a mechanical part using the prototype developed. Both the quantitative and the evaluation experiments have validated that the force model is able to provide users with realistic force feedback and better ergonomic feelings of immersion. Moreover, they are also positive on the flexibility and efficiency of using the prototype for virtual robot teaching of belt lapping processes. [ABSTRACT FROM AUTHOR]

One of the big challenges in robotics today is to learn from human users that are inexperienced in interacting with robots but yet are often used to teach skills flexibly to other humans and to children in particular. A potential route toward natural and efficient learning and teaching in Human-Robot Interaction (HRI) is to leverage the social competences of humans and the underlying interactional mechanisms. In this perspective, this article discusses the importance of pragmatic frames as flexible interaction protocols that provide important contextual cues to enable learners to infer new action or language skills and teachers to convey these cues. After defining and discussing the concept of pragmatic frames, grounded in decades of research in developmental psychology, we study a selection of HRI work in the literature which has focused on learning-teaching interaction and analyze the interactional and learning mechanisms that were used in the light of pragmatic frames. This allows us to show that many of the works have already used in practice, but not always explicitly, basic elements of the pragmatic frames machinery. However, we also show that pragmatic frames have so far been used in a very restricted way as compared to how they are used in human-human interaction and argue that this has been an obstacle preventing robust natural multi-task learning and teaching in HRI. In particular, we explain that two central features of human pragmatic frames, mostly absent of existing HRI studies, are that (1) social peers use rich repertoires of frames, potentially combined together, to convey and infer multiple kinds of cues; (2) new frames can be learnt continually, building on existing ones, and guiding the interaction toward higher levels of complexity and expressivity. To conclude, we give an outlook on the future research direction describing the relevant key challenges that need to be solved for leveraging pragmatic frames for robot learning and teaching. [ABSTRACT FROM AUTHOR]

This article reports three case studies on the usability and acceptance of an industrial robotic prototype in the context of human-robot cooperation. The three case studies were conducted in the framework of a two-year project named AssistMe, which aims at developing different means of interaction for programming and using collaborative robots in a user-centered manner. Together with two industrial partners and a technological partner, two different application scenarios were implemented and studied with an off-the-shelf robotic system. The operators worked with the robotic prototype in laboratory conditions (two days), in a factory context (one day) and in an automotive assembly line (three weeks). In the article, the project and procedures are described in detail, including the quantitative and qualitative methodology. Our results show that close human-robot cooperation in the industrial context needs adaptive pacing mechanisms in order to avoid a change of working routines for the operators and that an off-the-shelf robotic system is still limited in terms of usability and acceptance. The touch panel, which is needed for controlling the robot, had a negative impact on the overall user experience. It creates a further intermediate layer between the user, the robot and the work piece and potentially leads to a decrease in productivity. Finally, the fear of the worker of being replaced by an improved robotic system was regularly expressed and adds an additional anthropocentric dimension to the discussion of human-robot cooperation, smart factories and the upcoming Industry 4.0. [ABSTRACT FROM AUTHOR]

This article reports three case studies on the usability and acceptance of an industrial robotic prototype in the context of human-robot cooperation. The three case studies were conducted in the framework of a two-year project named AssistMe, which aims at developing different means of interaction for programming and using collaborative robots in a user-centered manner. Together with two industrial partners and a technological partner, two different application scenarios were implemented and studied with an off-the-shelf robotic system. The operators worked with the robotic prototype in laboratory conditions (two days), in a factory context (one day) and in an automotive assembly line (three weeks). In the article, the project and procedures are described in detail, including the quantitative and qualitative methodology. Our results show that close human-robot cooperation in the industrial context needs adaptive pacing mechanisms in order to avoid a change of working routines for the operators and that an off-the-shelf robotic system is still limited in terms of usability and acceptance. The touch panel, which is needed for controlling the robot, had a negative impact on the overall user experience. It creates a further intermediate layer between the user, the robot and the work piece and potentially leads to a decrease in productivity. Finally, the fear of the worker of being replaced by an improved robotic system was regularly expressed and adds an additional anthropocentric dimension to the discussion of human-robot cooperation, smart factories and the upcoming Industry 4.0.

For robots to be more capable interaction partners they will necessarily need to adapt to the needs and requirements of their human companions. One way that the human could aid this adaptation may be by teaching the robot new ways of doing things by physically demonstrating different behaviours and tasks such that the robot learns new skills by imitating the learnt behaviours in appropriate contexts. In human-human teaching, the concept of scaffolding describes the process whereby the teacher guides the pupil to new competence levels by exploiting and extending existing competencies. In addition, the idea of event structuring can be used to describe how the teacher highlights important moments in an overall interaction episode. Scaffolding and event structuring robot skills in this way may be an attractive route in achieving robot adaptation; however, there are many ways in which a particular behaviour might be scaffolded or structured and the interaction process itself may have an effect on the robot's resulting performance. Our overall research goal is to understand how to design an appropriate human-robot interaction paradigm where the robot will be able to intervene and elicit knowledge from the human teacher in order to better understand the taught behaviour. In this article we examine some of these issues in two exploratory human-robot teaching scenarios. The first considers task structuring from the robot's viewpoint by varying the way in which a robot is taught. The experimental results illustrate that the way in which teaching is carried out, and primarily how the teaching steps are decomposed, has a critical effect on the efficiency of human teaching and the effectiveness of robot learning. The second experiment studies the problem from the human's viewpoint in an attempt to study the human teacher's spontaneous levels of event segmentation when analysing their own demonstrations of a routine home task to a robot. The results suggest the existence of some individual differences regarding the level of granularity spontaneously considered for the task segmentation and for those moments in the interaction which are viewed as most important. [ABSTRACT FROM AUTHOR]

A robot trajectory teaching system with a vision-based positioning pen, which we called Solpen, is developed to generate pose paths of six degrees of freedom (6-DoF) for vision-guided robotics applications such as welding, cutting, painting, or polishing, which can achieve a millimeter dynamic accuracy within a meter working distance from the camera. The system is simple and requires only a 2D camera and the printed ArUco markers which are hand-glued on 31 surfaces of the designed 3D-printed Solpen. Image processing techniques are implemented to remove noise and sharpen the edge of the ArUco images and also enhance the contrast of the ArUco edge intensity generated by the pyramid reconstruction. In addition, the least squares method is implemented to optimize parameters for the center pose of the truncated Icosahedron center, and the vector of the Solpen-tip. From dynamic experiments conducted with ChArUco board to verify exclusively the pen performance, the developed system is robust within its working range, and achieves a minimum axis-accuracy at approximately 0.8 mm. [ABSTRACT FROM AUTHOR]