Search

Laboratory automation requires reliable and precise handling of microplates, but existing robotic systems often struggle to achieve this, particularly when navigating around the dynamic and variable nature of laboratory environments. This work introduces a novel method integrating simultaneous localization and mapping (SLAM), computer vision, and tactile feedback for the precise and autonomous placement of microplates. Implemented on a bi-manual mobile robot, the method achieves fine-positioning accuracies of ± 1.2 mm and ± 0.4°. The approach was validated through experiments using both mockup and real laboratory instruments, demonstrating at least a 95% success rate across varied conditions and robust performance in a multi-stage protocol. Compared to existing methods, our framework effectively generalizes to different instruments without compromising efficiency. These findings highlight the potential for enhanced robotic manipulation in laboratory automation, paving the way for more reliable and reproducible experimental workflows. [ABSTRACT FROM AUTHOR]

As the first step in grasping operations, vision-guided grasping actions play a crucial role in enabling intelligent robots to perform complex interactive tasks. In order to solve the difficulties in data set preparation and consumption of computing resources before and during training network, we introduce a method of training human grasping strategies based on small sample representative data sets, and learn a human grasping strategy through only one depth image. Our key idea is to use the entire human grasping area instead of multiple grasping gestures so that we can greatly reduce the preparation of dataset. Then the grasping strategy is trained through the q-learning framework, the agent is allowed to continuously explore the environment so that it can overcome lack of data annotation and prediction in early stage of the visual network, then successfully map the human strategy into visual prediction. Considering the widespread clutter environment in real tasks, we introduce push actions and adopt a staged reward function to make it conducive to the grasping. Finally we learned the human grasping strategy and applied it successfully, and stably executed it on objects that not seen before, improved the convergence speed and grasping effect while reducing the consumption of computing resources. We conducted experiments on a Doosan robotic arm equipped with an Intel Realsense camera and a two-finger gripper, and achieved human strategy grasping with a high success rate in cluttered scenes. [ABSTRACT FROM AUTHOR]

Featured Application: The force/torque control, considering the incurved shape of the TMS coil shape, improved the adherence between the coil and the subject's head. It also enhanced the accuracy of the coil. This paper proposes the force/torque control strategy for the robotized transcranial magnetic stimulation (TMS) system, considering the shape of the TMS coil case. Hybrid position/force control is used to compensate for the error between the current and target position of the coil and to maintain the contact between the coil and the subject's head. The desired force magnitude of the force control part of the hybrid controller is scheduled by the error between the current and target position of the TMS coil for fast error reduction and the comfort of the subject. Additionally, the torque proportional to the torque acting on the coil's center is generated to stabilize the contact. Compliance control, which makes the robot adaptive to the environment, stabilizes the coil and head interaction during force/torque control. The experimental results showed that the force controller made the coil generate a relatively large force for a short time (less than 10 s) for the fast error reduction, and a relatively small interaction force was maintained for the contact. They showed that the torque controller made the contact area inside the coil. The experiment also showed that the proposed strategy could be used for tracking a new target point estimated by the neuronavigation system when the head moved slightly. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) that autonomously explores optimal control policies has become a crucial direction for developing intelligent robots while Dynamic Movement Primitives (DMPs) serve as a powerful tool for efficiently expressing robot trajectories. This article explores an efficient integration of RL and DMP to enhance the learning efficiency and control performance of reinforcement learning in robot manipulation tasks by focusing on the forms of control actions and their smoothness. A novel approach, DDPG-DMP, is proposed to address the efficiency and feasibility issues in the current RL approaches that employ DMP to generate control actions. The proposed method naturally integrates a DMP-based policy into the actor–critic framework of the traditional RL approach Deep Deterministic Policy Gradient (DDPG) and derives the corresponding update formulas to learn the networks that properly decide the parameters of DMPs. A novel inverse controller is further introduced to adaptively learn the translation from observed states into various robot control signals through DMPs, eliminating the requirement for human prior knowledge. Evaluated on five robot arm control benchmark tasks, DDPG-DMP demonstrates significant advantages in control performance, learning efficiency, and smoothness of robot actions compared to related baselines, highlighting its potential in complex robot control applications. [ABSTRACT FROM AUTHOR]

Collaborative human-robot task execution approaches require mutual adaptation, allowing both the human and robot partners to take active roles in action selection and role assignment to achieve a single shared goal. Prior works have utilized a leader-follower paradigm in which either agent must follow the actions specified by the other agent. We introduce the User-aware Hierarchical Task Planning (UHTP) framework, a communication-free human-robot collaborative approach for adaptive execution of multi-step tasks that moves beyond the leader-follower paradigm. Specifically, our approach enables the robot to observe the human, perform actions that support the human's decisions, and actively select actions that maximize the expected efficiency of the collaborative task. In turn, the human chooses actions based on their observation of the task and the robot, without being dictated by a scheduler or the robot. We evaluate UHTP both in simulation and in a human subjects experiment of a collaborative drill assembly task. Our results show that UHTP achieves more efficient task plans and shorter task completion times than non-adaptive baselines across a wide range of human behaviors, that interacting with a UHTP-controlled robot reduces the human's cognitive workload, and that humans prefer to work with our adaptive robot over a fixed-policy alternative. [ABSTRACT FROM AUTHOR]

Laboratory automation requires reliable and precise handling of microplates, but existing robotic systems often struggle to achieve this, particularly when navigating around the dynamic and variable nature of laboratory environments. This work introduces a novel method integrating simultaneous localization and mapping (SLAM), computer vision, and tactile feedback for the precise and autonomous placement of microplates. Implemented on a bi-manual mobile robot, the method achieves fine-positioning accuracies of ±1.2 mm and ±0.4°. The approach was validated through experiments using both mockup and real laboratory instruments, demonstrating at least a 95% success rate across varied conditions and robust performance in a multi-stage protocol. Compared to existing methods, our framework effectively generalizes to different instruments without compromising efficiency. These findings highlight the potential for enhanced robotic manipulation in laboratory automation, paving the way for more reliable and reproducible experimental workflows.

Pose estimation is crucial for automating assembly tasks, yet achieving sufficient accuracy for assembly automation remains challenging and part-specific. This paper presents a novel, streamlined approach to pose estimation that facilitates automation of assembly tasks. Our proposed method employs deep learning on a limited number of annotated images to identify a set of keypoints on the parts of interest. To compensate for network shortcomings and enhance accuracy we incorporated a Bayesian updating stage that leverages our detailed knowledge of the assembly part design. This Bayesian updating step refines the network output, significantly improving pose estimation accuracy. For this purpose, we utilized a subset of network-generated keypoint positions with higher quality as measurements, while for the remaining keypoints, the network outputs only serve as priors. The geometry data aid in constructing likelihood functions, which in turn result in enhanced posterior distributions of keypoint pixel positions. We then employed the maximum a posteriori (MAP) estimates of keypoint locations to obtain a final pose, allowing for an update to the nominal assembly trajectory. We evaluated our method on a 14-point snap-fit dash trim assembly for a Ford Mustang dashboard, demonstrating promising results. Our approach does not require tailoring to new applications, nor does it rely on extensive machine learning expertise or large amounts of training data. This makes our method a scalable and adaptable solution for the production floors.

Objects in cluttered environments may have similar sizes and shapes, which remains a huge challenge for robot grasping manipulation. The existing segmentation methods, such as Mask R-CNN and Yolo-v8, tend to lose the shape details of objects when dealing with messy scenes, and this loss of detail limits the grasp performance of robots in complex environments. This paper proposes a high-performance grasp detection algorithm with a multi-target semantic segmentation model, which can effectively improve a robot's grasp success rate in cluttered environments. The algorithm consists of two cascades: Semantic Segmentation and Grasp Detection modules (SS-GD), in which the backbone network of the semantic segmentation module is developed by using the state-of-the-art Swin Transformer structure. It can extract the detailed features of objects in cluttered environments and enable a robot to understand the position and shape of the candidate object. To construct the grasp schema SS-GD focused on important vision features, a grasp detection module is designed based on the Squeeze-and-Excitation (SE) attention mechanism, to predict the corresponding grasp configuration accurately. The grasp detection experiments were conducted on an actual UR5 robot platform to verify the robustness and generalization of the proposed SS-GD method in cluttered environments. A best grasp success rate of 91.7% was achieved for cluttered multi-target workspaces. [ABSTRACT FROM AUTHOR]

This paper proposes the force/torque control strategy for the robotized transcranial magnetic stimulation (TMS) system, considering the shape of the TMS coil case. Hybrid position/force control is used to compensate for the error between the current and target position of the coil and to maintain the contact between the coil and the subject’s head. The desired force magnitude of the force control part of the hybrid controller is scheduled by the error between the current and target position of the TMS coil for fast error reduction and the comfort of the subject. Additionally, the torque proportional to the torque acting on the coil’s center is generated to stabilize the contact. Compliance control, which makes the robot adaptive to the environment, stabilizes the coil and head interaction during force/torque control. The experimental results showed that the force controller made the coil generate a relatively large force for a short time (less than 10 s) for the fast error reduction, and a relatively small interaction force was maintained for the contact. They showed that the torque controller made the contact area inside the coil. The experiment also showed that the proposed strategy could be used for tracking a new target point estimated by the neuronavigation system when the head moved slightly.

Deep reinforcement learning (DRL) can learn an agent’s optimal behavior from the experience it gains through interacting with its environment. However, since the decision-making process of DRL agents is a black-box, it is difficult for users to understand the reasons for the agents’ actions. To date, conventional visual explanation methods for DRL agents have focused only on the policy and not on the state value. In this work, we propose a DRL method called Mask-Attention A3C (Mask A3C) to analyze agents’ decision-making by focusing on both the policy and value branches, which have different outputs. Inspired by the Actor-Critic method, our method introduces an Attention mechanism that applies mask processing to the feature map of the policy and value branches using mask-attention, which is a heat-map representation of the basis for judging the policy and state values. We also propose the introduction of a Mask-attention Loss to obtain highly interpretable mask-attention. By introducing this loss function, the agent learns not to gaze at regions that do not affect its decision-making. Our evaluations with Atari 2600 as a video game strategy task and robot manipulation as a robot control task showed that visualizing the mask-attention of an agent during its action selection facilitates the analysis of the agent’s decision-making. We also investigated the effect of Mask-attention Loss and confirmed that it is useful for analyzing agents’ decision-making. In addition, we showed that these mask-attentions are highly interpretable to the user by conducting a user survey on the prediction of the agent’s behavior.

Reinforcement learning (RL) that autonomously explores optimal control policies has become a crucial direction for developing intelligent robots while Dynamic Movement Primitives (DMPs) serve as a powerful tool for efficiently expressing robot trajectories. This article explores an efficient integration of RL and DMP to enhance the learning efficiency and control performance of reinforcement learning in robot manipulation tasks by focusing on the forms of control actions and their smoothness. A novel approach, DDPG-DMP, is proposed to address the efficiency and feasibility issues in the current RL approaches that employ DMP to generate control actions. The proposed method naturally integrates a DMP-based policy into the actor–critic framework of the traditional RL approach Deep Deterministic Policy Gradient (DDPG) and derives the corresponding update formulas to learn the networks that properly decide the parameters of DMPs. A novel inverse controller is further introduced to adaptively learn the translation from observed states into various robot control signals through DMPs, eliminating the requirement for human prior knowledge. Evaluated on five robot arm control benchmark tasks, DDPG-DMP demonstrates significant advantages in control performance, learning efficiency, and smoothness of robot actions compared to related baselines, highlighting its potential in complex robot control applications.

With the increasing demand for robotic manipulators to operate in complex environments, it is important to develop designs that work in obstacle-rich environments and can navigate around obstacles. This paper aims to demonstrate the capabilities of a bio-inspired, underactuated multilink manipulator in environments with fixed and/or movable obstacles. To simplify the system design, a single rotational actuator is used at the base of the manipulator. We present a modeling method for flexible, multilink underactuated manipulators, including their interaction with obstacles. We also demonstrate how to plan a trajectory for the manipulator in environments with fixed obstacles. The robustness of the manipulator is examined by analyzing the effects of uncertainty in its initial state and the position of obstacles. Next, we demonstrate the performance of the manipulator in environments with movable obstacles and show the advantages of controlling the obstacles' radii and positions. Lastly, we showcase the process of picking up an object in workspaces with obstacles. All the findings are supported by simulations as well as hardware experiments. [ABSTRACT FROM AUTHOR]

Learning visual predictive models has great potential for real-world robot manipulations. Visual predictive models serve as a model of real-world dynamics to comprehend the interactions between the robot and objects. However, prior works in the literature have focused mainly on low-level elementary robot actions, which typically result in lengthy, inefficient, and highly complex robot manipulation. In contrast, humans usually employ top–down thinking of high-level actions rather than bottom–up stacking of low-level ones. To address this limitation, we present a novel formulation for robot manipulation that can be accomplished by pick-and-place, a commonly applied high-level robot action, through grasping. We propose a novel visual predictive model that combines an action decomposer and a video prediction network to learn the intrinsic semantic information of high-level actions. Experiments show that our model can accurately predict the object dynamics (i.e., the object movements under robot manipulation) while trained directly on observations of high-level pick-and-place actions. We also demonstrate that, together with a sampling-based planner, our model achieves a higher success rate using high-level actions on a variety of real robot manipulation tasks. [ABSTRACT FROM AUTHOR]

Trampolines are recognized as a valuable tool in exercise and rehabilitation due to their unique properties like elasticity, rebound force, low-impact exercise, and enhancement of posture, balance, and cardiopulmonary function. To quantitatively assess the effects of trampoline exercises, it is essential to estimate factors such as stiffness, elements influencing jump dynamics, and user safety. Previous studies assessing trampoline characteristics had limitations in performing repetitive experiments at various locations on the trampoline. Therefore, this research introduces a robotic system equipped with foot-shaped jigs to evaluate trampoline stiffness and quantitatively measure exercise effects. This system, through automated, repetitive movements at various locations on the trampoline, accurately measures the elastic coefficient and vertical forces. The robot maneuvers based on the coordinates of the trampoline, as determined by its torque and position sensors. The force sensor measures data related to the force exerted, along with the vertical force data at X, Y, and Z coordinates. The model's accuracy was evaluated using linear regression based on Hooke's Law, with Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Correlation Coefficient Squared (R-squared) metrics. In the analysis including only the distance between X and the foot-shaped jigs, the average MAE, RMSE, and R-squared values were 17.9702, 21.7226, and 0.9840, respectively. Notably, expanding the model to include distances in X, Y, and between the foot-shaped jigs resulted in a decrease in MAE to 15.7347, RMSE to 18.8226, and an increase in R-squared to 0.9854. The integrated model, including distances in X, Y, and between the foot-shaped jigs, showed improved predictive capability with lower MAE and RMSE and higher R-squared, indicating its effectiveness in more accurately predicting trampoline dynamics, vital in fitness and rehabilitation fields. [ABSTRACT FROM AUTHOR]

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills. Qualitative results are made available at https://sites.google.com/stanford.edu/text2motion. [ABSTRACT FROM AUTHOR]

Abstract Learning visual predictive models has great potential for real-world robot manipulations. Visual predictive models serve as a model of real-world dynamics to comprehend the interactions between the robot and objects. However, prior works in the literature have focused mainly on low-level elementary robot actions, which typically result in lengthy, inefficient, and highly complex robot manipulation. In contrast, humans usually employ top–down thinking of high-level actions rather than bottom–up stacking of low-level ones. To address this limitation, we present a novel formulation for robot manipulation that can be accomplished by pick-and-place, a commonly applied high-level robot action, through grasping. We propose a novel visual predictive model that combines an action decomposer and a video prediction network to learn the intrinsic semantic information of high-level actions. Experiments show that our model can accurately predict the object dynamics (i.e., the object movements under robot manipulation) while trained directly on observations of high-level pick-and-place actions. We also demonstrate that, together with a sampling-based planner, our model achieves a higher success rate using high-level actions on a variety of real robot manipulation tasks.

Objects in cluttered environments may have similar sizes and shapes, which remains a huge challenge for robot grasping manipulation. The existing segmentation methods, such as Mask R-CNN and Yolo-v8, tend to lose the shape details of objects when dealing with messy scenes, and this loss of detail limits the grasp performance of robots in complex environments. This paper proposes a high-performance grasp detection algorithm with a multi-target semantic segmentation model, which can effectively improve a robot’s grasp success rate in cluttered environments. The algorithm consists of two cascades: Semantic Segmentation and Grasp Detection modules (SS-GD), in which the backbone network of the semantic segmentation module is developed by using the state-of-the-art Swin Transformer structure. It can extract the detailed features of objects in cluttered environments and enable a robot to understand the position and shape of the candidate object. To construct the grasp schema SS-GD focused on important vision features, a grasp detection module is designed based on the Squeeze-and-Excitation (SE) attention mechanism, to predict the corresponding grasp configuration accurately. The grasp detection experiments were conducted on an actual UR5 robot platform to verify the robustness and generalization of the proposed SS-GD method in cluttered environments. A best grasp success rate of 91.7% was achieved for cluttered multi-target workspaces.

This paper presents Ex-RRT*, a modification of the rapidly-exploring random tree star (RRT*) algorithm that allows the robot to avoid obstacles with a margin. RRT* generates the shortest path to a destination while avoiding obstacles. However, if the robot's embedded trajectory generation algorithm interpolates the waypoints generated by the RRT* to make a motion, collisions may occur with the edges or overhang of obstacles. This algorithm adds a cost function for the distance from each node to the nearest obstacle to ensure that the waypoints generated by the path planner have an appropriate margin for obstacles. It is designed to provide safer control from collisions when each robot's embedded trajectory generation algorithm operates by interpolating waypoints derived by path planning. Through simulation, we compare the proposed Ex-RRT* and conventional RRT* with performance indices such as total distance traveled and collision avoidance norm. Experiments are conducted on the task of moving an object inside a box with a commercial robot to validate the proposed algorithm. The proposed algorithm generates paths with improved safety and can be applied to various robotic arms and mobile platforms. [ABSTRACT FROM AUTHOR]

Abstract In recent years, human hand‐based robotic hands or dexterous hands have gained attention due to their enormous capabilities of handling soft materials compared to traditional grippers. Back in the earlier days, the development of a hand model close to that of a human was an impossible task but with the advancements made in technology, dexterous hands with three, four or five‐fingered robotic hands have been developed to mimic human hand nature. However, human‐like manipulation of dexterous hands to this date remains a challenge. Thus, this review focuses on (a) the history and motivation behind the development of dexterous hands, (b) a brief overview of the available multi‐fingered hands, and (c) learning‐based methods such as traditional and data‐driven learning methods for manipulating dexterous hands. Additionally, it discusses the challenges faced in terms of the manipulation of multi‐fingered or dexterous hands.

Despite the great promise of service robots in everyday tasks, many roboethics issues remain to be addressed before these robots can physically work in human environments. Robot safety is one of the essential concerns for roboethics which is not just a design-time issue. It is also crucial to devise the required onboard monitoring and control strategies to enable robots to be aware of and react to anomalies (i.e., unexpected deviations from intended outcomes) that arise during their operations in the real world. The detection and identification of these anomalies is an essential first step toward fulfilling these requirements. Although several architectures have been proposed for anomaly detection; identification has not yet been thoroughly investigated. This task is challenging since indicators may appear long before anomalies are detected. In this paper, we propose a ConvoLUtional threE-stream Anomaly Identification (CLUE-AI) framework to address this problem. The framework fuses visual, auditory and proprioceptive data streams to identify everyday object manipulation anomalies. A stream of 2D images gathered through an RGB-D camera placed on the head of the robot is processed within a self-attention-enabled visual stage to capture visual anomaly indicators. The auditory modality provided by the microphone placed on the robot’s lower torso is processed within a designed convolutional neural network (CNN) in the auditory stage. Last, the force applied by the gripper and the gripper state is processed within a CNN to obtain proprioceptive features. These outputs are then combined with a late fusion scheme. Our novel three-stream framework design is analyzed on everyday object manipulation tasks with a Baxter humanoid robot in a semi-structured setting. The results indicate that CLUE-AI achieves an f-score of 94%, outperforming the other baselines in classifying anomalies.

This paper introduces a novel method for translating natural-language instructions into executable robot actions using OpenAI’s ChatGPT in a few-shot setting. We propose customizable input prompts for ChatGPT that can easily integrate with robot execution systems or visual recognition programs, adapt to various environments, and create multi-step task plans while mitigating the impact of token limit imposed on ChatGPT. In our approach, ChatGPT receives both instructions and textual environmental data, and outputs a task plan and an updated environment. These environmental data are reused in subsequent task planning, thus eliminating the extensive record-keeping of prior task plans within the prompts of ChatGPT. Experimental results demonstrated the effectiveness of these prompts across various domestic environments, such as manipulations in front of a shelf, a fridge, and a drawer. The conversational capability of ChatGPT allows users to adjust the output via natural-language feedback. Additionally, a quantitative evaluation using VirtualHome showed that our results are comparable to previous studies. Specifically, 36% of task planning met both executability and correctness, and the rate approached 100% after several rounds of feedback. Our experiments revealed that ChatGPT can reasonably plan tasks and estimate post-operation environments without actual experience in object manipulation. Despite the allure of ChatGPT-based task planning in robotics, a standardized methodology remains elusive, making our work a substantial contribution. These prompts can serve as customizable templates, offering practical resources for the robotics research community. Our prompts and source code are open source and publicly available at https://github.com/microsoft/ChatGPT-Robot-Manipulation-Prompts.

In this study, we developed a novel method to determine the optimal viewpoint from which an operator could realize faster and more accurate robot teleoperation using reinforcement learning. The reinforcement learning model was trained using images obtained from several candidate viewpoints from scratch, and the viewpoint at which the model achieved the highest rewards was considered the optimal viewpoint. The target robot, task, and environment were modeled using computer simulations and the candidate viewpoint images were obtained using those simulations. We employed the world model as our reinforcement learning model to maximize rewards in the reaching task of a robot arm. The reward function was designed to encourage the robot arm to reach the target position both quickly and accurately. The experimental results validated the choice of the world model as the reinforcement learning model. Moreover, subject experiments wherein subjects operated a robot arm remotely to reach the target position were conducted. The experiments produced results that strongly aligned with the performance obtained through computer simulations, indicating that the proposed method is capable of selecting the optimal viewpoint without handcrafted design and subject experiments.

Robots operating in household environments should detect and estimate the properties of articulated objects to efficiently perform tasks given by human operators. This paper presents the design and implementation of a system for estimating a point-cloud-based model of a scene, enhanced with information about articulated objects, based on a single RGB-D image. This article describes the neural method used to detect handles, detect and extract fronts, detect rotational joints, and build a point cloud model. It compares various architectures of neural networks to detect handles, the fronts of drawers and cabinets, and estimate rotational joints. In the end, the results are merged to build a 3D model of articulated objects in the environment. [ABSTRACT FROM AUTHOR]

We present a novel framework, multi-view unified reinforcement learning for robotic manipulation (MURM), which efficiently utilizes multiple camera views to train a goal-conditioned policy for a robot to perform complex tasks. The MURM framework consists of three main phases: (i) demo collection from an expert, (ii) representation learning, and (iii) offline reinforcement learning. In the demo collection phase, we design a scripted expert policy that uses privileged information, such as Cartesian coordinates of a target and goal, to solve the tasks. We add noise to the expert policy to provide sufficient interactive information about the environment, as well as suboptimal behavioral trajectories. We designed three tasks in a Pybullet simulation environment, including placing an object in a desired goal position and picking up various objects that are randomly positioned in the environment. In the representation learning phase, we use a vector-quantized variational autoencoder (VQVAE) to learn a more structured latent representation that makes it feasible to train for RL compared to high-dimensional raw images. We train VQVAE models for each distinct camera view and define the best viewpoint settings for training. In the offline reinforcement learning phase, we use the Implicit Q-learning (IQL) algorithm as our baseline and introduce a separated Q-functions method and dropout method that can be implemented in multi-view settings to train the goal-conditioned policy with supervised goal images. We conduct experiments in simulation and show that the single-view baseline fails to solve complex tasks, whereas MURM is successful. [ABSTRACT FROM AUTHOR]

Extended reality (XR), which encompasses Augmented reality (AR), Virtual reality (VR) and Mixed reality(MR) has been extending its value in various fields. Collaborative robot, also known as cobot, has become a major technical trend in manufacturing due to its productivity and efficiency. This study proposes an integrated system of MR and collaborative robot using vision-based camera pose estimation and Vuforia SDK. It utilizes the hand and gesture recognition of a MR device to ensure the intuitive manipulation. The proposed system is expected to be developed as a collision avoidance system using MR techniques in the future. And other types of robot which is available of vision technique can also be employed for intuitive manipulation system as well. [ABSTRACT FROM AUTHOR]

Abstract In this study, we propose a method to automate fruit harvesting with a fruit harvesting robot equipped with robotic arms. Given the future growth of the world population, food shortages are expected to accelerate. Since much of Japan’s agriculture is dependent on imports, it is expected to be greatly affected by this upcoming food shortage. In recent years, the number of agricultural workers in Japan has been decreasing and the population is aging. As a result, there is a need to automate and reduce labor in agricultural work using agricultural machinery. In particular, fruit cultivation requires a lot of manual labor due to the variety of orchard conditions and tree shapes, causing mechanization and automation to lag behind. In this study, a dual-armed fruit harvesting robot was designed and fabricated to reach most of the fruits on joint V-shaped trellis that was cultivated and adjusted for the robot. To harvest the fruit, the fruit harvesting robot uses sensors and computer vision to detect and estimate the position of the fruit and then inserts end-effectors into the lower part of the fruit. During this process, there is a possibility of collision within the robot itself or with other fruits depending on the position of the fruit to be harvested. In this study, inverse kinematics and a fast path planning method using random sampling is used to harvest fruits with robot arms. This method makes it possible to control the robot arms without interfering with the fruit or the other robot arm by considering them as obstacles. Through experiments, this study showed that these methods can be used to detect pears and apples outdoors and automatically harvest them using the robot arms.

Visual object grasping can be complex when dealing with different shapes, points of view, and environments since the robotic manipulator must estimate the most feasible place to grasp. This work proposes a new selective grasping system using only point clouds of objects. For the selection of the object of interest, a deep learning network for object classification is proposed, named Point Encoder Convolution (PEC). The network is trained with a dataset obtained in a realistic simulator and uses an autoencoder with 1D convolution. The developed grasping algorithm used in the system uses geometry primitives and lateral curvatures to estimate the best region to grasp without previously knowing the object's point cloud. Experimental results show a success ratio of 94% for a dataset with five classes, and the proposed visual selective grasping system can be executed in around 0.004 s, suitable for tasks that require a low execution time or use low-cost hardware. [ABSTRACT FROM AUTHOR]

Autonomous indoor service robots are affected by multiple factors when they are directly involved in manipulation tasks in daily life, such as scenes, objects, and actions. It is of self-evident importance to properly parse these factors and interpret intentions according to human cognition and semantics. In this study, the design of a semantic representation framework based on a knowledge graph is presented, including (1) a multi-layer knowledge-representation model, (2) a multi-module knowledge-representation system, and (3) a method to extract manipulation knowledge from multiple sources of information. Moreover, with the aim of generating semantic representations of entities and relations in the knowledge base, a knowledge-graph-embedding method based on graph convolutional neural networks is proposed in order to provide high-precision predictions of factors in manipulation tasks. Through the prediction of action sequences via this embedding method, robots in real-world environments can be effectively guided by the knowledge framework to complete task planning and object-oriented transfer. [ABSTRACT FROM AUTHOR]

In recent years, human hand‐based robotic hands or dexterous hands have gained attention due to their enormous capabilities of handling soft materials compared to traditional grippers. Back in the earlier days, the development of a hand model close to that of a human was an impossible task but with the advancements made in technology, dexterous hands with three, four or five‐fingered robotic hands have been developed to mimic human hand nature. However, human‐like manipulation of dexterous hands to this date remains a challenge. Thus, this review focuses on (a) the history and motivation behind the development of dexterous hands, (b) a brief overview of the available multi‐fingered hands, and (c) learning‐based methods such as traditional and data‐driven learning methods for manipulating dexterous hands. Additionally, it discusses the challenges faced in terms of the manipulation of multi‐fingered or dexterous hands. [ABSTRACT FROM AUTHOR]

In the development of digital agriculture, agricultural robots play a unique role and confer numerous advantages in farming production. From the invention of the first industrial robots in the 1950s, robots have begun to capture the attention of both research and industry. Thanks to the recent advancements in computer science, sensing, and control approaches, agricultural robots have experienced a rapid evolution, relying on various cutting-edge technologies for different application scenarios. Indeed, significant refinements have been achieved by integrating perception, decision-making, control, and execution techniques. However, most agricultural robots continue to require intelligence solutions, limiting them to small-scale applications without quantity production because of their lack of integration with artificial intelligence. Therefore, to help researchers and engineers grasp the prevalent research status of agricultural robots, in this review we refer to more than 100 pieces of literature according to the category of agricultural robots under discussion. In this context, we bring together diverse agricultural robot research statuses and applications and discuss the benefits and challenges involved in further applications. Finally, directional indications are put forward with respect to the research trends relating to agricultural robots. [ABSTRACT FROM AUTHOR]