Search

The visual system of a robot has different requirements depending on the application: it may require high accuracy or reliability, be constrained by limited resources or need fast adaptation to dynamically changing environments. In this work, we focus on the instance segmentation task and provide a comprehensive study of different techniques that allow adapting an object segmentation model in presence of novel objects or different domains. We propose a pipeline for fast instance segmentation learning designed for robotic applications where data come in stream. It is based on an hybrid method leveraging on a pre-trained CNN for feature extraction and fast-to-train Kernel-based classifiers. We also propose a training protocol that allows to shorten the training time by performing feature extraction during the data acquisition. We benchmark the proposed pipeline on two robotics datasets and we deploy it on a real robot, i.e. the iCub humanoid. To this aim, we adapt our method to an incremental setting in which novel objects are learned on-line by the robot. The code to reproduce the experiments is publicly available on GitHub., \copyright 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Human-object interaction is of great relevance for robots to operate in human environments. However, state-of-the-art robotic hands are far from replicating humans skills. It is, therefore, essential to study how humans use their hands to develop similar robotic capabilities. This article presents a deep dive into hand-object interaction and human demonstrations, highlighting the main challenges in this research area and suggesting desirable future developments. To this extent, the article presents a general definition of the hand-object interaction problem together with a concise review for each of the main subproblems involved, namely: sensing, perception, and learning. Furthermore, the article discusses the interplay between these subproblems and describes how their interaction in learning from demonstration contributes to the success of robot manipulation. In this way, the article provides a broad overview of the interdisciplinary approaches necessary for a robotic system to learn new manipulation skills by observing human behavior in the real world., This work is supported by the CHIST-ERA (2014-2020) project InDex and received funding from the Italian Ministry of Education and Research (MIUR), Austrian Science Fund (FWF) under grant agreement No. I3969-N30, Engineering and Physical Sciences Research Council (EPSRC-UK) with reference EP/S032355/1 and Agence Nationale de la Recherche (ANR) under grant agreement No. ANR-18-CHR3-0004. This work is also supported by the ERA-net CHIST-ERA project BURG (PCIN2019-103447).

Conventional approaches to robot navigation in unstructured environments rely on information acquired from the LiDAR mounted on the robot base to detect and avoid obstacles. This approach fails to detect obstacles that are too small, or that are invisible because they are outside the LiDAR’s field of view. A possible strategy is to integrate information from other sensors. In this paper, we explore the possibility of using depth information from a movable RGB-D camera mounted on the head of the robot, and investigate, in particular, active control strategies to effectively scan the environment. Existing works combine RGBD-D and 2D LiDAR data passively by fusing the current point-cloud from the RGB-D camera with the occupancy grid computed from the 2D LiDAR data, while the robot follows a given path. In contrast, we propose an optimization strategy that actively changes the position of the robot’s head, where the camera is mounted, at each point of the given navigation path, thus, we can fully exploit the RGB-D camera to detect, and hence avoid, obstacles undetected by the 2D LiDAR, such as overhanging obstacles or obstacles in blind spots. We validate our approach in both simulation environments to gather statistically significant data and real environments to show the applicability of our method to real robots. The platform used is the humanoid robot R1.

Recent visual pose estimation and tracking solutions provide notable results on popular datasets such as T-LESS and YCB. However, in the real world, we can find ambiguous objects that do not allow exact classification and detection from a single view. In this work, we propose a framework that, given a single view of an object, provides the coordinates of a next viewpoint to discriminate the object against similar ones, if any, and eliminates ambiguities. We also describe a complete pipeline from a real object's scans to the viewpoint selection and classification. We validate our approach with a Franka Emika Panda robot and common household objects featured with ambiguities. We released the source code to reproduce our experiments., Accepted version at 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021)

There is a growing interest in Behavior Trees (BTs) as a tool to describe and implement robot behaviors. BTs were devised in the video game industry and their adoption in robotics resulted in the development of ad-hoc libraries to design and execute BTs that fit complex robotics software architectures. While there is broad consensus on how BTs work, some characteristics rely on the implementation choices done in the specific software library used. In this letter, we outline practical aspects in the adoption of BTs and the solutions devised by the robotics community to fully exploit the advantages of BTs in real robots. We also overview the solutions proposed in open-source libraries used in robotics, we show how BTs fit in a robotic software architecture, and we present a use case example.

Hand-eye calibration of laser profile sensors is the process of extracting the homogeneous transformation between the laser profile sensor frame and the end-effector frame of a robot in order to express the data extracted by the sensor in the robot's global coordinate system. For laser profile scanners this is a challenging procedure, as they provide data only in two dimensions and state-of-the-art calibration procedures require the use of specialised calibration targets. This paper presents a novel method to extract the translation-part of the hand-eye calibration matrix with rotation-part known a priori in a target-agnostic way. Our methodology is applicable to any 2D image or 3D object as a calibration target and can also be performed in situ in the final application. The method is experimentally validated on a real robot-sensor setup with 2D and 3D targets., The first two authors contributed equally to this work. Accepted to the IEEE International Conference on Robotics and Automation (ICRA) 2021

In this paper, we enable automated property verification of deliberative components in robot control architectures. We focus on formalizing the execution context of Behavior Trees (BTs) to provide a scalable, yet formally grounded, methodology to enable runtime verification and prevent unexpected robot behaviors. To this end, we consider a message-passing model that accommodates both synchronous and asynchronous composition of parallel components, in which BTs and other components execute and interact according to the communication patterns commonly adopted in robotic software architectures. We introduce a formal property specification language to encode requirements and build runtime monitors. We performed a set of experiments, both on simulations and on the real robot, demonstrating the feasibility of our approach in a realistic application and its integration in a typical robot software architecture. We also provide an OS-level virtualization environment to reproduce the experiments in the simulated scenario., Comment: Accepted version of IROS2021 conference

Object segmentation is a key component in the visual system of a robot that performs tasks like grasping and object manipulation, especially in presence of occlusions. Like many other computer vision tasks, the adoption of deep architectures has made available algorithms that perform this task with remarkable performance. However, adoption of such algorithms in robotics is hampered by the fact that training requires large amount of computing time and it cannot be performed on-line. In this work, we propose a novel architecture for object segmentation, that overcomes this problem and provides comparable performance in a fraction of the time required by the state-of-the-art methods. Our approach is based on a pre-trained Mask R-CNN, in which various layers have been replaced with a set of classifiers and regressors that are re-trained for a new task. We employ an efficient Kernel-based method that allows for fast training on large scale problems. Our approach is validated on the YCB-Video dataset which is widely adopted in the computer vision and robotics community, demonstrating that we can achieve and even surpass performance of the state-of-the-art, with a significant reduction (${\sim}6\times$) of the training time. The code to reproduce the experiments is publicly available on GitHub., Manuscript accepted to 2021 IEEE International Conference on Robotics and Automation (ICRA)

In this paper, we propose Belief Behavior Trees (BBTs), an extension to Behavior Trees (BTs) that allows to automatically create a policy that controls a robot in partially observable environments. We extend the semantic of BTs to account for the uncertainty that affects both the conditions and action nodes of the BT. The tree gets synthesized following a planning strategy for BTs proposed recently: from a set of goal conditions we iteratively select a goal and find the action, or in general the subtree, that satisfies it. Such action may have preconditions that do not hold. For those preconditions, we find an action or subtree in the same fashion. We extend this approach by including, in the planner, actions that have the purpose to reduce the uncertainty that affects the value of a condition node in the BT (for example, turning on the lights to have better lighting conditions). We demonstrate that BBTs allows task planning with non-deterministic outcomes for actions. We provide experimental validation of our approach in a real robotic scenario and - for sake of reproducibility - in a simulated one., Acccepted to 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Behavior Trees (BTs) are gaining acceptance in robotics to specify action policies at the deliberative level. Their advantages include modularity, ease of use and increasing tool support. In this paper, we define Conditional Behavior Trees (CBTs) as an extension of BTs wherein actions are decorated considering pre- and post-conditions. CBTs improve on basic BTs in that they enable monitoring the execution of single actions by checking pre- and post-conditions, respectively. Since there might exist action sequences wherein some preconditions are violated, CBT executability may depend on the success/failure of specific actions. We developed an encoding of CBT executability into satisfiability of propositional formulas to be checked off-line in a publicly-available tool that computes the encoding for generic CBTs. For the kind of application scenarios and related behavior specifications that we consider, we show that our approach is effective and yields formal guarantees about the executability of deliberative policies designed as CBTs.

Behavior Trees (BTs) have become a popular framework for designing controllers of autonomous agents in the computer game and in the robotics industry. One of the key advantages of BTs lies in their modularity, where independent modules can be composed to create more complex ones. In the classical formulation of BTs, modules can be composed using one of the three operators: Sequence, Fallback, and Parallel. The Parallel operator is rarely used despite its strong potential against other control architectures such as Finite State Machines. This is due to the fact that concurrent actions may lead to unexpected problems similar to the ones experienced in concurrent programming. In this paper, we outline how to tackle the aforementioned problem by introducing Concurrent BTs (CBTs) as a generalization of BTs in which we include the notions of progress and resource usage. We show how CBTs allow safe concurrent executions of actions and we analyze the approach from a mathematical standpoint. To illustrate the use of CBTs, we provide a set of use cases in realistic robotics scenarios.

This work illustrates the design phases leading to the development of a new YARP device interface along with its client/server implementation. In order to obtain a smoother integration and a more reliable software usability, while avoiding common errors during the design phases, a new interface is created in the YARP network when a new family of devices is introduced.

International audience; Tactile sensing technology has achieved much progress in last decades. Recently developed tactile sensing systems have been utilized for implementation of bodily-aware robots, enhancement of robotic hand's dexterous manipulation, development of wearable devices for facilitating the human robot interaction, and so on. While tactile sensing has resulted in many possibilities in robotic research, robot's manipulation is still far from that of its counterpart-human-in term of dexterity. To overcome this issue, we are considering the following three aspects that need to be tackled: - Available tactile sensing technology is struggling to meet the capabilities of humans’ sense of touch; - Tactile data processing algorithms and data-driven learning technology are still immature; - General and robust robot manipulation control frameworks are missing

We report on an extensive study of the benefits and limitations of current deep learning approaches to object recognition in robot vision scenarios, introducing a novel dataset used for our investigation. To avoid the biases in currently available datasets, we consider a natural human-robot interaction setting to design a data-acquisition protocol for visual object recognition on the iCub humanoid robot. Analyzing the performance of off-the-shelf models trained off-line on large-scale image retrieval datasets, we show the necessity for knowledge transfer. We evaluate different ways in which this last step can be done, and identify the major bottlenecks affecting robotic scenarios. By studying both object categorization and identification problems, we highlight key differences between object recognition in robotics applications and in image retrieval tasks, for which the considered deep learning approaches have been originally designed. In a nutshell, our results confirm the remarkable improvements yield by deep learning in this setting, while pointing to specific open challenges that need be addressed for seamless deployment in robotics., 21 pages + supplementary material

In this paper we propose a novel method for in-hand object recognition. The method is composed of a grasp stabilization controller and two exploratory behaviours to capture the shape and the softness of an object. Grasp stabilization plays an important role in recognizing objects. First, it prevents the object from slipping and facilitates the exploration of the object. Second, reaching a stable and repeatable position adds robustness to the learning algorithm and increases invariance with respect to the way in which the robot grasps the object. The stable poses are estimated using a Gaussian mixture model (GMM). We present experimental results showing that using our method the classifier can successfully distinguish 30 objects. We also compare our method with a benchmark experiment, in which the grasp stabilization is disabled. We show, with statistical significance, that our method outperforms the benchmark method.

This paper addresses recursive markerless estimation of a robot's end-effector using visual observations from its cameras. The problem is formulated into the Bayesian framework and addressed using Sequential Monte Carlo (SMC) filtering. We use a 3D rendering engine and Computer Aided Design (CAD) schematics of the robot to virtually create images from the robot's camera viewpoints. These images are then used to extract information and estimate the pose of the end-effector. To this aim, we developed a particle filter for estimating the position and orientation of the robot's end-effector using the Histogram of Oriented Gradient (HOG) descriptors to capture robust characteristic features of shapes in both cameras and rendered images. We implemented the algorithm on the iCub humanoid robot and employed it in a closed-loop reaching scenario. We demonstrate that the tracking is robust to clutter, allows compensating for errors in the robot kinematics and servoing the arm in closed loop using vision.

Stretchable capacitive devices are instrumental for new‐generation multifunctional haptic technologies particularly suited for soft robotics and electronic skin applications. A majority of elongating soft electronics still rely on silicone for building devices or sensors by multiple‐step replication. In this study, fabrication of a reliable elongating parallel‐plate capacitive touch sensor, using nitrile rubber gloves as templates, is demonstrated. Spray coating both sides of a rubber piece cut out of a glove with a conductive polymer suspension carrying dispersed carbon nanofibers (CnFs) or graphene nanoplatelets (GnPs) is sufficient for making electrodes with low sheet resistance values (≈10 Ω sq−1). The electrodes based on CnFs maintain their conductivity up to 100% elongation whereas the GnPs‐based ones form cracks before 60% elongation. However, both electrodes are reliable under elongation levels associated with human joints motility (≈20%). Strikingly, structural damages due to repeated elongation/recovery cycles could be healed through annealing. Haptic sensing characteristics of a stretchable capacitive device by wrapping it around the fingertip of a robotic hand (ICub) are demonstrated. Tactile forces as low as 0.03 N and as high as 5 N can be easily sensed by the device under elongation or over curvilinear surfaces.

Deep Learning (DL) methods are notoriously data hungry. Their adoption in robotics is challenging due to the cost associated with data acquisition and labeling. In this paper we focus on the problem of object detection, i.e. the simultaneous localization and recognition of objects in the scene, for which various DL architectures have been proposed in the literature. We propose to use an automatic annotation procedure, which leverages on human-robot interaction and depth-based segmentation, for the acquisition and labeling of training examples. We fine-tune the Faster R-CNN [36] network with these data acquired by the robot autonomously. We measure the performance on the same dataset and investigate the generalization abilities of the network on different settings and in absence of explicit segmentation, showing good detection performance. Experiments on the iCub humanoid robot [25] show that the proposed strategy is effective and can be used to deploy deep object detection algorithms on a robot.

This paper addresses 6-DOF (degree-of-freedom) tactile localization, i.e. the pose estimation of tridimensional objects given tactile measurements. This estimation problem is fundamental for the operation of autonomous robots that are often required to manipulate and grasp objects whose pose is a-priori unknown. The nature of tactile measurements, the strict time requirements for real-time operation and the multimodality of the involved probability distributions pose remarkable challenges and call for advanced nonlinear filtering techniques. Following a Bayesian approach, this paper proposes a novel and effective algorithm, named Memory Unscented Particle Filter (MUPF), which solves the 6-DOF localization problem recursively in real-time by only exploiting contact point measurements. MUPF combines a modified particle filter that incorporates a sliding memory of past measurements to better handle multimodal distributions, along with the unscented Kalman filter that moves the particles towards regions of the search space that are more likely with the measurements. The performance of the proposed MUPF algorithm has been assessed both in simulation and on a real robotic system equipped with tactile sensors (i.e., the iCub humanoid robot). The experiments show that the algorithm provides accurate and reliable localization even with a low number of particles and, hence, is compatible with real-time requirements.

We consider object recognition in the context of lifelong learning, where a robotic agent learns to discriminate between a growing number of object classes as it accumulates experience about the environment. We propose an incremental variant of the Regularized Least Squares for Classification (RLSC) algorithm, and exploit its structure to seamlessly add new classes to the learned model. The presented algorithm addresses the problem of having an unbalanced proportion of training examples per class, which occurs when new objects are presented to the system for the first time. We evaluate our algorithm on both a machine learning benchmark dataset and two challenging object recognition tasks in a robotic setting. Empirical evidence shows that our approach achieves comparable or higher classification performance than its batch counterpart when classes are unbalanced, while being significantly faster., 8 pages, 3 figures

The complexity of humanoid robots is increasing with the availability of new sensors, embedded CPUs and actuators. This wealth of technologies allows researchers to investigate new problems like whole-body force control, multi-modal human-robot interaction and sensory fusion. Under the hood of these robots, the software architecture has an important role: it allows researchers to get access to the robot functionalities focusing primarily on their research problems, it supports code reuse to minimize development and debugging, especially when new hardware becomes available. But more importantly it allows increasing the complexity of the experiments that can be implemented before system integration becomes unmanageable and debugging draws more resources than research itself. In this paper we illustrate the software architecture of the iCub humanoid robot and the software engineering best practices that have emerged driven by the needs of our research community. We describe the latest developments at the level of the middleware supporting interface definition and automatic code generation, logging, ROS compatibility and channel prioritization. We show the robot abstraction layer and how it has been modified to better address the requirements of the users and to support new hardware as it became available. We also describe the testing framework we have recently adopted for developing code using a test driven methodology. We conclude the paper discussing the lessons we have learned during the past eleven years of software development on the iCub humanoid robot.

Multibody dynamics simulation is widely used for prototyping and testing controllers. However, the transfer of controllers initially developed in simulation to real mechatronics platforms requires updating the code in order to interface with physical sensors and actuators. Due to this strong coupling with specific hardware, the controller re-usability is often severely compromised. In the present contribution, we solve this issue by adding a middleware between the controller and the controlled platform (either real or simulated). This framework decouples the controller from the hardware, allows fast controller developments and eases collaborations on large scale projects. Moreover, it offers the possibility to simultaneously control the real and the simulated robot from a unique controller. This paper presents the interface of the Robotran dynamic simulator with the YARP middleware. Robotran leverages symbolic generation of the multibody equations to provide fast and accurate simulations of multibody systems. The speed and accuracy of Robotran make it possible to test real-time controllers in a realistic simulation environment. This framework is illustrated with applications using the COMAN and WALK-MAN humanoid robots.

Motivated by the impact of superresolution methods for imaging, we undertake a detailed and systematic analysis of localization acuity for a biomimetic fingertip and a flat region of tactile skin. We identify three key factors underlying superresolution that enable the perceptual acuity to surpass the sensor resolution: 1) the sensor is constructed with multiple overlapping, broad but sensitive receptive fields; 2) the tactile perception method interpolates between receptors (taxels) to attain subtaxel acuity; and 3) active perception ensures robustness to unknown initial contact location. All factors follow from active Bayesian perception applied to biomimetic tactile sensors with an elastomeric covering that spreads the contact over multiple taxels. In consequence, we attain extreme superresolution with a 35-fold improvement of localization acuity (0.12 mm) over sensor resolution (4 mm). We envisage that these principles will enable cheap high-acuity tactile sensors that are highly customizable to suit their robotic use. Practical applications encompass any scenario where an end-effector must be placed accurately via the sense of touch.

Real-Time communication are important in all those distributed applications where timing constraints on data proccessing and task executation play a fundamental role. Standards-base software engineering does not yet specify how real-time properties should be integrated into a publish/subscribe middleware. This article describes an approach for integration of priority quality of service in a publish/subscribe middleware. The approach simply leverages the operating system functionalities to provide a framework where specific communication channels can be prioritized at run-time. The quality of service is implemented in YARP (Yet Another Robot Platform) framework and the primarily results of performance tests are presented., In 8th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS)

This paper describes the design and realization of novel tactile sensors based on soft materials and magnetic sensing. In particular, the goal was to realize: 1) soft; 2) robust; 3) small; and 4) low-cost sensors that can be easily fabricated and integrated on robotic devices that interact with the environment. We targeted a number of desired features, the most important being: 1) high sensitivity; 2) low hysteresis; and 3) repeatability. The sensor consists of a silicone body in which a small magnet is immersed; an Hall-effect sensor placed below the silicone body measures the magnetic field generated by the magnet, which changes when the magnet is displaced due to an applied external pressure. Two different versions of the sensor have been manufactured, characterized, and mounted on an anthropomorphic robotic hand. Experiments, in which the hand interacts with real-world objects, are reported.

Independent motion detection aims at identifying elements in the scene whose apparent motion is not due to the robot egomotion. In this work, we propose a method that learns the input-output relationship between the robot motion — described by the position and orientation sensors embedded on the robot — and the sparse visual motion detected by the cameras. We detect independent motion by observing discrepancies (anomalies) between the perceived motion and the motion that is expected given the position and orientation sensors on the robot. We then perform a higher level analysis based on the available disparity map, where we obtain dense profile of the objects moving independently from the robot. We implemented the proposed pipeline on the iCub humanoid robot. In this work, we report a thorough experimental analysis that covers typical laboratory settings, where the effectiveness of the method is demonstrated. The analysis shows in particular the robustness of the method to scene and object variations and to different kinds of robot's movements.

Gaze stabilization is an important requisite for humanoid robots. Previous work on this topic has focused on the integration of inertial and visual information. Little attention has been given to a third component, which is the knowledge that the robot has about its own movement. In this work we propose a comprehensive framework for gaze stabilization in a humanoid robot. We focus on the problem of compensating for disturbances induced in the cameras due to self-generated movements of the robot. In this work we employ two separate signals for stabilization: (1) an anticipatory term obtained from the velocity commands sent to the joints while the robot moves autonomously; (2) a feedback term from the on board gyroscope, which compensates unpredicted external disturbances. We first provide the mathematical formulation to derive the forward and the differential kinematics of the fixation point of the stereo system. We finally test our method on the iCub robot. We show that the stabilization consistently reduces the residual optical flow during the movement of the robot and in presence of external disturbances. We also demonstrate that proper integration of the neck DoF is crucial to achieve correct stabilization., 6 pages, appears in 2014 IEEE-RAS International Conference on Humanoid Robots

Legged robots are typically in rigid contact with the environment at multiple locations, which add a degree of complexity to their control. We present a method to control the motion and a subset of the contact forces of a floating-base robot. We derive a new formulation of the lexicographic optimization problem typically arising in multitask motion/force control frameworks. The structure of the constraints of the problem (i.e. the dynamics of the robot) allows us to find a sparse analytical solution. This leads to an equivalent optimization with reduced computational complexity, comparable to inverse-dynamics based approaches. At the same time, our method preserves the flexibility of optimization based control frameworks. Simulations were carried out to achieve different multi-contact behaviors on a 23-degree-offreedom humanoid robot, validating the presented approach. A comparison with another state-of-the-art control technique with similar computational complexity shows the benefits of our controller, which can eliminate force/torque discontinuities., Pre-print of paper presented at Intelligent Robots and Systems (IROS 2014), IEEE International Conference on, Chicago, USA, 2014

This paper describes a developmental framework for action-driven perception in anthropomorphic robots. The key idea of the framework is that action generation develops the agent's perception of its own body and actions. Action-driven development is critical for identifying changing body parts and understanding the effects of actions in unknown or nonstationary environments. We embedded minimal knowledge into the robot's cognitive system in the form of motor synergies and actions to allow motor exploration. The robot voluntarily generates actions and develops the ability to perceive its own body and the effect that it generates on the environment. The robot, in addition, can compose this kind of learned primitives to perform complex actions and characterize them in terms of their sensory effects. After learning, the robot can recognize manipulative human behaviors with cross-modal anticipation for recovery of unavailable sensory modality, and reproduce the recognized actions afterward. We evaluated the proposed framework in the experiments with a real robot. In the experiments, we achieved autonomous body identification, learning of fixation, reaching and grasping actions, and developmental recognition of human actions as well as their reproduction.