Search

Among the main challenges in robotics, target-driven visual navigation has gained increasing interest in recent years. In this task, an agent has to navigate in an environment to reach a user specified target, only through vision. Recent fruitful approaches rely on deep reinforcement learning, which has proven to be an effective framework to learn navigation policies. However, current state-of-the-art methods require to retrain, or at least fine-tune, the model for every new environment and object. In real scenarios, this operation can be extremely challenging or even dangerous. For these reasons, we address generalization in target-driven visual navigation by proposing a novel architecture composed of two networks, both exclusively trained in simulation. The first one has the objective of exploring the environment, while the other one of locating the target. They are specifically designed to work together, while separately trained to help generalization. In this article, we test our agent in both simulated and real scenarios, and validate its generalization capabilities through extensive experiments with previously unseen goals and unknown mazes, even much larger than the ones used for training.

This letter presents a new learning framework that leverages the knowledge from imitation learning, deep reinforcement learning, and control theories to achieve human-style locomotion that is natural, dynamic, and robust for humanoids. We proposed novel approaches to introduce human bias, i.e. motion capture data and a special Multi-Expert network structure. We used the Multi-Expert network structure to smoothly blend behavioral features, and used the augmented reward design for the task and imitation rewards. Our reward design is composable, tunable, and explainable by using fundamental concepts from conventional humanoid control. We rigorously validated and benchmarked the learning framework which consistently produced robust locomotion behaviors in various test scenarios. Further, we demonstrated the capability of learning robust and versatile policies in the presence of disturbances, such as terrain irregularities and external pushes.

Fast feedback control and safety guarantees are essential in modern robotics. We present an approach that achieves both by combining novel robust model predictive control (MPC) with function approximation via (deep) neural networks (NNs). The result is a new approach for complex tasks with nonlinear, uncertain, and constrained dynamics as are common in robotics. Specifically, we leverage recent results in MPC research to propose a new robust setpoint tracking MPC algorithm, which achieves reliable and safe tracking of a dynamic setpoint while guaranteeing stability and constraint satisfaction. The presented robust MPC scheme constitutes a one-layer approach that unifies the often separated planning and control layers, by directly computing the control command based on a reference and possibly obstacle positions. As a separate contribution, we show how the computation time of the MPC can be drastically reduced by approximating the MPC law with a NN controller. The NN is trained and validated from offline samples of the MPC, yielding statistical guarantees, and used in lieu thereof at run time. Our experiments on a state-of-the-art robot manipulator are the first to show that both the proposed robust and approximate MPC schemes scale to real-world robotic systems., 8 pages, 4 figures

We present a framework for visual action planning of complex manipulation tasks with high-dimensional statespaces, focusing on manipulation of deformable objects. Wepropose a Latent Space Roadmap (LSR) for task planningwhich is a graph-based structure globally capturing the systemdynamics in a low-dimensional latent space. Our frameworkconsists of three parts: (1) a Mapping Module (MM) that mapsobservations given in the form of images into a structuredlatent space extracting the respective states as well as generatesobservations from the latent states, (2) the LSR which builds andconnects clusters containing similar states in order to find thelatent plans between start and goal states extracted by MM, and(3) the Action Proposal Module that complements the latent planfound by the LSR with the corresponding actions. We presenta thorough investigation of our framework on simulated boxstacking and rope/box manipulation tasks, and a folding taskexecuted on a real robot. QC 20211109

In this article, we show that learned policies can be applied to solve legged locomotion control tasks with extensive flight phases, such as those encountered in space exploration. Using an off-the-shelf deep reinforcement learning algorithm, we train a neural network to control a jumping quadruped robot while solely using its limbs for attitude control. We present tasks of increasing complexity leading to a combination of 3-D (re)orientation and landing locomotion behaviors of a quadruped robot traversing simulated low-gravity celestial bodies. We show that our approach easily generalizes across these tasks and successfully trains policies for each case. Using sim-to-real transfer, we deploy trained policies in the real world on the SpaceBok robot placed on an experimental testbed designed for 2-D microgravity experiments. The experimental results demonstrate that repetitive controlled jumping and landing with natural agility is possible., IEEE Transactions on Robotics, 38 (1), ISSN:1552-3098, ISSN:1042-296X, ISSN:1941-0468

Endobronchial intervention is increasingly used as a minimally invasive means of lung intervention. Vision-based localization approaches are often sensitive to image artifacts in bronchoscopic videos. In this letter, a robust navigation system based on a context-aware depth recovery approach for monocular video images is presented. To handle the artifacts, a conditional generative adversarial learning framework is proposed for reliable depth recovery. The accuracy of depth estimation and camera localization is validated on an in vivo dataset. Both quantitative and qualitative results demonstrate that the depth recovered with the proposed method preserves better structural information of airway lumens in the presence of image artifacts, and the improved camera localization accuracy demonstrates its clinical potential for bronchoscopic navigation.

Wir führen ein Framework ein, um Kernel-Approximationen in der Mini-Batch-Einstellung mit Stochastic Gradient Descent (SGD) als Alternative zu Deep Learning zu verwenden. Basierend auf Random Kitchen Sinks bieten wir eine C++ Bibliothek für ML in großem Maßstab. Es enthält eine CPU-optimierte Implementierung des Algorithmus in Le et al. 2013, mit der ungefähre Kernel-Erweiterungen in logarithmisch linearer Zeit berechnet werden können. Der Algorithmus erfordert die Berechnung des Produkts der Matrizen Walsh Hadamard. Es wurde ein cachefreundlicher Fast Walsh Hadamard entwickelt, der eine überzeugende Geschwindigkeit erreicht und die aktuellen Methoden auf dem neuesten Stand der Technik übertrifft. McKernel legt die Grundlage für eine neue Lernarchitektur, die es ermöglicht, eine nichtlineare Klassifizierung in großem Maßstab zu erhalten, die Blitzkernerweiterungen und einen linearen Klassifizierer kombiniert. Es funktioniert in der Mini-Batch-Einstellung analog zu neuronalen Netzen. Wir zeigen die Gültigkeit unserer Methode durch umfangreiche Experimente mit MNIST und FASHION MNIST.Wir schlagen auch eine neue Architektur vor, um die Überparametrisierung in neuronalen Netzen zu reduzieren. Es wird ein Operand für die schnelle Berechnung im Rahmen von Deep Learning eingeführt, der gelernte Gewichte nutzt. Der Formalismus wird ausführlich beschrieben und bietet sowohl eine genaue Aufklärung der Mechanik als auch der theoretischen Implikationen.; We introduce a framework to use kernel approximates in the mini-batch setting with Stochastic Gradient Descent (SGD) as an alternative to Deep Learning. Based on Random Kitchen Sinks, we provide a C++ library for Large-scale ML. It contains a CPU optimized implementation of the algorithm in Le et al. 2013, that allows the computation of approximated kernel expansions in log-linear time. The algorithm requires to compute the product of matrices Walsh Hadamard. A cache friendly Fast Walsh Hadamard that achieves compelling speed and outperforms current state-of-the-art methods has been developed.McKernel establishes the foundation of a new architecture of learning that allows to obtain large-scale non-linear classification combining lightning kernel expansions and a linear classifier. It travails in the mini-batch setting working analogously to Neural Networks. We show the validity of our method through extensive experiments on MNIST and FASHION MNIST.We also propose a new architecture to reduce over-parametrization in Neural Networks. It introduces an operand for rapid computation in the framework of Deep Learning that leverages learned weights. The formalism is described in detail providing both an accurate elucidation of the mechanics and the theoretical implications.; Nous introduisons un framework pour utiliser les méthodes à noyaux dans le paramètre mini-batch avec Stochastic Gradient Descent (SGD) comme alternative à Deep Learning. Basé sur Random Kitchen Sinks, nous fournissons une bibliothèque C ++ pour le ML à grande échelle. Il contient une implémentation optimisée pour le processeur de l'algorithme de Le et al. 2013, qui permet le calcul des extensions approximatives du noyau en temps log-linéaire. L'algorithme nécessite de calculer le produit des matrices de Walsh Hadamard. Un Fast Walsh Hadamard compatible avec le cache, qui atteint une vitesse irréprochable et surpasse les méthodes actuelles de pointe, a été développé. McKernel jette les bases d'une nouvelle architecture d'apprentissage qui permet d'obtenir une classification non linéaire à grande échelle combinant des méthodes à noyaux rapides et un classificateur linéaire. Il fonctionne dans le cadre du mini-lot fonctionnant de manière analogue aux réseaux de neurones. Nous montrons la validité de notre méthode à travers des expériences approfondies sur MNIST et FASHION MNIST. Nous proposons également une nouvelle architecture pour réduire la sur-paramétrisation dans les réseaux de neurones. Il introduit un opérande pour le calcul rapide dans le cadre du Deep Learning qui exploite les poids appris. Le formalisme est décrit en détail, fournissant à la fois une élucidation précise de la mécanique et des implications théoriques.

Wir führen ein Framework ein, um Kernel-Approximationen in der Mini-Batch-Einstellung mit Stochastic Gradient Descent (SGD) als Alternative zu Deep Learning zu verwenden. Basierend auf Random Kitchen Sinks bieten wir eine C++ Bibliothek für ML in großem Maßstab. Es enthält eine CPU-optimierte Implementierung des Algorithmus in Le et al. 2013, mit der ungefähre Kernel-Erweiterungen in logarithmisch linearer Zeit berechnet werden können. Der Algorithmus erfordert die Berechnung des Produkts der Matrizen Walsh Hadamard. Es wurde ein cachefreundlicher Fast Walsh Hadamard entwickelt, der eine überzeugende Geschwindigkeit erreicht und die aktuellen Methoden auf dem neuesten Stand der Technik übertrifft. McKernel legt die Grundlage für eine neue Lernarchitektur, die es ermöglicht, eine nichtlineare Klassifizierung in großem Maßstab zu erhalten, die Blitzkernerweiterungen und einen linearen Klassifizierer kombiniert. Es funktioniert in der Mini-Batch-Einstellung analog zu neuronalen Netzen. Wir zeigen die Gültigkeit unserer Methode durch umfangreiche Experimente mit MNIST und FASHION MNIST.Wir schlagen auch eine neue Architektur vor, um die Überparametrisierung in neuronalen Netzen zu reduzieren. Es wird ein Operand für die schnelle Berechnung im Rahmen von Deep Learning eingeführt, der gelernte Gewichte nutzt. Der Formalismus wird ausführlich beschrieben und bietet sowohl eine genaue Aufklärung der Mechanik als auch der theoretischen Implikationen., We introduce a framework to use kernel approximates in the mini-batch setting with Stochastic Gradient Descent (SGD) as an alternative to Deep Learning. Based on Random Kitchen Sinks, we provide a C++ library for Large-scale ML. It contains a CPU optimized implementation of the algorithm in Le et al. 2013, that allows the computation of approximated kernel expansions in log-linear time. The algorithm requires to compute the product of matrices Walsh Hadamard. A cache friendly Fast Walsh Hadamard that achieves compelling speed and outperforms current state-of-the-art methods has been developed.McKernel establishes the foundation of a new architecture of learning that allows to obtain large-scale non-linear classification combining lightning kernel expansions and a linear classifier. It travails in the mini-batch setting working analogously to Neural Networks. We show the validity of our method through extensive experiments on MNIST and FASHION MNIST.We also propose a new architecture to reduce over-parametrization in Neural Networks. It introduces an operand for rapid computation in the framework of Deep Learning that leverages learned weights. The formalism is described in detail providing both an accurate elucidation of the mechanics and the theoretical implications., Nous introduisons un framework pour utiliser les méthodes à noyaux dans le paramètre mini-batch avec Stochastic Gradient Descent (SGD) comme alternative à Deep Learning. Basé sur Random Kitchen Sinks, nous fournissons une bibliothèque C ++ pour le ML à grande échelle. Il contient une implémentation optimisée pour le processeur de l'algorithme de Le et al. 2013, qui permet le calcul des extensions approximatives du noyau en temps log-linéaire. L'algorithme nécessite de calculer le produit des matrices de Walsh Hadamard. Un Fast Walsh Hadamard compatible avec le cache, qui atteint une vitesse irréprochable et surpasse les méthodes actuelles de pointe, a été développé. McKernel jette les bases d'une nouvelle architecture d'apprentissage qui permet d'obtenir une classification non linéaire à grande échelle combinant des méthodes à noyaux rapides et un classificateur linéaire. Il fonctionne dans le cadre du mini-lot fonctionnant de manière analogue aux réseaux de neurones. Nous montrons la validité de notre méthode à travers des expériences approfondies sur MNIST et FASHION MNIST. Nous proposons également une nouvelle architecture pour réduire la sur-paramétrisation dans les réseaux de neurones. Il introduit un opérande pour le calcul rapide dans le cadre du Deep Learning qui exploite les poids appris. Le formalisme est décrit en détail, fournissant à la fois une élucidation précise de la mécanique et des implications théoriques.

In this work, we present an end-to-end network for fast panoptic segmentation. This network, called Fast Panoptic Segmentation Network (FPSNet), does not require computationally costly instance mask predictions or rule-based merging operations. This is achieved by casting the panoptic task into a custom dense pixel-wise classification task, which assigns a class label or an instance id to each pixel. We evaluate FPSNet on the Cityscapes and Pascal VOC datasets, and find that FPSNet is faster than existing panoptic segmentation methods, while achieving better or similar panoptic segmentation performance. On the Cityscapes validation set, we achieve a Panoptic Quality score of 55.1%, at prediction times of 114 milliseconds for images with a resolution of 1024 × 2048 pixels. For lower resolutions of the Cityscapes dataset and for the Pascal VOC dataset, FPSNet achieves prediction times as low as 45 and 28 milliseconds, respectively.

Vision-based localization of an agent in a map is an important problem in robotics and computer vision. In that context, localization by learning matchable image features is gaining popularity due to recent advances in machine learning. Features that uniquely describe the visual contents of images have a wide range of applications, including image retrieval and understanding. In this work, we propose a method that learns image features targeted for image-retrieval-based localization. Retrieval-based localization has several benefits, such as easy maintenance and quick computation. However, the state-of-the-art features only provide visual similarity scores which do not explicitly reveal the geometric distance between query and retrieved images. Knowing this distance is highly desirable for accurate localization, especially when the reference images are sparsely distributed in the scene. Therefore, we propose a novel loss function for learning image features which are both visually representative and geometrically relatable. This is achieved by guiding the learning process such that the feature and geometric distances between images are directly proportional. In our experiments we show that our features not only offer significantly better localization accuracy, but also allow to estimate the trajectory of a query sequence in absence of the reference images., Implementation available at https://github.com/janinethoma/geometrically_mappable

This work proposes a novel deep network architecture to solve the camera ego-motion estimation problem. A motion estimation network generally learns features similar to optical flow (OF) fields starting from sequences of images. This OF can be described by a lower dimensional latent space. Previous research has shown how to find linear approximations of this space. We propose to use an autoencoder network to find a nonlinear representation of the OF manifold. In addition, we propose to learn the latent space jointly with the estimation task, so that the learned OF features become a more robust description of the OF input. We call this novel architecture latent space visual odometry (LS-VO). The experiments show that LS-VO achieves a considerable increase in performances with respect to baselines, while the number of parameters of the estimation network only slightly increases.

On the pursuit of autonomous flying robots, the scientific community has been developing onboard real-time algorithms for localisation, mapping and planning. Despite recent progress, the available solutions still lack accuracy and robustness in many aspects. While mapping for autonomous cars had a substantive boost using deep-learning techniques to enhance LIDAR measurements using image-based depth completion, the large viewpoint variations experienced by aerial vehicles are still posing major challenges for learning-based mapping approaches. In this paper, we propose a depth completion and uncertainty estimation approach that better handles the challenges of aerial platforms, such as large viewpoint and depth variations, and limited computing resources. The core of our method is a novel compact network that performs both depth completion and confidence estimation using an image-guided approach. Real-time performance onboard a GPU suitable for small flying robots is achieved by sharing deep features between both tasks. Experiments demonstrate that our network outperforms the state-of-the-art in depth completion and uncertainty estimation for single-view methods on mobile GPUs. We further present a new photorealistic aerial depth completion dataset that exhibits more challenging depth completion scenarios than the established indoor and car driving datasets. The dataset includes an open-source, visual-inertial UAV simulator for photo-realistic data generation. Our results show that our network trained on this dataset can be directly deployed on real-world outdoor aerial public datasets without fine-tuning or style transfer., IEEE Robotics and Automation Letters, 5 (2), ISSN:2377-3766

In this work, we address the problem of visual navigation by following instructions. In this task, the robot must interpret a natural language instruction in order to follow a predefined path in a possibly unknown environment. Despite different approaches have been proposed in the last years, they are all based on the assumption that the environment contains objects or other elements that can be used to formulate instructions, such as houses or offices. On the contrary, we focus on situations where the environment objects cannot be used to specify a navigation path. In particular, we consider 3D maze-like environments as our test bench because they can be very large and offer very intricate structures. We show that without reference points, visual navigation and instruction following can be rather challenging, and that standard approaches can not be applied successfully. For this reason, we propose a new architecture that explicitly learns both visual navigation and instruction understanding. We demonstrate with simulated experiments that our method can effectively follow instructions and navigate in previously unseen mazes of various sizes.

We introduce a novel method for oriented place recognition with 3D LiDAR scans. A Convolutional Neural Network is trained to extract compact descriptors from single 3D LiDAR scans. These can be used both to retrieve near-by place candidates from a map, and to estimate the yaw discrepancy needed for bootstrapping local registration methods. We employ a triplet loss function for training and use a hard-negative mining strategy to further increase the performance of our descriptor extractor. In an extensive evaluation on the NCLT and KITTI datasets, we demonstrate that our method outperforms related state-of-the-art approaches based on both data-driven and handcrafted data representation in challenging long-term outdoor conditions., 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), ISBN:978-1-7281-4004-9, ISBN:978-1-7281-4003-2

Fruit counting is a fundamental component for yield estimation.Most of the SotA approaches address the problem using fruit models or by explicitly learning to count. In this paper, we tackle the problem by proposing a framework that learns to count fruits without the need for task-specific supervision labels. The experiments on different varieties of fruits show that our approach reaches performances that are comparable with SotA approaches based on the supervised paradigm., https://youtu.be/p4F3PGE9Lks

Soft hands are robotic systems that embed compliant elements in their mechanical design. This enables an effective adaptation with the items and the environment, and ultimately, an increase in their grasping performance. These hands come with clear advantages in terms of ease-to-use and robustness if compared with classic rigid hands, when operated by a human. However, their potential for autonomous grasping is still largely unexplored, due to the lack of suitable control strategies. To address this issue, in this letter, we propose an approach to enable soft hands to autonomously grasp objects, starting from the observations of human strategies. A classifier realized through a deep neural network takes as input the visual information on the object to be grasped, and predicts which action a human would perform to achieve the goal. This information is hence used to select one among a set of human-inspired primitives, which define the evolution of the soft hand posture as a combination of anticipatory action and touch-based reactive grasp. The architecture is completed by the hardware component, which consists of an RGB camera to look at the scene, a 7-DoF manipulator, and a soft hand. The latter is equipped with inertial measurement units at the fingernails for detecting contact with the object. We extensively tested the proposed architecture with 20 objects, achieving a success rate of 81.1% over 111 grasps.

Deep learning has the potential to have the impact on robot touch that it has had on robot vision. Optical tactile sensors act as a bridge between the subjects by allowing techniques from vision to be applied to touch. In this paper, we apply deep learning to an optical biomimetic tactile sensor, the TacTip, which images an array of papillae (pins) inside its sensing surface analogous to structures within human skin. Our main result is that the application of a deep CNN can give reliable edge perception and thus a robust policy for planning contact points to move around object contours. Robustness is demonstrated over several irregular and compliant objects with both tapping and continuous sliding, using a model trained only by tapping onto a disk. These results relied on using techniques to encourage generalization to tasks beyond which the model was trained. We expect this is a generic problem in practical applications of tactile sensing that deep learning will solve. A video demonstrating the approach can be found at https://www.youtube.com/watch?v=QHrGsG9AHts, Accepted in RAL and ICRA 2019. N. Lepora and J. Lloyd contributed equally to this work