Search

Joint detection of drivable areas and road anomalies is very important for mobile robots. Recently, many semantic segmentation approaches based on convolutional neural networks (CNNs) have been proposed for pixelwise drivable area and road anomaly detection. In addition, some benchmark datasets, such as KITTI and Cityscapes, have been widely used. However, the existing benchmarks are mostly designed for self-driving cars. There lacks a benchmark for ground mobile robots, such as robotic wheelchairs. Therefore, in this article, we first build a drivable area and road anomaly detection benchmark for ground mobile robots, evaluating existing state-of-the-art (SOTA) single-modal and data-fusion semantic segmentation CNNs using six modalities of visual features. Furthermore, we propose a novel module, referred to as the dynamic fusion module (DFM), which can be easily deployed in existing data-fusion networks to fuse different types of visual features effectively and efficiently. The experimental results show that the transformed disparity image is the most informative visual feature and the proposed DFM-RTFNet outperforms the SOTAs. In addition, our DFM-RTFNet achieves competitive performance on the KITTI road benchmark. [ABSTRACT FROM AUTHOR]

We present a unified model-based and data-driven approach for quadrupedal planning and control to achieve dynamic locomotion over uneven terrain. We utilize on-board proprioceptive and exteroceptive feedback to map sensory information and desired base velocity commands into footstep plans using a reinforcement learning (RL) policy. This RL policy is trained in simulation over a wide range of procedurally generated terrains. When run online, the system tracks the generated footstep plans using a model-based motion controller. We evaluate the robustness of our method over a wide variety of complex terrains. It exhibits behaviors that prioritize stability over aggressive locomotion. Additionally, we introduce two ancillary RL policies for corrective whole-body motion tracking and recovery control. These policies account for changes in physical parameters and external perturbations. We train and evaluate our framework on a complex quadrupedal system, ANYmal version B, and demonstrate transferability to a larger and heavier robot, ANYmal C, without requiring retraining. [ABSTRACT FROM AUTHOR]

Constrained motion planning is a challenging field of research, aiming for computationally efficient methods that can find a collision-free path on the constraint manifolds between a given start and goal configuration. These planning problems come up surprisingly frequently, such as in robot manipulation for performing daily life assistive tasks. However, few solutions to constrained motion planning are available, and those that exist struggle with high computational time complexity in finding a path solution on the manifolds. To address this challenge, we present Constrained Motion Planning Networks X (CoMPNetX). It is a neural planning approach, comprising a conditional deep neural generator and discriminator with neural gradients-based fast projection operator. We also introduce neural task and scene representations conditioned on which the CoMPNetX generates implicit manifold configurations to turbo-charge any underlying classical planner such as sampling-based motion planning methods for quickly solving complex constrained planning tasks. We show that our method finds path solutions with high success rates and lower computation times than state-of-the-art traditional path-finding tools on various challenging scenarios. [ABSTRACT FROM AUTHOR]

The rapid development of autonomous vehicles (AVs) holds vast potential for transportation systems through improved safety, efficiency, and access to mobility. However, the progression of these impacts, as AVs are adopted, is not well understood. Numerous technical challenges arise from the goal of analyzing the partial adoption of autonomy: partial control and observation, multivehicle interactions, and the sheer variety of scenarios represented by real-world networks. To shed light into near-term AV impacts, this article studies the suitability of deep reinforcement learning (RL) for overcoming these challenges in a low AV-adoption regime. A modular learning framework is presented, which leverages deep RL to address complex traffic dynamics. Modules are composed to capture common traffic phenomena (stop-and-go traffic jams, lane changing, intersections). Learned control laws are found to improve upon human driving performance, in terms of system-level velocity, by up to 57% with only 4–7% adoption of AVs. Furthermore, in single-lane traffic, a small neural network control law with only local observation is found to eliminate stop-and-go traffic—surpassing all known model-based controllers to achieve near-optimal performance—and generalize to out-of-distribution traffic densities. [ABSTRACT FROM AUTHOR]

Recent methods for reinforcement learning from images use auxiliary tasks to learn image features that are used by the agent's policy or Q-function. In particular, methods based on contrastive learning that induce linearity of the latent dynamics or invariance to data augmentation have been shown to greatly improve the sample efficiency of the reinforcement learning algorithm and the generalizability of the learned embedding. We further argue, that explicitly improving Markovianity of the learned embedding is desirable and propose a self-supervised representation learning method which integrates contrastive learning with dynamic models to synergistically combine these three objectives: (1) We maximize the InfoNCE bound on the mutual information between the state- and action-embedding and the embedding of the next state to induce a linearly predictive embedding without explicitly learning a linear transition model, (2) we further improve Markovianity of the learned embedding by explicitly learning a non-linear transition model using regression, and (3) we maximize the mutual information between the two nonlinear predictions of the next embeddings based on the current action and two independent augmentations of the current state, which naturally induces transformation invariance not only for the state embedding, but also for the nonlinear transition model. Experimental evaluation on the Deepmind control suite shows that our proposed method achieves higher sample efficiency and better generalization than state-of-art methods based on contrastive learning or reconstruction. [ABSTRACT FROM AUTHOR]

In this article, we show that learned policies can be applied to solve legged locomotion control tasks with extensive flight phases, such as those encountered in space exploration. Using an off-the-shelf deep reinforcement learning algorithm, we train a neural network to control a jumping quadruped robot while solely using its limbs for attitude control. We present tasks of increasing complexity leading to a combination of 3-D (re)orientation and landing locomotion behaviors of a quadruped robot traversing simulated low-gravity celestial bodies. We show that our approach easily generalizes across these tasks and successfully trains policies for each case. Using sim-to-real transfer, we deploy trained policies in the real world on the SpaceBok robot placed on an experimental testbed designed for 2-D microgravity experiments. The experimental results demonstrate that repetitive controlled jumping and landing with natural agility is possible. [ABSTRACT FROM AUTHOR]

Electrical impedance tomography (EIT) based tactile sensor offers significant benefits on practical deployment because of its sparse electrode allocation, including durability, large-area scalability, and low fabrication cost, but the degradation of a tactile spatial resolution has remained challenging. This article describes a deep neural network based EIT reconstruction framework, the EIT neural network (EIT-NN), alleviating this tradeoff between tactile sensing performance and hardware simplicity. EIT-NN learns a computationally efficient, nonlinear reconstruction attribute, achieving high-resolution tactile sensation and well-generalized reconstruction capability to address arbitrary complex touch modalities. We train EIT-NN by presenting a sim-to-real dataset synthesis strategy for computationally efficient generalizability. Furthermore, we propose a spatial sensitivity aware mean-squared error loss function, which uses an intrinsic spatial sensitivity of the sensor to guarantee a well-posed EIT operation. We validate an outperformance of EIT-NN against conventional EIT sensing methods by conducting a simulation study, a single-touch indentation test, and a two-point discrimination test. The results show improved spatial resolution, sensitivity, and localization accuracy. The beneficial features of the generalized sensing of EIT-NN were demonstrated by examining touch modality discrimination performance. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNN) have been used successfully in solving many challenging visual perception tasks facing mobile robots and self-driving cars. To facilitate deploying such models on embedded hardware onboard mobile robots that have limited resources, multitask learning approaches have become common. In typically used multitask learning, a shared encoder network extracts features from inputs whereas multiple task-specific decoders transform these features into their target output. However, properly combining different tasks' losses into the final network loss such that each task is making progress learning is a major challenge in these approaches. In this article, we present an innovative approach to extend a typical single-task network with the capability of performing two tasks without multiple decoders, i.e., a single-stream two-task network. The two output tasks are semantic segmentation and monocular depth prediction which are essential tasks in visual perception for autonomous driving. The method is centered on solving semantic segmentation with a regression loss function rather than a classification one. With our approach, we seize multitask learning benefits of reduced overhead and enhanced generalization while alleviating the need to balance different loss functions. Experimental evaluations with baseline single tasks and a multitask network are presented. [ABSTRACT FROM AUTHOR]

This article describes motion planning networks (MPNet), a computationally efficient, learning-based neural planner for solving motion planning problems.MPNet uses neural networks to learn general near-optimal heuristics for path planning in seen and unseen environments. It takes environment information such as raw point cloud from depth sensors, as well as a robot's initial and desired goal configurations and recursively calls itself to bidirectionally generate connectable paths. In addition to finding directly connectable and near-optimal paths in a single pass, we show that worst-case theoretical guarantees can be proven if we merge this neural network strategy with classical sample-based planners in a hybrid approach while still retaining significant computational and optimality improvements. To train the MPNet models, we present an active continual learning approach that enables MPNet to learn from streaming data and actively ask for expert demonstrations when needed, drastically reducing data for training. We validate MPNet against gold-standard and state-of-the-art planning methods in a variety of problems from two-dimensional to seven-dimensional robot configuration spaces in challenging and cluttered environments, with results showing significant and consistently stronger performance metrics, and motivating neural planning in general as a modern strategy for solving motion planning problems efficiently. [ABSTRACT FROM AUTHOR]

In this article, we are interested in robotic visual object classification using a deep convolutional neural network (DCNN) classifier. We show that the correlation coefficient of the automatically learned DCNN features of two object images carries robust information on their similarity, and can be utilized to significantly improve the robot's classification accuracy, without additional training. More specifically, we first probabilistically analyze how the feature correlation carries vital similarity information and build a correlation-based Markov random field (CoMRF) for joint object labeling. Given query and motion budgets, we then propose an optimization framework to plan the robot's query and path based on our CoMRF. This gives the robot a new way to optimally decide which object sites to move close to for better sensing and for which objects to ask a remote human for help with classification, which considerably improves the overall classification. We extensively evaluate our proposed approach on two large datasets (e.g., drone imagery and indoor scenes) and several real-world robotic experiments. The results show that our proposed approach significantly outperforms the benchmarks. [ABSTRACT FROM AUTHOR]

Over the past few years, we have witnessed a considerable diffusion of data-driven visual odometry (VO) approaches as viable alternatives to standard geometric-based strategies. Their success is mainly related to the improved robustness to image nonideal conditions (e.g., blur, high or low contrast, texture-poor scenarios). However, most of the data-driven State-of-the-Art (SotA) approaches do not provide any kind of information about the uncertainty of their estimates, which is crucial to effectively integrate them into robotic navigation systems. Inspired by this considerations, we propose uncertainty-aware VO (UA-VO), a novel deep neural network (DNN) architecture that computes relative pose predictions by processing sequence of images and, at the same time, provides uncertainty measures about those estimations. The confidence measure computed by UA-VO considers both epistemic and aleatoric uncertainties and accounts for heteroscedasticity, i.e., it is sample-dependent. We assess the benefits of UA-VO with different typology of experiments on three publicly available datasets and on a brand new set of sequences, we gathered to extend the evaluation. [ABSTRACT FROM AUTHOR]

Although the state-of-the-art learning approaches exhibit impressive results for dynamical systems, only a few applications on real physical systems have been presented. One major impediment is that the intermediate policy during the training procedure may result in behaviors that are not only harmful to the system itself but also to the environment. In essence, imposing safety guarantees for learning algorithms is vital for autonomous systems acting in the real world. In this article, we propose a computationally effective and general safe learning framework, specifically for complex dynamical systems. With a proper definition of the safe region, a supervisory control strategy, which switches the actions applied on the system between the learning-based controller and a predefined corrective controller, is given. A simplified system facilitates the estimation of the safe region for the high-dimensional dynamical system. During the learning phase, the belief of the safe region is updated with the actual execution results of the corrective controller, which in turn enables the learning-based controller to have more freedom in choosing its actions. Two examples are given to demonstrate the performance of the proposed framework, one simple inverted pendulum to illustrate the online adaptation method, and one quadcopter control task to show the overall performance. [ABSTRACT FROM AUTHOR]

Among the main challenges in robotics, target-driven visual navigation has gained increasing interest in recent years. In this task, an agent has to navigate in an environment to reach a user specified target, only through vision. Recent fruitful approaches rely on deep reinforcement learning, which has proven to be an effective framework to learn navigation policies. However, current state-of-the-art methods require to retrain, or at least fine-tune, the model for every new environment and object. In real scenarios, this operation can be extremely challenging or even dangerous. For these reasons, we address generalization in target-driven visual navigation by proposing a novel architecture composed of two networks, both exclusively trained in simulation. The first one has the objective of exploring the environment, while the other one of locating the target. They are specifically designed to work together, while separately trained to help generalization. In this article, we test our agent in both simulated and real scenarios, and validate its generalization capabilities through extensive experiments with previously unseen goals and unknown mazes, even much larger than the ones used for training. [ABSTRACT FROM AUTHOR]

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. It is nontrivial to manually design a robot controller that combines these modalities, which have very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to train directly on real robots due to sample complexity. In this article, we use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. Evaluating our method on a peg insertion task, we show that it generalizes over varying geometries, configurations, and clearances, while being robust to external perturbations. We also systematically study different self-supervised learning objectives and representation learning architectures. Results are presented in simulation and on a physical robot. [ABSTRACT FROM AUTHOR]

To collect a human-annotated dataset for training deep convolutional neural networks is a very time-consuming and laborious process. To reduce this burden, we previously proposed an automated annotation by placing one visual marker above the detection target object in the training phase. However, in this approach, occasionally the marker hides the object surface. To avoid this issue, we propose placing a pedestal with multiple markers at the bottom of the object. If we use multiple markers, the object can be annotated even when the object hides some of the markers. Besides that, the simple modification of placing the markers on the bottom allows the use of simple background masking to avoid the neural network learning the remaining markers in the training image as a feature of the object. Background masking can completely remove the markers during the training process. Experiments showed the proposed vision system using our automatic object annotation outperformed the vision system using manual annotation in terms of object detection, orientation estimation, and 2D position estimation while reducing the time required for dataset collection from 16.1 hours to 7.30 hours. [ABSTRACT FROM AUTHOR]

Fast and precise robot motion is needed in many industrial applications. Most industrial robot motion controllers allow externally commanded motion profiles, but the trajectory tracking performance is affected by the robot dynamics and joint servo controllers, to which users have no direct access and about which they have little information. The performance is further compromised by time delays in transmitting the external command as a setpoint to the inner control loop. This paper presents an approach for combining neural networks and iterative learning controls to improve the trajectory tracking performance for a multi-axis articulated industrial robot. For a given desired trajectory, the external command is iteratively refined using a high-fidelity dynamical simulator to compensate for the robot inner-loop dynamics. These desired trajectories and the corresponding refined input trajectories are then used to train multi-layer neural networks to emulate the dynamical inverse of the nonlinear inner-loop dynamics. We show that with a sufficiently rich training set, the trained neural networks generalize well to trajectories beyond the training set as tested in the simulator. In applying the trained neural networks to a physical robot, the tracking performance still improves but not as much as in the simulator. We show that transfer learning effectively bridges the gap between simulation and the physical robot. Finally, we test the trained neural networks on other robot models in simulation and demonstrate the possibility of a general purpose network. Development and evaluation of this methodology are based on the ABB IRB6640-180 industrial robot and ABB RobotStudio software packages.

In this research, a global path planning method based on recurrent neural networks by means of a new Loss function is presented, which regardless of the complexity of the configuration space, generates the path in a relatively constant time. The new Loss function is defined in such a way that in addition to learning the input data of the network, it creates an adjustable safety margin around the obstacles and ultimately creates a safe path. Moreover, a new global path planning method is also introduced, which is used to create the dataset required to train the proposed neural network. The convergence of this method is mathematically proven and it is shown that this method can also produce a suboptimal path in a much shorter time than the common methods of global path planning reported in the literature. In short, the main purpose of this research consists in providing a method which can create a suboptimal, fast and safe path for a mobile robot from any random starting point to any random destination in a known environment. First, the proposed methods will be implemented for different two-dimensional environments consisting of convex and non-convex obstacles, considering the robot as a point-mass, and then it will be implemented in a simulation environment, AI2THOR. Compared to classical global path planning algorithms, such as RRT and A*, the proposed approach demonstrates better performance in complex and challenging environments. • Proposing a novel fast global path planner for generating the LSTM's input path data. • A novel loss function is proposed that helps the LSTM network predict safe paths. • Comparing and evaluating proposed and existing methods in different environments. • AI2THOR simulation environment results validate the effectiveness and feasibility. [ABSTRACT FROM AUTHOR]

Fast and precise robot motion is needed in many industrial applications. Most industrial robot motion controllers allow externally commanded motion profiles, but the trajectory tracking performance is affected by the robot dynamics and joint servo controllers, to which users have no direct access and about which they have little information. The performance is further compromised by time delays in transmitting the external command as a setpoint to the inner control loop. This paper presents an approach for combining neural networks and iterative learning controls to improve the trajectory tracking performance for a multi-axis articulated industrial robot. For a given desired trajectory, the external command is iteratively refined using a high-fidelity dynamical simulator to compensate for the robot inner-loop dynamics. These desired trajectories and the corresponding refined input trajectories are then used to train multi-layer neural networks to emulate the dynamical inverse of the nonlinear inner-loop dynamics. We show that with a sufficiently rich training set, the trained neural networks generalize well to trajectories beyond the training set as tested in the simulator. In applying the trained neural networks to a physical robot, the tracking performance still improves but not as much as in the simulator. We show that transfer learning effectively bridges the gap between simulation and the physical robot. Finally, we test the trained neural networks on other robot models in simulation and demonstrate the possibility of a general purpose network. Development and evaluation of this methodology are based on the ABB IRB6640-180 industrial robot and ABB RobotStudio software packages. [ABSTRACT FROM AUTHOR]