Search

Tracking an object’s 6D pose, while either the object itself or the observing camera is moving, is important for many robotics and augmented reality applications. While exploiting temporal priors eases this problem, object-specific knowledge is required to recover when tracking is lost. Under the tight time constraints of the tracking task, RGB(D)-based methods are often conceptionally complex or rely on heuristic motion models. In comparison, we propose to simplify object tracking to a reinforced point cloud (depth only) alignment task. This allows us to train a streamlined approach from scratch with limited amounts of sparse 3D point clouds, compared to the large datasets of diverse RGBD sequences required in previous works. We incorporate temporal frame-to-frame registration with object-based recovery by frame-to-model refinement using a reinforcement learning (RL) agent that jointly solves for both objectives. We also show that the RL agent’s uncertainty and a rendering-based mask propagation are effective reinitialization triggers.

Robots operating in the real world are expected to detect, classify, segment, and estimate the pose of objects to accomplish their task. Modern approaches using deep learning not only require large volumes of data but also pixel-accurate annotations in order to evaluate the performance and therefore safety of these algorithms. At present, publicly available tools for annotating data are scarce and those that are available rely on depth sensors, which excludes their use for transparent, metallic, and general non-Lambertian objects. To address this issue, we present a novel method for creating valuable datasets that can be used in these more difficult cases. Our key contribution is a purely RGB-based scene-level annotation approach that uses a neural radiance field-based method to automatically align objects. A set of user studies demonstrates the accuracy and speed of our approach over a purely manual or depth sensor assisted pipeline. We provide an open-source implementation of each component and a ROS-based recorder for capturing data with a eye-in-hand robot system. Code will be made available at https://github.com/markus-suchi/3D-DAT.

Detecting changes such as moved, removed, or new objects is the essence for numerous indoor applications in robotics such as tidying-up, patrolling, and fetch/carry tasks. The problem is particularly challenging in open-world scenarios where novel objects may appear at any time. The main idea of this paper is to detect objects from partial 3D reconstructions of interesting areas in the environment. In our pipeline we first identify planes, consider clusters on top as objects, and compute their point-pair-features. They are used to match potential objects and categorize them robustly into static, moved, removed, and novel objects even in the presence of partial object reconstructions and clutter. Our approach dissolves heaps of objects without specific object knowledge, but only with the knowledge acquired from change detection. The evaluation is performed on real-world data that includes challenges affecting the quality of the reconstruction as a result of noisy input data. We present the novel dataset ObChange for quantitative evaluation, and we compare our method against a baseline using learning-based object detection. The results show that, even with a targeted training set, our approach outperforms the baseline for most test cases. Lastly, we also demonstrate our method's effectiveness in real robot experiments.

Observational noise inaccurate segmentation and ambiguity due to symmetry and occlusion lead to inaccurate object pose estimates While depth and RGB based pose refinement approaches increase the accuracy of the resulting pose estimates they are susceptible to ambiguity in the observation as they consider visual alignment We propose to leverage the fact that we often observe static rigid scenes Thus the objects therein need to be under physically plausible poses We show that considering plausibility reduces ambiguity and in consequence allows poses to be more accurately predicted in cluttered environments To this end we extend a recent RL based registration approach towards iterative refinement of object poses Experiments on the LINEMOD and YCB VIDEO datasets demonstrate the state of the art performance of our depth based refinement approach Code is available at github com dornik sporeagent

While object semantic understanding is essential for service robotic tasks, 3D object classification is still an open problem. Learning from artificial 3D models alleviates the cost of the annotation necessary to approach this problem, but today's methods still struggle with the differences between artificial and real 3D data. We conjecture that one of the causes of this issue is the fact that today's methods learn directly from point coordinates, which makes them highly sensitive to scale changes. We propose to learn from a graph of reproducible object parts whose scale is more reliable. In combination with a voting scheme, our approach achieves significantly more robust classification and improves upon state-of-the-art by up to 16% when transferring from artificial to real objects.

Structured light-based depth sensors provide accurate depth information independently of the scene appearance by extracting pattern positions from the captured pixel intensities. Spatial neighborhood encoding, in particular, is a popular structured light approach for off-the-shelf hardware. However, it suffers from the distortion and fragmentation of the projected pattern by the scene’s geometry in the vicinity of a pixel. This forces algorithms to find a delicate balance between depth prediction accuracy and robustness to pattern fragmentation or appearance change. While stereo matching provides more robustness at the expense of accuracy, we show that learning to regress a pixel’s position within the projected pattern is not only more accurate when combined with classification but can be made equally robust. We propose to split the regression problem into smaller classification sub-problems in a coarse-to-fine manner with the use of a weight-adaptive layer that efficiently implements branching per-pixel Multilayer Perceptrons applied to features extracted by a Convolutional Neural Network. As our approach requires full supervision, we train our algorithm on a rendered dataset sufficiently close to the real-world domain. On a separately captured real-world dataset, we show that our network outperforms state-of-the-art and is significantly more robust than other regression-based approaches.

This paper addresses the problem of detecting natural contaminants in freshly shorn wool fleece in RGB images using deep learning-based semantic segmentation. The challenge of inconsistent annotation is overcome by learning the probability of contamination as opposed to a discrete class. From the continuous value predictions, contaminated regions can be extracted by selectively thresholding on the probability of contamination. Furthermore, the imbalance of the class distributions is accounted for by adaptively weighting each pixel’s contribution to the loss function. Results show that the adaptive weight improves the prediction accuracy and overall outperforms learning an approximated representation by quantising the distributions.

For visual assistance systems deployed in an industrial setting, precise object pose estimation is an important task in order to support scene understanding and to enable subsequent grasping and manipulation. Industrial environments are especially challenging since mesh-models are usually available while physical objects are not or are expensive to model. Manufactured objects are often similar in appearance, have limited to no textural cues and exhibit symmetries. Thus, these are especially challenging for recognizers that are meant to provide detection, classification and pose estimation on instance level. A usability study of a recent synthetically trained learning-based recognizer for these particular challenges is conducted. Experiments are performed on the challenging T-LESS dataset due to its relevance for industry.

© 2020, Springer Nature Switzerland AG. Recent methods for 6D pose estimation of objects assume either textured 3D models or real images that cover the entire range of target poses. However, it is difficult to obtain textured 3D models and annotate the poses of objects in real scenarios. This paper proposes a method, Neural Object Learning (NOL), that creates synthetic images of objects in arbitrary poses by combining only a few observations from cluttered images. A novel refinement step is proposed to align inaccurate poses of objects in source images, which results in better quality images. Evaluations performed on two public datasets show that the rendered images created by NOL lead to state-of-the-art performance in comparison to methods that use 13 times the number of real images. Evaluations on our new dataset show multiple objects can be trained and recognized simultaneously using a sequence of a fixed scene.

© 2016 IEEE. Object classification with 3D data is an essential component of any scene understanding method. It has gained significant interest in a variety of communities, most notably in robotics and computer graphics. While the advent of deep learning has progressed the field of 3D object classification, most work using this data type are solely evaluated on CAD model datasets. Consequently, current work does not address the discrepancies existing between real and artificial data. In this work, we examine this gap in an indoor service robotic context by specifically addressing the problem of classification when transferring from artificial CAD models to real reconstructed objects. This is performed by training on ModelNet (CAD models) and evaluating on ScanNet (objects extracted from reconstructed rooms). We show that standard methods do not perform well in this task. We thus introduce a method that carefully samples object parts that are reproducible under various transformations and hence robust. Using graph convolution to classify the composed graph of parts, our method improves upon the baseline. Code is publicly available at https://rgit.acin.tuwien.ac.at/jb.weibel/cad2real_object_clf.

© 2016 IEEE. Accurate and robust object pose estimation for robotics applications requires verification and refinement steps. In this work, we propose to integrate hypotheses verification with object pose refinement guided by physics simulation. This allows the physical plausibility of individual object pose estimates and the stability of the estimated scene to be considered in a unified optimization. The proposed method is able to adapt to scenes of multiple objects and efficiently focuses on refining the most promising object poses in multi-hypotheses scenarios. We call this integrated approach VeREFINE and evaluate it on three datasets with varying scene complexity. The generality of the approach is shown by using three state-of-the-art pose estimators and three baseline refiners. Results show improvements over all baselines and on all datasets. Furthermore, our approach is applied in real-world grasping experiments and outperforms competing methods in terms of grasp success rate. Code is publicly available at github.com/dornik/verefine.

This article presents a method for grasping novel objects by learning from experience. Successful attempts are remembered and then used to guide future grasps such that more reliable grasping is achieved over time. To transfer the learned experience to unseen objects, we introduce the dense geometric correspondence matching network (DGCM-Net). This applies metric learning to encode objects with similar geometry nearby in feature space. Retrieving relevant experience for an unseen object is thus a nearest neighbor search with the encoded feature maps. DGCM-Net also reconstructs 3D-3D correspondences using the view-dependent normalized object coordinate space to transform grasp configurations from retrieved samples to unseen objects. In comparison to baseline methods, our approach achieves an equivalent grasp success rate. However, the baselines are significantly improved when fusing the knowledge from experience with their grasp proposal strategy. Offline experiments with a grasping dataset highlight the capability to transfer grasps to new instances as well as to improve success rate over time from increasing experience. Lastly, by learning task-relevant grasps, our approach can prioritize grasp configurations that enable the functional use of objects.

To enable robots to reason about manipulation of objects and AR applications to present augmented scenes to human users, accurate scene explanations based on objects and their 6D pose are required. With the pose-error functions commonly used to evaluate 6D object pose estimation approaches, the accuracy of estimates is measured by surface alignment of a target object under the estimated and true pose. However, an object floating above the ground may yield the same error as an object translated on the ground by the same magnitude. We argue that, to be intelligible for human observers, pose estimates additionally need to adhere to physical principles. To this end, we provide a definition of physical plausibility in scenes of static rigid objects, derive novel pose-error functions and compare them to existing evaluation approaches in 6D object pose estimation. Code to compute the presented pose-error functions is publicly available at github.com/dornik/plausible-poses.

© 2017, Springer Science+Business Media New York. Classifying objects in complex unknown environments is a challenging problem in robotics and is fundamental in many applications. Modern sensors and sophisticated perception algorithms extract rich 3D textured information, but are limited to the data that are collected from a given location or path. We are interested in closing the loop around perception and planning, in particular to plan paths for better perceptual data, and focus on the problem of planning scanning sequences to improve object classification from range data. We formulate a novel time-constrained active classification problem and propose solution algorithms that employ a variation of Monte Carlo tree search to plan non-myopically. Our algorithms use a particle filter combined with Gaussian process regression to estimate joint distributions of object class and pose. This estimator is used in planning to generate a probabilistic belief about the state of objects in a scene, and also to generate beliefs for predicted sensor observations from future viewpoints. These predictions consider occlusions arising from predicted object positions and shapes. We evaluate our algorithms in simulation, in comparison to passive and greedy strategies. We also describe similar experiments where the algorithms are implemented online, using a mobile ground robot in a farm environment. Results indicate that our non-myopic approach outperforms both passive and myopic strategies, and clearly show the benefit of active perception for outdoor object classification.

© 2015 IEEE. We present an end-to-end method for active object classification in cluttered scenes from RGB-D data. Our algorithms predict the quality of future viewpoints in the form of entropy using both class and pose. Occlusions are explicitly modeled in predicting the visible regions of objects, which modulates the corresponding discriminatory value of a given view. We implement a one-step greedy planner and demonstrate our method online using a mobile robot. We also analyze the performance of our method compared to similar strategies in simulated execution using the Willow Garage dataset. Results show that our active method usefully reduces the number of views required to accurately classify objects in clutter as compared to traditional passive perception.

A multi-robot system is a team of autonomous robots that work together to perform a given task. Multi-robot systems have great potential for use in horticulture applications. Robots have the potential to perform crop surveillance, efficiently apply fertiliser and chemical inputs, and perform weeding and harvesting. In all of these tasks, robots must visit many trees or plants over a large area in a time-sensitive manner. Multi-robot systems are appropriate because many robots can work efficiently in parallel. However, a fundamental challenge to be addressed is how to coordinate the motion of many robots while also respecting resource constraints such as limited energy storage, liquid payload, and harvested product storage. The algorithmic problem of multi-robot coverage planning with resource constraints is similar to the NP-hard vehicle routing problem, but the computational complexity of general resource-constrained coverage remains unknown. We show that one variant of this problem, coverage with fixed replenishment stations and zero queuing time, can be solved in polynomial time using area partitioning and graph search. We present algorithms and analysis for this variant, and demonstrate the behaviour of our algorithms in simulation experiments with up to 100 robots. The robots cover a large area organised as a collection of sub-areas with defined boundaries and row orientations. Robots plan to visit one of several possible replenishment stations in order to satisfy resource constraints. Each robot may replenish itself multiple times throughout its mission. This work is practically applicable to systems where refill time is short relative to working time.

We propose a decentralised variant of Monte Carlo tree search (MCTS) that is suitable for a variety of tasks in multi-robot active perception. Our algorithm allows each robot to optimise its own individual action space by maintaining a probability distribution over plans in the joint-action space. Robots periodically communicate a compressed form of these search trees, which are used to update the locally-stored joint distributions using an optimisation approach inspired by variational methods. Our method admits any objective function defined over robot actions, assumes intermittent communication, and is anytime. We extend the analysis of the standard MCTS for our algorithm and characterise asymptotic convergence under reasonable assumptions. We evaluate the practical performance of our method for generalised team orienteering and active object recognition using real data, and show that it compares favourably to centralised MCTS even with severely degraded communication. These examples support the relevance of our algorithm for real-world active perception with multi-robot systems.

© Springer International Publishing Switzerland 2015. This paper describes the development of small low-cost cooperative robots for sustainable broad-acre agriculture to increase broad-acre crop production and reduce environmental impact. The current focus of the project is to use robotics to deal with resistant weeds, a critical problem for Australian farmers. To keep the overall system affordable our robot uses low-cost cameras and positioning sensors to perform a large scale coverage task while also avoiding obstacles. A multi-robot coordinator assigns parts of a given field to individual robots. The paper describes the modification of an electric vehicle for autonomy and experimental results from one real robot and twelve simulated robots working in coordination for approximately two hours on a 55 hectare field in Emerald Australia. Over this time the real robot ‘sprayed’ 6 hectares missing 2.6% and overlapping 9.7% within its assigned field partition, and successfully avoided three obstacles.

Farmers are under growing pressure to increase production, a challenge that robotics has the potential to address. A possible solution is to replace large farm machinery with numerous smaller robots. However, with a large number of robots it will become increasingly time consuming for the farmer to monitor and control them all, hence the need for an effective user interface and automatic multi-robot coordination. This paper describes the design of a user interface and coverage planner suitable for controlling multiple robots for typical coverage style farm operations. The cross-platform user interface allows the farmer to specify their farm including fields, roads and docking stations. The coverage planner splits the workload between the robots and plans periodic docking. The results for the different multi-robot coverage strategies demonstrate the advantage of the robots sequentially moving between fields rather than freely moving between them. The multi-robot system has been used for a coverage task on a real farm for controlling two real robots and four simulated robots operating for two days.

Information gathering at large spatial scales can be addressed with teams of decentralised robots. Many existing methods search over a limited time horizon and do not provide strong performance guarantees. Near-optimal methods that exploit submodular objective functions have been proposed, given a fixed time budget. We propose a revised problem formulation that seeks to near-optimally maximise information gain quickly. We present a novel, near-optimal polynomial-time decentralised algorithm for multiple robots and analyse the expected path length with respect to the number of robots, the size of the area, and the number of observations. Our approach is based on area partitioning and is practically beneficial in that it allows for superlinear speedup in the time required to maximise the submodular objective function, is decentralised, and is easy to implement. We show extensive simulation results that compare the performance of our algorithm to existing sequential allocation methods.