Search

This letter introduces a dataset for training and evaluating methods for 6D pose estimation of hand-held tools in task demonstrations captured by a standard RGB camera. Despite the significant progress of 6D pose estimation methods, their performance is usually limited for heavily occluded objects, which is a common case in imitation learning, where the object is typically partially occluded by the manipulating hand. Currently, there is a lack of datasets that would enable the development of robust 6D pose estimation methods for these conditions. To overcome this problem, we collect a new dataset (Imitrob) aimed at 6D pose estimation in imitation learning and other applications where a human holds a tool and performs a task. The dataset contains image sequences of nine different tools and twelve manipulation tasks with two camera viewpoints, four human subjects, and left/right hand. Each image is accompanied by an accurate ground truth measurement of the 6D object pose obtained by the HTC Vive motion tracking device. The use of the dataset is demonstrated by training and evaluating a recent 6D object pose estimation method (DOPE) in various setups., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Learning & Autonomous Control

With the development of 3D sensors, estimating the point movements between two consecutive point clouds is becoming increasingly attractive. Using correlation-based deep neural networks, existing solutions achieve promising performance. However, these methods couple the non-occluded points and occluded points into the same cost volume and attempt to regress the scene flows of the occluded points from those invalid matching costs directly, which severely suppress the performance of the flow predictor. As an alternative to previous scene flow estimation methods, our method adopts a subnet to predict the occlusion mask and explicitly masks those occluded points, which ensures our flow predictor to focus on estimating the motion flows of non-occluded points from those valid matching costs. Moreover, we further improve the flow estimation performance by employing a local-adaptive cost volume, which can deal with the local structure dissimilarity induced by the sparse sampling of depth sensors (LiDAR). For occluded points, we design an uncertainties-truncated propagation network to propagate the flows of non-occluded to those occluded points. We demonstrate the effectiveness of the proposed method by quantitative and qualitative comparisons with recent baseline works on the FlyingThings3D dataset and the KITTI 2015 dataset, and our results surpass all competing methods by a large margin.

This work explains the use of invariants in robotic perception and control skills. An 'invariant' is a mathematical constraint that remains unchanged under a particular transformation in a system. This property makes robotic functionalities more robust against 'disturbances' that cause these transformation. These invariants are stored in a world model (WM), which has a central role in the information architecture of the robot to share information between components. A 'robotic' football table serves as an example to illustrate the effectiveness of invariants. Despite looking different, all standard football tables satisfy the same set of invariants; the common layout of the field, line markings and puppets, which are not identical but satisfy the same set of constraints, such as parallelism, partial ordering, and relative colour and intensity differences between objects. During initialization, the invariants are actively identified and saved to the world model. During game play this world model is used by the perception and action skills and is updated when necessary. This work shows how the use of invariants creates robustness against variation in placement of the perception system and against variation in table dimensions and colours. When the camera is misaligned or moved mid-play, the world model is updated to ensure a smooth continuation of the game. The approach is tested on three standard football tables, with different dimensions and colours, showing that the approach is robust on standard tables that adhere to the invariants.

Supervised learning with deep convolutional neural networks (DCNNs) has seen huge adoption in stereo matching. However, the acquisition of large-scale datasets with well-labeled ground truth is cumbersome and labor-intensive, making supervised learning-based approaches often hard to implement in practice. To overcome this drawback, we propose a robust and effective self-supervised stereo matching approach, consisting of a pyramid voting module (PVM) and a novel DCNN architecture, referred to as OptStereo. Specifically, our OptStereo first builds multi-scale cost volumes, and then adopts a recurrent unit to iteratively update disparity estimations at high resolution; while our PVM can generate reliable semi-dense disparity images, which can be employed to supervise OptStereo training. Furthermore, we publish the HKUST-Drive dataset, a large-scale synthetic stereo dataset, collected under different illumination and weather conditions for research purposes. Extensive experimental results demonstrate the effectiveness and efficiency of our self-supervised stereo matching approach on the KITTI Stereo benchmarks and our HKUST-Drive dataset. PVStereo, our best-performing implementation, greatly outperforms all other state-of-the-art self-supervised stereo matching approaches. Our project page is available at sites.google.com/view/pvstereo

Curved refractive objects are common in the human environment, and have a complex visual appearance that can cause robotic vision algorithms to fail. Light-field cameras allow us to address this challenge by capturing the view-dependent appearance of such objects in a single exposure. We propose a novel image feature for light fields that detects and describes the patterns of light refracted through curved transparent objects. We derive characteristic points based on these features allowing them to be used in place of conventional 2D features. Using our features, we demonstrate improved structure-from-motion performance in challenging scenes containing refractive objects, including quantitative evaluations that show improved camera pose estimates and 3D reconstructions. Additionally, our methods converge 15-35% more frequently than the state-of-The-Art. Our method is a critical step towards allowing robots to operate around refractive objects, with applications in manufacturing, quality assurance, pick-And-place, and domestic robots working with acrylic, glass and other transparent materials.

Visual Place Recognition (VPR) is the task of matching current visual imagery from a camera to images stored in a reference map of the environment. While initial VPR systems used simple direct image methods or hand-crafted visual features, recent work has focused on learning more powerful visual features and further improving performance through either some form of sequential matcher / filter or a hierarchical matching process. In both cases the performance of the initial single-image based system is still far from perfect, putting significant pressure on the sequence matching or (in the case of hierarchical systems) pose refinement stages. In this paper we present a novel hybrid system that creates a high performance initial match hypothesis generator using short learnt sequential descriptors, which enable selective control sequential score aggregation using single image learnt descriptors. Sequential descriptors are generated using a temporal convolutional network dubbed SeqNet, encoding short image sequences using 1-D convolutions, which are then matched against the corresponding temporal descriptors from the reference dataset to provide an ordered list of place match hypotheses. We then perform selective sequential score aggregation using shortlisted single image learnt descriptors from a separate pipeline to produce an overall place match hypothesis. Comprehensive experiments on challenging benchmark datasets demonstrate the proposed method outperforming recent state-of-the-art methods using the same amount of sequential information. Source code and supplementary material can be found online.11[Online]. Available: https://github.com/oravus/seqNet.

Deep neural networks designed for vision tasks are often prone to failure when they encounter environmental conditions not covered by the training data. Single-modal strategies are insufficient when the sensor fails to acquire information due to malfunction or its design limitations. Multi-sensor configurations are known to provide redundancy, increase reliability, and are crucial in achieving robustness against asymmetric sensor failures. To address the issue of changing lighting conditions and asymmetric sensor degradation in object detection, we develop a multi-modal 2D object detector, and propose deterministic and stochastic sensor-aware feature fusion strategies. The proposed fusion mechanisms are driven by the estimated sensor measurement reliability values/weights. Reliable object detection in harsh lighting conditions is essential for applications such as self-driving vehicles and human-robot interaction. We also propose a new 'r-blended' hybrid depth modality for RGB-D sensors. Through extensive experimentation, we show that the proposed strategies outperform the existing state-of-the-art methods on the FLIR-Thermal dataset, and obtain promising results on the SUNRGB-D dataset. We additionally record a new RGB-Infra indoor dataset, namely L515-Indoors, and demonstrate that the proposed object detection methodologies are highly effective for a variety of lighting conditions., Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public., Learning & Autonomous Control

Visual place recognition is challenging because there are so many factors that can cause the appearance of a place to change, from day-night cycles to seasonal change to atmospheric conditions. In recent years a large range of approaches have been developed to address this challenge including deep-learnt image descriptors, domain translation, and sequential filtering, all with shortcomings including generality and velocity-sensitivity. In this letter we propose a novel descriptor derived from tracking changes in any learned global descriptor over time, dubbed Delta Descriptors. Delta Descriptors mitigate the offsets induced in the original descriptor matching space in an unsupervised manner by considering temporal differences across places observed along a route. Like all other approaches, Delta Descriptors have a shortcoming - volatility on a frame to frame basis - which can be overcome by combining them with sequential filtering methods. Using two benchmark datasets, we first demonstrate the high performance of Delta Descriptors in isolation, before showing new state-of-the-art performance when combined with sequence-based matching. We also present results demonstrating the approach working with four different underlying descriptor types, and two other beneficial properties of Delta Descriptors in comparison to existing techniques: their increased inherent robustness to variations in camera motion and a reduced rate of performance degradation as dimensional reduction is applied. Source code is made available at https://github.com/oravus/DeltaDescriptors.

This letter presents a novel, compute-efficient and training-free approach based on Histogram-of-Oriented-Gradients (HOG) descriptor for achieving state-of-the-art performance-per-compute-unit in Visual Place Recognition (VPR). The inspiration for this approach (namely CoHOG) is based on the convolutional scanning and regions-based feature extraction employed by Convolutional Neural Networks (CNNs). By using image entropy to extract regions-of-interest (ROI) and regional-convolutional descriptor matching, our technique performs successful place recognition in changing environments. We use viewpoint- and appearance-variant public VPR datasets to report this matching performance, at lower RAM commitment, zero training requirements and 20 times lesser feature encoding time compared to state-of-the-art neural networks. We also discuss the image retrieval time of CoHOG and the effect of CoHOG's parametric variation on its place matching performance and encoding time.

Many mine-site processes such as vehicle operation require localisation systems that are reliable, robust and work in a range of environmental conditions. In underground operations, GPS is not available: solutions instead rely on static infrastructure or expensive, laser-based solutions with limited operational capability. In this letter we present a new vision-based technique, Tunnel-IMage Texturally-Accorded Mosaic (TIMTAM), for sub-metre, infrastructure-free localisation in underground mining environments using a single camera. Our approach stitches upward-facing camera images to form planar mosaic maps, using locations generated by the coarse mapping engine based on a small number of manually anchored locations. Localisation is achieved by refining coarse location estimations with a best fit pixel location for the query image within a search neighbourhood in the mosaic map. Our direct pixel-based method is more robust to the challenging illumination and surface-texture environments encountered in underground mine operations than feature-based techniques. Localisation refinement is only triggered when a confidence threshold for the estimate is exceeded. The system is evaluated in a real world mine tunnel, with results showing that the confidence threshold approach is predictive of the quality of the location estimate refinement, and achieves a reduction in mean localisation metric error of up to ∼ 66% from simulated coarse results.

To be effective, robots will need to reliably operate in scenes with refractive objects in a variety of applications; however, refractive objects can cause many robotic vision algorithms, such as structure from motion, to become unreliable or even fail. We propose a novel method to distinguish between refracted and Lambertian image features using a light field camera. While previous refracted feature detection methods are limited to light field cameras with large baselines relative to the refractive object, our method achieves comparable performance, and we extend these capabilities to light field cameras with much smaller baselines than previously considered, where we achieve up to 50% higher refracted feature detection rates. Specifically, we propose to use textural cross correlation to characterize apparent feature motion in a single light field, and compare this motion to its Lambertian equivalent based on 4-D light field geometry. For structure from motion, we demonstrate that rejecting refracted features using our distinguisher yields lower reprojection error, lower failure rates, and more accurate pose estimates when the robot is approaching refractive objects. Our method is a critical step toward allowing robots to operate in the presence of refractive objects.

The commercial use of unmanned aerial vehicles (UAVs) would be enhanced by an ability to sense and avoid potential mid-air collision threats. In this paper we propose a new approach to aircraft detection for long range, vision-based sense and avoid. We first train a deep convolutional neural network to learn aircraft visual features using flight data of mid-air, head-on near collision course encounters between two fixed-wing aircraft. We then propose an approach that fuses these learnt aircraft features with hand-crafted features that are used by the current state of the art. Finally, we evaluate the performance of our proposed approach on real flight data captured from a UAV where it achieves a mean detection range of 2527m and a mean detection range improvement of 299m (or 13.4%) compared to the current state of the art with no additional false alarms.

We propose a novel approach for training deep convolutional neural networks (DCNNs) that allows us to tradeoff complexity and accuracy to learn lightweight models suitable for robotic platforms such as AgBot II (which performs automated weed management). Our approach consists of three stages, the first is to adapt a pre-trained model to the task at hand. This provides state-of-the-art performance but at the cost of high computational complexity resulting in a low frame rate of just 0.12 frames per second (fps). Second, we use the adapted model and employ model compression techniques to learn a lightweight DCNN that is less accurate but has two orders of magnitude fewer parameters. Third, $K$ lightweight models are combined as a mixture model to further enhance the performance of the lightweight models. Applied to the challenging task of weed segmentation, we improve the accuracy from 85.9%, using a traditional approach, to 93.9% by adapting a complicated pre-trained DCNN with 25M parameters (Inception-v3). The downside to this adapted model, Adapted-IV3, is that it can only process 0.12 fps. To make this approach fast while still retaining accuracy, we learn lightweight DCNNs which when combined can achieve accuracy greater than 90% while using considerably fewer parameters capable of processing between 1.07 and 1.83 fps, up to an order of magnitude faster and up to an order of magnitude fewer parameters.

This paper proposes the first derivation, implementation, and experimental validation of light field image-based visual servoing. Light field image Jacobians are derived based on a compact light field feature representation that is close to the form measured directly by light field cameras. We also enhance feature detection and correspondence by enforcing light field geometry constraints, and directly estimate the image Jacobian without knowledge of point depth. The proposed approach is implemented over a standard visual servoing control loop, and applied to a custom-mirror-based light field camera mounted on a robotic arm. Light field image-based visual servoing is then validated in both simulation and experiment. We show that the proposed method outperforms conventional monocular and stereo image-based visual servoing under field-of-view constraints and occlusions.

This paper presents a 3D visual detection method for the challenging task of detecting peduncles of sweet peppers (Capsicum annuum) in the field. Cutting the peduncle cleanly is one of the most difficult stages of the harvesting process, where the peduncle is the part of the crop that attaches it to the main stem of the plant. Accurate peduncle detection in 3D space is therefore a vital step in reliable autonomous harvesting of sweet peppers, as this can lead to precise cutting while avoiding damage to the surrounding plant. This paper makes use of both colour and geometry information acquired from an RGB-D sensor and utilises a supervised-learning approach for the peduncle detection task. The performance of the proposed method is demonstrated and evaluated using qualitative and quantitative results (the Area-Under-the-Curve (AUC) of the detection precision-recall curve). We are able to achieve an AUC of 0.71 for peduncle detection on field-grown sweet peppers. We release a set of manually annotated 3D sweet pepper and peduncle images to assist the research community in performing further research on this topic.

Hand-crafted feature functions are usually designed based on the domain knowledge of a presumably controlled environment and often fail to generalize, as the statistics of real-world data cannot always be modeled correctly. Data-driven feature learning methods, on the other hand, have emerged as an alternative that often generalize better in uncontrolled environments. We present a simple, yet robust, 2-D convolutional neural network extended to a concatenated 3-D network that learns to extract features from the spatio-temporal domain of raw video data. The resulting network model is used for content-based recognition of videos. Relying on a 2-D convolutional neural network allows us to exploit a pretrained network as a descriptor that yielded the best results on the largest and challenging ILSVRC-2014 dataset. Experimental results on commonly used benchmarking video datasets demonstrate that our results are state-of-the-art in terms of accuracy and computational time without requiring any preprocessing (e.g., optic flow) or a priori knowledge on data capture (e.g., camera motion estimation), which makes it more general and flexible than other approaches. Our implementation is made available.

© 20xx IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works., Hand-crafted feature functions are usually designed based on the domain knowledge of a presumably controlled environment and often fail to generalize, as the statistics of real-world data cannot always be modeled correctly. Data-driven feature learning methods, on the other hand, have emerged as an alternative that often generalize better in uncontrolled environments. We present a simple, yet robust, 2D convolutional neural network extended to a concatenated 3D network that learns to extract features from the spatio-temporal domain of raw video data. The resulting network model is used for content-based recognition of videos. Relying on a 2D convolutional neural network allows us to exploit a pretrained network as a descriptor that yielded the best results on the largest and challenging ILSVRC-2014 dataset. Experimental results on commonly used benchmarking video datasets demonstrate that our results are state-of-the-art in terms of accuracy and computational time without requiring any preprocessing (e.g., optic flow) or a priori knowledge on data capture (e.g., camera motion estimation), which makes it more general and flexible than other approaches. Our implementation is made available., Peer Reviewed, Postprint (author's final draft)