Search

The traditional visual SLAM systems take the monocular or stereo camera as input sensor, with complex map initialization and map point triangulation steps needed for 3D map reconstruction, which are easy to fail, computationally complex and can cause noisy measurements. The emergence of RGB-D camera which provides RGB image together with depth information breaks this situation. While a number of RGB-D SLAM systems have been proposed in recent years, the current classification research on RGB-D SLAM is very lacking, and their advantages and shortcomings remain unclear regarding different applications and perturbations, such as illumination transformation, noise and rolling shutter effect of sensors. In this paper, we mainly introduced the basic concept and structure of the RGB-D SLAM system, and then introduced the differences between the various RGB-D SLAM systems in the three aspects of tracking, mapping, and loop detection, and we make a classification study on different RGB-D SLAM algorithms according to the three aspect. Furthermore, we discuss some advanced topics and open problems of RGB-D SLAM, hoping that it will help for future exploring. In the end, we conducted a large number of evaluation experiments on multiple RGB-D SLAM systems, and analyzed their advantages and disadvantages, as well as performance differences in different application scenarios, and provided references for researchers and developers.

We propose a novel monocular depth estimator, which improves the prediction accuracy on human regions by utilizing pose information. The proposed algorithm consists of two networks — PoseNet and DepthNet — to estimate keypoint heatmaps and a depth map, respectively. We incorporate the pose information from PoseNet to improve the depth estimation performance of DepthNet. Specifically, we develop the feature blending block, which fuses the features from PoseNet and DepthNet and feeds them into the next layer of DepthNet, to make the networks learn to predict the depths of human regions more accurately. Furthermore, we develop a novel joint training scheme using partially labeled datasets, which balances multiple loss functions effectively by adjusting weights. Experimental results demonstrate that the proposed algorithm can improve depth estimation performance significantly, especially around human regions. For example, the proposed algorithm improves the depth estimation performance on the human regions of ResNet-50 by 2.8% and 7.0% in terms of $\delta _{1}$ and RMSE, respectively, on the proposed HD + P dataset.

Early detection of colorectal cancer (CRC) can reduce the risk of death. Polyps are the precursor to such cancer. Analyzing the polyps from the most significant frames out of thousands of endoscopy frames is vital for diagnosing and understanding disease. In this article, a deep learning-based monocular depth estimation (MDE) technique is proposed to select the most informative frames (key-frames) of an endoscopic video. In most cases, ground truth depth maps of polyps are not readily available, and that is why the transfer learning approach is adopted in our method. An endoscopic modality generally captures thousands of frames. In this scenario, it is quite essential to discard low-quality and clinically irrelevant frames of an endoscopic video while the most informative frames should be retained for clinical diagnosis. In this view, a key-frame selection strategy is proposed by utilizing the depth information of polyps. In our method, image moment, edge magnitude, and key points are considered for adaptively selecting the key-frames. One important application of our proposed method could be the 3D reconstruction of polyps with the help of extracted key-frames. It gives a surgeon a real-time 3D view of the polyp surface for resection which involves detaching the polyp from its mucosa layer. Also, polyps are localized with the help of extracted depth maps.

The development of simultaneous localization and mapping (SLAM) technology plays an important role in robot navigation and autonomous vehicle innovation. The ORB-SLAM2 is a unified SLAM solution for monocular, binocular, and RGBD cameras which constructs a sparse feature point map for real-time positioning. However, a sparse map based approach cannot effectively meet the requirements of robot navigation, environment reconstruction, and other tasks. In this paper, a dense mapping thread is added to the existing ORB-SLAM2 system. The depth map and color image obtained by the stereo matching of a binocular camera are used to generate a three-dimensional point cloud for keyframes; then, the point cloud is fused by tracking and optimizing the motion track of a feature frame to obtain a real-time point cloud map. Through the experiments conducted on the KITTI dataset and the real environment under the ROS, it is proved that the proposed system constructs a clear three-dimensional point cloud map while constructing an accurate trajectory.

This article demonstrates the importance of visual information on depth perception from monocular motion parallax presented at the time of change in the motion direction of head and stimulus movements. In head-tracking systems, a longer delay time between the head and stimulus movements degrades the depth perception from the monocular motion parallax. Because this delay is noticeable at this time, we hypothesized that the visual information given at the time of the direction change plays a critical role in the depth perception from motion parallax. We evaluated depth perception from monocular motion parallax with and without a visual stimulus at the time of the motion direction change to confirm our hypothesis and clarified that stable and unambiguous depth can be perceived by presenting the change of the stimulus motion direction. We also demonstrated that it is the change in motion direction itself that is important rather than the temporal stop between deceleration and acceleration of the stimulus motion.

The key issue in monocular depth estimation is how to construct the depth image better and improve the quality of the depth map. At present, most of the monocular depth estimation methods based on deep learning manipulate images at low resolution that leads to loss of detail and blurring of boundaries. Nevertheless, deep learning with a large number of parameters needs highly computational complexity, which makes it difficult to apply high-resolution (HR) images to the depth estimate. In this work, model accuracy and runtime are two important factors to be considered. To improve the depth map quality and reduce the running time of the network, we introduce super-resolution techniques as methods of up-sampling to generate high-quality depth images at a faster rate for the depth estimation network. A novel approach is proposed for collecting high-level features that are captured under different receptive fields. The gated multi-scale decoder allows us to effectively filter information by the gated module. By combining the gated module to aid the super resolution of depth images, our method reduces memory consumption while improves reconstruction quality. Experiment results on the challenging NYU Depth v2 dataset demonstrate that both contributions provide significant performance gains over the state-of-the-art in self-supervised depth estimation.

Image-based rendering (IBR) attempts to synthesize novel views using a set of observed images. Some IBR approaches (such as light fields) have yielded impressive high-quality results on small-scale scenes with dense photo capture. However, available wide-baseline IBR methods are still restricted by the low geometric accuracy and completeness of multi-view stereo (MVS) reconstruction on low-textured and non-Lambertian surfaces. The issues become more significant in large-scale outdoor scenes due to challenging scene content, e.g., buildings, trees, and sky. To address these problems, we present a novel IBR algorithm that consists of two key components. First, we propose a novel depth refinement method that combines MVS depth maps with monocular depth maps predicted via deep learning. A lookup table remap is proposed for converting the scale of the monocular depths to be consistent with the scale of the MVS depths. Then, the rescaled monocular depth is used as the constraint in the minimum spanning tree (MST)-based nonlocal filter to refine the per-view MVS depth. Second, we present an efficient shape-preserving warping algorithm that uses superpixels to generate the warped images and blend expected novel views of scenes. The proposed method has been evaluated on public MVS and view synthesis datasets, as well as newly captured large-scale outdoor datasets. In comparison with state-of-the-art methods, the experimental results demonstrated that the proposed method can obtain more complete and reliable depth maps for the challenging large-scale outdoor scenes, thereby resulting in more promising novel view synthesis.

In order to obtain the depth information of the target in the scene and realize three-dimensional (3D) reconstruction, in this paper, a target reconstruction method combining monocular focus stack image and deep neural network is proposed. This method makes full use of the advantages of light field imaging technology and can generate the all focus image. The method first collects multiple frames of continuous images at different focal lengths of the scene, using a divide and conquer algorithm strategy, uplink uses YOLO neural network to identify the target in 3D space and track the position information; the downlink reconstructs the four-dimensional (4D) light field data based on the focus stack image frequency domain back projection, and then uses light field imaging technology to invert the scene parallax; subsequently, achieve scene depth estimation and reconstruction of all focus image; finally, the uplink and downlink are merged to realize the reconstruction of the 3D point cloud of the space target. Experimental results on real scenes show the effectiveness of the proposed algorithm.

Depth estimation is a classic computer vision task and provides rich representation of objects and environment. In recent years, the performance of end-to-end depth estimation has been significantly improved. However, the stack of convolutions and pooling operations result in losing local detail spatial information, which is extremely important to monocular depth estimation. In order to overcome this problem, in this work, we propose an encoder-decoder framework with skip connections. Based on the self-attention mechanism, we apply the channel-spatial attention module as a transition layer, which captures the depth and spatial positional relationship and improves the presentation ability of channel and space. Then we propose a dense decoding module to make full use of the attention features of different scale ranges in the decoding process. It achieves a more massive and denser receptive field while obtaining multi-scale information. Finally, a novel distance-aware loss is introduced to predict more meticulous edges and local details in the distance. Experiments demonstrate that the proposed method outperforms the state-of-the-art on KITTI and NYU Depth V2 datasets.

In large-scale environments, scale drift is a crucial problem of monocular visual simultaneous localization and mapping (SLAM). A common solution is to utilize the camera height, which can be obtained using the reconstructed 3D ground points (3DGPs) from two successive frames, as prior knowledge. Increasing the number of 3DGPs by using more proceeding frames can be a natural extension of this solution to estimate a more precise camera height. However, merely employing multiple frames based on conventional methods is hard to be directly applicable in a real-world scenario because the vehicle motion and inaccurate feature matching inevitably cause large uncertainty and noisy 3DGPs. In this study, we propose an elaborate method to collect confident 3DGPs from multiple frames for robust scale estimation. First, we gather 3DGP candidates that can be seen in more than a predefined number of frames. To verify the 3DGP candidates, we filter out the 3D points at the exterior of the road region obtained by the deep-learning-based road segmentation model. In addition, we formulate an optimization problem constrained by a simple but effective geometric assumption that the normal vector of the ground plane lies in the null space of a movement vector of the camera center, and provide a closed-form solution. ORB-SLAM with the proposed scale estimation method achieves the average translation error with 1.19% on the KITTI dataset, which outperforms the state-of-the-art conventional monocular visual SLAM methods in road driving scenarios.

Monocular depth estimation is a foundation task of three-dimensional (3D) reconstruction which is used to improve the accuracy of environment perception. Because of the simpler hardware requirement, it is more suitable than other multi-view methods. In this study, a new monocular depth estimation algorithm based on graph convolution network (GCN) is proposed. The pixel-wise depth relationship is introduced into conventional convolution neural network (CNN) to make up the disadvantage of processing non-Euclidian data. And the remaining depth topological graph information on the spatial latent variables are extracted based on a multi-scale reconstruction strategy. The final results on NYU-v2 depth dataset and KITTI depth dataset demonstrate that our algorithm improves the quality of monocular depth estimation, especially there are several little objects coexisting in the scenes.

Augmented reality applications running on smartphones or tablets are becoming increasingly common. It is crucial to extract the physical structure of the scene perceived by the device camera in these applications. In such applications, employed in education, navigation and obstacle notification, the distance information between the device camera and the object must be derived and processed with sufficient accuracy and speed. In this study, the deep learning model, named "From Big To Small (BTS)", with superior performance metrics in depth extraction according to the literature reviews, was transformed into a web service and tested on an Android phone. Thus, a deep learning model with a high computational cost will be available on an Android device with average processing power. Test results were examined, and improvements were discussed.

Monocular 3D object detection is an important task for autonomous driving considering its advantage of low cost. It is much more challenging than conventional 2D cases due to its inherent ill-posed property, which is mainly reflected in the lack of depth information. Recent progress on 2D detection offers opportunities to better solving this problem. However, it is non-trivial to make a general adapted 2D detector work in this 3D task. In this paper, we study this problem with a practice built on a fully convolutional single-stage detector and propose a general framework FCOS3D. Specifically, we first transform the commonly defined 7-DoF 3D targets to the image domain and decouple them as 2D and 3D attributes. Then the objects are distributed to different feature levels with consideration of their 2D scales and assigned only according to the projected 3D-center for the training procedure. Furthermore, the center-ness is redefined with a 2D Gaussian distribution based on the 3D-center to fit the 3D target formulation. All of these make this framework simple yet effective, getting rid of any 2D detection or 2D-3D correspondence priors. Our solution achieves 1st place out of all the vision-only methods in the nuScenes 3D detection challenge of NeurIPS 2020. Code and models are released at https://github.com/open-mmlab/mmdetection3d., Comment: Camera-ready version of 3DODI workshop at ICCV 2021; Technical report for the best vision-only method (1st place of the camera track) in the nuScenes 3D detection challenge of NeurIPS 2020

There is a growing interest in using augmented reality for simulation and training across fields such as medicine, military, and others. An important question to address is whether distances to virtual objects displayed in augmented reality are perceived accurately and similarly to the real world. Current augmented reality (AR) devices, such as the HoloLens, provide an advantage over virtual reality with regard to depth perception because they display virtual objects in a real world context that preserves depth cues of the real world. However, these AR devices limit real world experience because they restrict the field of view (FOV), add weight, and may be cumbersome to wear over time. In the current study, we investigated whether distance judgments were similar when objects were presented in the real world or in augmented reality (via the HoloLens version 1). All participants wore the device to control for the effect of weight on distance judgments. We also tested whether reducing cues to depth by asking participants to view the targets both monocularly and binocularly affected judgments. Participants blind walked to a cube placed at varying distances in both a condition where the cube was viewed monocularly and binocularly. There was no main effect for viewing in augmented reality vs. the real world. But, viewing condition (either monocular or binocular) did interact with group such that viewing monocularly resulted in greater underestimation of farther distances for AR targets compared to real world targets. These findings provide support for the notion that distance estimations to real and augmented reality objects can be similar, given binocular viewing conditions. Our study suggests that AR may be well suited for training and simulation applications that utilize the range of distances tested herein.

Visual consistency between virtual objects and the real environment is essential to improve user experience in Augmented Reality (AR). Occlusion handling is one of the key factors for maintaining visual consistency. In an application scenario for small areas such as indoors, various methods are applicable to acquire a depth information required for occlusion handling. However, in an application scenario in wide environment such as outdoor especially a scene including many buildings, occlusion handling is a challenging task because acquiring an accurate depth map is challenging. Several studies that have tackled this problem utilized 3D models of real buildings, but they have suffered from the accuracy of 3D models and camera localization. In this study, we propose a novel occlusion handling method using a monocular RGB camera and map data. Our method detects the regions of buildings in a camera image using an instance segmentation method and then obtains accurate occlusion handling in the image from each building instance and corresponding building map. The qualitative evaluation shows the improvement in the occlusion handling with buildings. The user study also shows the better performance of the perception of depth and distance than a model-based method.

Autonomous navigation in agriculture is very challenging as it usually takes place outdoors where there is rough terrain, uncontrolled natural lighting, constantly changing organic scenarios and sometimes the absence of Global Navigation Satellite System (GNSS) signal. In this work, a monocular visual system is proposed to estimate angular orientation and navigate between woody crops, more specifically a vineyard, using a Proportional Integrative Derivative (PID)-based controller. The guidance is provided by combining two ways to find the center of the vineyard: First, by estimating the vanishing point and second, by averaging the position of the two closest base trunk detections. Then, by the monocular angle perception, the angular error is determined. For obtaining the trunk position in the image, object detection using Deep Learning (DL) based Neural Networks (NN) is used. To evaluate the proposed controller, a visual vineyard simulation is created using Gazebo. The proposed joint controller is able to travel along a simulated straight vineyard with an RMS error of 1.17 cm. Moreover, a simulated curved vineyard modeled after the Douro region is tested in this work, where the robot was able to steer with an RMS error of 7.28 cm.

Relying on monocular image data for precise 3D object detection remains an open problem, whose solution has broad implications for cost-sensitive applications such as traffic monitoring. We present UrbanNet, a modular architecture for long range monocular 3D object detection with static cameras. Our proposed system combines commonly available urban maps along with a mature 2D object detector and an efficient 3D object descriptor to accomplish accurate detection at long range even when objects are rotated along any of their three axes. We evaluate UrbanNet on a novel challenging synthetic dataset and highlight the advantages of its design for traffic detection in roads with changing slope, where the flat ground approximation does not hold. Data and code are available at https://github.com/TRAILab/UrbanNet

Monocular 3D object detection is an ill-posed problem due to the loss of geometry information in 2D images. To improve the reliability of monocular vision system, we propose to replace the deterministic estimation with a probabilistic prediction by computing both aleatoric and epistemic uncertainty in 3D detection task. We further design a continuous focal loss that can improve location accuracy and when combining with aleatoric uncertainty estimation, also reduce the effect of noise data during training. Our method is capable of estimating uncertainty in a single forward pass while achieving nearly real-time performance with 0.1s per image on a modern platform. Additionally, to make fair comparison between point and probabilistic prediction, we present a novel probabilistic localization evaluation metric which computes the average error over the predicted location distribution. Our experiments indicate that we achieve competitive result on the official KITTI 3D detection leaderboard, among other deterministic methods. Furthermore, our method gains more accurate localization accuracy under different localization evaluation metrics.

Mapping and 3D detection are two major issues in vision-based robotics, and self-driving. While previous works only focus on each task separately, we present an innovative and efficient multi-task deep learning framework (SM3D) for Simultaneous Mapping and 3D Detection by bridging the gap with robust depth estimation and "Pseudo-LiDAR" point cloud for the first time. The Mapping module takes consecutive monocular frames to generate depth and pose estimation. In 3D Detection module, the depth estimation is projected into 3D space to generate "Pseudo-LiDAR" point cloud, where LiDAR-based 3D detector can be leveraged on point cloud for vehicular 3D detection and localization. By end-to-end training of both modules, the proposed mapping and 3D detection method outperforms the state-of-the-art baseline by 10.0% and 13.2% in accuracy, respectively. While achieving better accuracy, our monocular multi-task SM3D is more than 2 times faster than pure stereo 3D detector, and 18.3% faster than using two modules separately.

This paper proposes a new framework called ATFVO which can be deployed on the edge device to resolve monocular visual odometry problem. The vast majority of visual odometry algorithms using deep learning are equivalent to or beyond the traditional visual odometry algorithms in performance, however they do not consider the computing capability of edge equipment. In this paper, convolution neural network (CNN) and attentive tensor-compressed compression LSTM (A-T-LSTM) are used, with optical flow feature as input and a 6-DoF absolute-scale pose as output. The framework is fused with the spatio-temporal feature and deal with the overfitting problem of over-parameterized LSTM with high-dimensional inputs, and utilizes attention mechanism to get poses from the sequence output of T-LSTM. The poses are estimated from the original RGB images sequence without depending on any prior knowledge. The experimental outcomes at the KITTI dataset display that, in compared with the performance of the most advanced methods, the single T-LSTM model is 141× smaller than the original LSTM model, and the entire model is nearly one-seventh of DeepVO with a speed 23× faster than Flowdometry. The proposed VO is deployed to the robot based on raspberry pi, which can achieve real-time inference and navigate a cruise.

SLAM is a key technology in the field of mobile robot. However, it is difficult to meet the requirements of positioning accuracy by using single sensor to locate and navigate for a mobile robot. Multi-sensor information fusion technology has become an important method to solve the problem of mobile robot positioning and navigation. By establishing information fusion model, building sensor experimental platform such as laser radar, ultrasonic ranging module and monocular camera, based on the laser RBPF-SLAM algorithm, a SLAM algorithm based on multi-sensor information fusion is proposed. The mobile robot can locate and construct road sign map at the same time. The laser RBPF-SLAM positioning and monocular visual positioning are advanced by using the information fusion model Information fusion under the maximum posterior probability criterion. Through experiments, the accuracy, effectiveness and practicability of the proposed method are verified.

Using heatmaps to predict body joint locations has become one of the best performing pose estimation methods, however, these methods often have the high demands for memory and computation, which make them difficult to apply into practice. This paper proposes an effective compression method to reduce the size of heatmaps, namely lies Respective Volumetric Heatmap Autoencoder(RVHA) to represent the ground truth heatmaps with smaller data size, then a RVHA-based pose estimation framework is built to achieve the human joint locations from monocular RGB images. Thanks to our compression strategy which takes each human joint volumetric heatmap as an input frame individually, our method performs favorably when compared to state of the art on the JTA datasets.

The task of depth recovery is to calculate the depth map using only the rgb images. The existing deep network greatly compresses the resolution in the process of feature extraction, resulting in serious loss of depth information in the edge area of the object with large depth changes and the area far away from the sensor. This paper proposes a convolutional neural network based on self-attention mechanism, which can use a single rgb image to calculate its corresponding high-resolution depth map. Specifically, we use the Fully Convolutional Network with jump connections and bilinear upsampling, and add a self-attention mechanism module inside it. The self-attention mechanism module aggregates the features at each location through the weighting and selective aggregation of the features at all locations. Compared with the baseline, our method has more obvious object edge information, and also has better results on some indicators.

Monocular depth estimation is an essential technique for tasks like 3D reconstruction. Although many works have emerged in recent years, they can be improved by better utilizing the multi-scale information of the input images, which is proved to be one of the keys in generating high-quality depth estimations. In this paper, we propose a new monocular depth estimation method named HMA-Depth, in which we follow the encoder-decoder scheme and combine several techniques such as skip connections and the atrous spatial pyramid pooling. To obtain more precise local information from the image while keeping a good understanding of the global context, a hierarchical multi-scale attention module is adopted and its outputs are combined to generate the final output that is with both good details and good overall accuracy. Experimental results on two commonly-used datasets prove that HMA-Depth can outperform the existing approaches. Code is available 1 1 https://github.com/saranew/HMADepth., MVA2021,17th International Conference on Machine Vision Applications,July 25–27, 2021, Fully Online

Monocular 3D object detection is a prevalent research topic with several issues yet to be explored. In this paper, we present a novel framework for monocular 3D object detection with autonomous driving as the target application. As the first contribution, wed esign a novel Convolution-Involution Hybrid module for the 3D regression of geometric properties. The peculiar characteristics of involutions are delicately combined with the mature design of convolutions to yield enhanced network performance. The second contribution of us is the Dynamic Error- Tolerant loss. A tolerant threshold is integrated into the disentangled L1 loss to allow for minor errors from the network output. The threshold gradually changes in accordance with both the training iterations and the depth of objects to diminish the impact of outlying objects. To corroborate the effectiveness of the proposed method, extensive experiments and ablation study have been conducted on the prestigious KITTI dataset, and the experimental results demonstrate that our network outperforms the baseline by a large margin while retaining a real-time inference speed.

Unsupervised deep learning methods have demonstrated an impressive performance for understanding the structure of 3D scene from videos. These data-based learning methods are able to learn the tasks, such as depth, ego-motion, and optical flow estimation. In this paper, we propose a novel unsupervised deep learning method to jointly estimate scene depth, camera ego-motion, and optical flow from stereo images. Consecutive stereo images are used to train the system. After training stage, the system is able to estimate dense depth map, camera 6D pose, and optical flow by using a sequence of monocular images. No labelled data set is required for training. The supervision signals for training three deep neural networks of the system come from various forms of image warping. Due to the use of optical flow, the impact caused by occlusions and moving objects on the estimation results is alleviated. Experiments on the KITTI and Cityscapes datasets show that the proposed system demonstrates a better performance in terms of accuracy in depth, ego-motion, and optical flow estimation.

International audience; Many algorithms for visible light positioning (VLP) localization do not consider the shapes of the transmitters, which leads to the impracticality of the algorithm and the low localization accuracy. Therefore, this paper proposes a novel VLP algorithm and addresses the problems in terms of practicality and complexity by using one non-point transmitter based on a monocular. Because the shape of the transmitter is considered, the proposed algorithm is easy for practice and has wide applicability. Besides, it decreases the computation for simple geometric model and expands the coverage for there is a greater chance of receiving signals from one light than that of receiving signals from multiple lights.

Localization in six degrees of freedom is becoming increasingly relevant, especially in indoor environments where GPS is not available. To localize autonomous vehicles like UAVs in such areas, reliable methods for self-localization with low-weight sensors are required. In this paper, we present an approach to precisely localize systems with monocular cameras in polygonal 3D maps annotated with keypoints and feature descriptors computed from LiDAR data and associated reference images. Our contribution consists of offline map computation from high resolution 3D point clouds with corresponding reference images as well as online localization within these maps using low cost sensors. During localization, features extracted from the vehicle’s camera image stream are matched against the reference map. The proposed method is capable of real-time localization and suitable for precise global localization. The evaluation shows comparable results to state of the art with high re-localization accuracy.

The rapid development of UAV has brought great convenience to various application fields. In the meanwhile, its extensive utilization has also resulted in many problems such as public safety hazards, personal security threats and personal privacy violations. UAV is difficult to capture in real-time because of its small scale and complex flight environment. In order to solve the above problems from the perspective of security protection, a low-cost UAV detection, distance measure and protection scheme are proposed based on deep learning in this paper. The influences of different loss functions and thresholds are studied on the detection accuracy of YOLOv4 to improve the detection performance of YOLOv4 on UAV. At the same time, in order to achieve more effective prevention and control of UAV, the monocular ranging method based on PnP is introduced to get the distance between camera and UAV. Finally, the study is applied in the real-world scene, and a good target detection and ranging effect have been achieved so that the proposed model is verified in the feasibility and effectiveness.