Search

Simultaneous Localization and Mapping (SLAM) combining visual and inertial measurements has achieved significant attention in the community of Robotics and Computer Vision. However, it is still a challenge to balance real-time requirements and accuracy. Therefore, this paper proposes a feedback mechanism for stereo Visual-Inertial SLAM (VISLAM) to provide accurate and real-time motion estimation and map reconstruction. The key idea of the feedback mechanism is that the frontend and backend in the VISLAM system can promote each other. The results of the backend optimization are fed back to the Kalman Filter (KF)-based frontend to reduce the motion estimate error caused by the well-known linearization of the KF estimator. Conversely, this more accurate motion estimate of the frontend can accelerate the backend optimization since it provides a more accurate initial state for the backend. In addition, we design a relocalization and continued SLAM framework with the feedback mechanism for the application of autonomous robot navigation or continuing SLAM. We evaluated the performance of the proposed VISLAM system through experiments on public EuRoC dataset and real-world environments. The experimental results demonstrate that our system is a promising VISLAM system compared with other state-of-the-art VISLAM systems in terms of both computing cost and accuracy.

This paper presents an self-supervised deep learning network for monocular visual inertial odometry (named DeepVIO). DeepVIO provides absolute trajectory estimation by directly merging 2D optical flow feature (OFF) and Inertial Measurement Unit (IMU) data. Specifically, it firstly estimates the depth and dense 3D point cloud of each scene by using stereo sequences, and then obtains 3D geometric constraints including 3D optical flow and 6-DoF pose as supervisory signals. Note that such 3D optical flow shows robustness and accuracy to dynamic objects and textureless environments. In DeepVIO training, 2D optical flow network is constrained by the projection of its corresponding 3D optical flow, and LSTM-style IMU preintegration network and the fusion network are learned by minimizing the loss functions from ego-motion constraints. Furthermore, we employ an IMU status update scheme to improve IMU pose estimation through updating the additional gyroscope and accelerometer bias. The experimental results on KITTI and EuRoC datasets show that DeepVIO outperforms state-of-the-art learning based methods in terms of accuracy and data adaptability. Compared to the traditional methods, DeepVIO reduces the impacts of inaccurate Camera-IMU calibrations, unsynchronized and missing data., Comment: Accepted by IROS 2019, demo video: https://www.youtube.com/watch?v=fMeqCcpBCdM&feature=youtu.be

In this paper, we propose a deep learning based assistive system to improve the environment perception experience of visually impaired (VI). The system is composed of a wearable terminal equipped with an RGBD camera and an earphone, a powerful processor mainly for deep learning inferences and a smart phone for touch-based interaction. A data-driven learning approach is proposed to predict safe and reliable walkable instructions using RGBD data and the established semantic map. This map is also used to help VI understand their 3D surrounding objects and layout through well-designed touchscreen interactions. The quantitative and qualitative experimental results show that our learning based obstacle avoidance approach achieves excellent results in both indoor and outdoor datasets with low-lying obstacles. Meanwhile, user studies have also been carried out in various scenarios and showed the improvement of VI's environment perception experience with our system.

From a service provider's perspective, the functionality provided by a single web service is relatively simple. It is necessary to combine the isolated single web services to form a combined web service which contains more complex features. This paper designs a visual drag-and-drop system and introduces an asynchronous mechanism based on message subscription, we can easily combine web services asynchronously or synchronously and publish them quickly.

Although a wide variety of deep neural networks for robust Visual Odometry (VO) can be found in the literature, they are still unable to solve the drift problem in long-term robot navigation. Thus, this paper aims to propose novel deep end-to-end networks for long-term 6-DoF VO task. It mainly fuses relative and global networks based on Recurrent Convolutional Neural Networks (RCNNs) to improve the monocular localization accuracy. Indeed, the relative sub-networks are implemented to smooth the VO trajectory, while global sub-networks are designed to avoid drift problem. All the parameters are jointly optimized using Cross Transformation Constraints (CTC), which represents temporal geometric consistency of the consecutive frames, and Mean Square Error (MSE) between the predicted pose and ground truth. The experimental results on both indoor and outdoor datasets show that our method outperforms other state-of-the-art learning-based VO methods in terms of pose accuracy.

This paper presents a novel framework for simultaneously implementing localization and segmentation, which are two of the most important vision-based tasks for robotics. While the goals and techniques used for them were considered to be different previously, we show that by making use of the intermediate results of the two modules, their performance can be enhanced at the same time. Our framework is able to handle both the instantaneous motion and long-term changes of instances in localization with the help of the segmentation result, which also benefits from the refined 3D pose information. We conduct experiments on various datasets, and prove that our framework works effectively on improving the precision and robustness of the two tasks and outperforms existing localization and segmentation algorithms., 7 pages, 5 figures.This work has been accepted by ICRA 2019. The demo video can be found at https://youtu.be/Bkt53dAehjY

Data registration method using special three dimensional target to track the structured light measurement system is discussed. Optical scanning device, tracking target and stereo vision system are integrated together to fulfill profile inspection of large-scale free-form surface objects without extra mark points. System architecture and processing steps are introduced and layout optimization methods of three dimensional target are illustrated. Experimental results are showed to evaluate the validity of the registration method and suggests are given to improve the accuracy of the system.

This paper introduces an invariant Hough random ferns (IHRF) incorporating rotation and scale invariance into the local feature description, random ferns classifier training, and Hough voting stages. It is especially suited for object detection under changes in object appearance and scale, partial occlusions, and pose variations. The efficacy of this approach is validated through experiments on a large set of challenging benchmark datasets, and the results demonstrate that the proposed method outperforms state-of-the-art conventional methods such as bounding-box-based and part-based methods. Additionally, we also propose an efficient clustering scheme based on the local patches’ appearance and their geometric relations that can provide pixel-accurate, top-down segmentations from IHRF back-projections. This refined segmentation can be used to improve the quality of online object tracking because it avoids the drifting problem. Thus, an online tracking framework based on IHRF, which is trained and updated in each frame to distinguish and segment the object from the background, is established. Finally, the experimental results on both object segmentation and long-term object tracking show that this method yields accurate and robust tracking performance in a variety of complex scenarios, especially in cases of severe occlusions and nonrigid deformations.

The optical triangulation probe (OTP), which consists of a light spot projector and a camera, has found widespread applications for three-dimensional (3D) measurement and quality control of products in the industrial manufacturing. The OTP calibration is an extremely important issue, since the performances such as high accuracy and repeatability are crucially depended on the calibration results. This paper presents a flexible approach for modeling and calibration of the OTP, which only requires planar patterns observed from a few different orientations and light spots projected on the planes as well. For the calibration procedure, the structure parameters of the OTP are calculated, such as the camera extrinsic and intrinsic parameters which include the coefficients of the lens distortion, and the directional equation for the light axis of the projector. For the measuring procedure, the formulations of 3D computation are concisely described using the calibration results. Experimental tests of the real system confirm the suitable accuracy and repeatability. Furthermore, the technique proposed here is easily generalized for the OTP integration in robot arms or Coordinate Measuring Machines (CMMs).

Dense stereo correspondence enabling reconstruction of depth information in a scene is of great importance in the field of computer vision. Recently, some local solutions based on matching cost filtering with an edge-preserving filter have been proved to be capable of achieving more accuracy than global approaches. Unfortunately, the computational complexity of these algorithms is quadratically related to the window size used to aggregate the matching costs. The recent trend has been to pursue higher accuracy with greater efficiency in execution. Therefore, this paper proposes a new cost-aggregation module to compute the matching responses for all the image pixels at a set of sampling points generated by a hierarchical clustering algorithm. The complexity of this implementation is linear both in the number of image pixels and the number of clusters. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art local methods in terms of both accuracy and speed. Moreover, performance tests indicate that parameters such as the height of the hierarchical binary tree and the spatial and range standard deviations have a significant influence on time consumption and the accuracy of disparity maps.

This paper studies the challenging problem of object detection using rich image and depth features. An invariant Hough random ferns framework for RGB-D images is proposed here, which primarily consists of a rotation-invariant RGB-D local binary feature, random ferns classifier training, Hough mapping and voting, searches for the maxima, and back projection. In comparison with traditional three-dimensional local feature extraction techniques, this method is effective in reducing the amount of computation required for feature extraction and matching. Moreover, the detection results showed that the proposed method is robust against rotation and scale variations, changes in illumination, and part-occlusions. The authors believe that this method will facilitate the use of perception in fields such as robotics.

Belonging to the dynamic and real-time metrology, the key on 3D shape measurement of moving object is one-shot technology including one image acquisition, real-time processing and displaying. According to different speeds of motion, a few structured light methods such as spatial encoding, FTP, phase shifting method and spacetime stereo are introduced and summarized in the form of principle, development, performance and application for shape acquisition of moving object in this paper. Although structured light for measuring 3D shape of moving object is significant and valuable in theory and engineering applications, the research still has some technological challenges and problems to be explored and solved.

A point cloud registration method based on stereovision movement tracking is proposed in this paper. The movement tracking and analysis system is composed of a stereovision measurement system and target bars. Target bars are fixed on a structured light profile scanning system. Retro-reflect target points pasted on the bars are captured by the stereovision measurement system to compute location and orientation of the scanner. The scanner is controlled to move in the view field of the movement tracking system to complete a whole scan of the large scale object. Transformation parameters including the rotation matrixes and translation vectors between local scanning coordinate systems and the global movement tracking coordinate system are computed by tracking the retro-reflect target points. Then, local cloud data of each scan is transformed into the global tracking coordinate system which is obviously an easier registration method. Experimental system is built and experiments are carried out. A movement tracking experiment is designed to give a maximum error of movement tracking less than 0.3 millimeters. Registration algorithm is verified useful by another experiment which gives a complete profile scanning of a large scale fan blade work piece. Accuracy experiments are designed to result in an average registration error less than 0.3 millimeters and standard deviation less than 0.2 millimeters.

A binocular structured light measurement system using heterodyne multi-frequency method is researched in this paper. Three steps must be done to recover 3D profile information: cameras calibration; points matching; 3D information reconstruction. The intrinsic and extrinsic parameters of cameras are calibrated by Zhengyou Zhan's method [1] . Stereo vision model is the basic rule of 3D recovering. Techniques of phase shift and heterodyne multi-frequency are used to aid points matching. Some coded digital fringes are projected to the free-form surfaces and corresponding images are captured by two cameras simultaneously. The wrapped phase map is obtained through four-step phase shift. In a single cycle, phase value is calculated using arctangent function and it is between -π and π. The heterodyne multi-frequency method is applied to get the absolute phase map. When the number of frequencies is changed from 2 to 3, or 4, the unwrapping accuracy and matching results are improved. The processing theory and experimental results are illustrated and analyzed. The experimental results show that accurate and reliable phase result can be obtained on phase map boundaries and break points, which proves its feasibility in industrial situations.

In order to improve the precision of camera calibration in the field of computer vision, we have to detect the points of the calibration pattern precisely. A new approach to sub-pixel corner detection of a grid is proposed in this paper, which is based on the combination of Hough Transform and least square fit. The procedure of the approach is as follows: (1) The image is divided into small regions to avoid the influence of camera lens distortion on linear fitting and each region has only one corner definitely. Edges of the grid in each region are detected by the Canny arithmetic operator. (2) Straight lines in each region are detected by Hough Transform. (3) Two straight lines with a certain separation angle are selected arbitrarily in each region as initial location. Then, edge-points are searched and recorded in the neighborhoods of each straight line in view of that the results of Hough Transform may not be sufficient for the edge location in practice. (4) Four straight lines are fitted by least square using the edge-points detected in each region. Center of the quadrilateral formed by the straight lines is calculated as the sub-pixel corner location. Finally, experimental results show that sub-pixel corner location of the grid can be obtained correctly and precisely, and none of them are missed. Consequently, it has been proved that this approach is feasible in the application of sub-pixel corner detection of a grid.