Search

Visual-inertial odometry (VIO) algorithms, fusing various features such as points and lines, are able to improve their performance in challenging scenes while the running time severely increases. In this paper, we propose a novel lightweight point–line visual–inertial odometry algorithm to solve this problem, called LRPL-VIO. Firstly, a fast line matching method is proposed based on the assumption that the photometric values of endpoints and midpoints are invariant between consecutive frames, which greatly reduces the time consumption of the front end. Then, an efficient filter-based state estimation framework is designed to finish information fusion (point, line, and inertial). Fresh measurements of line features with good tracking quality are selected for state estimation using a unique feature selection scheme, which improves the efficiency of the proposed algorithm. Finally, validation experiments are conducted on public datasets and in real-world tests to evaluate the performance of LRPL-VIO and the results show that we outperform other state-of-the-art algorithms especially in terms of speed and robustness. [ABSTRACT FROM AUTHOR]

Structural geometry constraints, such as perpendicularity, parallelism and coplanarity, are widely existing in man-made scene, especially in Manhattan scene. By fully exploiting these structural properties, we propose a monocular visual-inertial odometry (VIO) using point and line features with structural constraints. First, a coarse-to-fine vanishing points estimation method with line segment consistency verification is presented to classify lines into structural and non-structural lines accurately with less computation cost. Then, to get precise estimation of camera pose and the position of 3D landmarks, a cost function which combines structural line constraints with feature reprojection residual and inertial measurement unit residual is minimized under a sliding window framework. For geometric representation of lines, Plücker coordinates and orthonormal representation are utilized for 3D line transformation and non-linear optimization respectively. Sufficient evaluations are conducted using two public datasets to verify that the proposed system can effectively enhance the localization accuracy and robustness than other existing state-of-the-art VIO systems with acceptable time consumption. [ABSTRACT FROM AUTHOR]

Commercial visual–inertial odometry (VIO) systems have been gaining attention as cost-effective, off-the-shelf, six-degree-of-freedom (6-DoF) ego-motion-tracking sensors for estimating accurate and consistent camera pose data, in addition to their ability to operate without external localization from motion capture or global positioning systems. It is unclear from existing results, however, which commercial VIO platforms are the most stable, consistent, and accurate in terms of state estimation for indoor and outdoor robotic applications. We assessed four popular proprietary VIO systems (Apple ARKit, Google ARCore, Intel RealSense T265, and Stereolabs ZED 2) through a series of both indoor and outdoor experiments in which we showed their positioning stability, consistency, and accuracy. After evaluating four popular VIO sensors in challenging real-world indoor and outdoor scenarios, Apple ARKit showed the most stable and high accuracy/consistency, and the relative pose error was a drift error of about 0.02 m per second. We present our complete results as a benchmark comparison for the research community. [ABSTRACT FROM AUTHOR]

Efficient multi‐sensor fusion is crucial to provide accurate pose estimates for navigating various next‐generation autonomous vehicles such as self‐driving cars, personal air vehicles, urban air mobilities, and electronic vertical take‐off and landing aircraft with respect to a unified global reference frame. The integration of the Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) provides globally uniform coordinate information, and the Visual‐Inertial Odometry (VIO) gives stable and accurate local pose estimates. However, the error of the integrated INS/GNSS increases rapidly in GNSS‐challenged environments, and the VIO system suffers from the long‐term error accumulation. To cope with these difficulties, this paper proposes a new tightly coupled Visual‐Inertial‐GNSS (VIG) system to achieve robust and drift‐free pose estimation utilising GNSS raw measurements, image features, and inertial measurements. A novel compressed state constraint Kalman Filter (CSCKF) is formulated to combine time‐propagated feature measurements and differential GNSS observations to aid INS. Compared to the conventional integrated navigation systems, the proposed CSCKF‐based VIG system is advantageous in maintaining the minimum number of states without unnecessary state augmentation and no loss of performance. The proposed system is evaluated by field experiments in different GNSS availability situations. The results show that accuracy is improved significantly in GNSS‐degraded environments compared to that of the conventional systems. [ABSTRACT FROM AUTHOR]

Underwater positioning presents a challenging issue, because of the rapid attenuation of electronic magnetic waves, the disturbances and uncertainties in the environment. Conventional methods usually employed acoustic devices to localize Unmanned Underwater Vehicles (UUVs), which suffer from a slow refresh rate, low resolution, and are susceptible to the environmental noise. In addition, the complex terrain can also degrade the accuracy of the acoustic navigation systems. The applications of underwater positioning methods based on visual sensors are prevented by difficulties of acquiring the depth maps due to the sparse features, the changing illumination condition, and the scattering phenomenon. In the paper, a novel visual-based underwater positioning system is proposed based on a Light Detection and Ranging (LiDAR) camera and an inertial measurement unit. The LiDAR camera, benefiting from the laser scanning techniques, could simultaneously generate the associated depth maps. The inertial sensor would offer information about its altitudes. Through the fusion of the data from multiple sensors, the positions of the UUVs can be predicted. After that, the Bundle Adjustment (BA) method is used to recalculate the rotation matrix and the translation vector to improve the accuracy. The experiments are carried out in a tank to illustrate the effects and accuracy of the investigated method, in which the ultra-wideband (UWB) positioning system is used to provide reference trajectories. It is concluded that the developed positioning system is able to estimate the trajectory of UUVs accurately, whilst being stable and robust. [ABSTRACT FROM AUTHOR]

Nowadays, visual SLAM (Simultaneous Localization And Mapping) has become a hot research topic due to its low costs and wide application scopes. Traditional visual SLAM frameworks are usually designed for single-agent systems, completing both the localization and the mapping with sensors equipped on a single robot or a mobile device. However, the mobility and work capacity of the single agent are usually limited. In reality, robots or mobile devices sometimes may be deployed in the form of clusters, such as drone formations, wearable motion capture systems, and so on. As far as we know, existing SLAM systems designed for multi-agents are still sporadic, and most of them have non-negligible limitations in functions. Specifically, on one hand, most of the existing multi-agent SLAM systems can only extract some key features and build sparse maps. On the other hand, schemes that can reconstruct the environment densely cannot get rid of the dependence on depth sensors, such as RGBD cameras or LiDARs. Systems that can yield high-density maps just with monocular camera suites are temporarily lacking. As an attempt to fill in the research gap to some extent, we design a novel collaborative SLAM system, namely CVIDS (Collaborative Visual-Inertial Dense SLAM), which follows a centralized and loosely coupled framework and can be integrated with any existing Visual-Inertial Odometry (VIO) to accomplish the co-localization and the dense reconstruction. Integrating our proposed robust loop closure detection module and two-stage pose-graph optimization pipeline, the co-localization module of CVIDS can estimate the poses of different agents in a unified coordinate system efficiently from the packed images and local poses sent by the client-ends of different agents. Besides, our motion-based dense mapping module can effectively recover the 3D structures of selected keyframes and then fuse their depth information to the global map for reconstruction. The superior performance of CVIDS is corroborated by both quantitative and qualitative experimental results. To make our results reproducible, the source code has been released at https://cslinzhang.github.io/CVIDS. [ABSTRACT FROM AUTHOR]

Combining camera, IMU and wheel encoder is a wise choice for car positioning because of the low cost and complementarity of the sensors. We propose a novel extended visual-inertial odometry algorithm based on sliding window tightly fusing data from the above three sensors. Firstly we propose an IMU-odometer pre-integration approach utilizing complete IMU measurements and wheel encoder readings, to make scale estimation more accurate in subsequent 4-degrees of freedom (DoF) optimization. Secondly we develop an original initialization module where encoder readings are fully utilized to refine gravity direction and provide an initial value for camera pose in real scale. Thirdly, we design a computationally efficient online extrinsic calibration method by fixing the linearization point for the rotational component of IMU-odometer extrinsic parameters, which is deployed depending on the convergence of accelerometer bias. Fourthly, we give an observability analysis of our optimization based approach under more general assumption. Extensive experiments are performed on two sets of data in various scenes, bringing the state-of-the-art visual odometry and visual-inertial odometry algorithms into comparison. Experimental results prove the overwhelmingly better performance of our proposed approach on the above two sets of data, as well as the robustness of our initialization module and the improvement resulted from online extrinsic calibration. [Display omitted] • IMU-odometer pre-integration fusing IMU measurements and wheel encoder readings at pre-integration stage. • Robust initialization exploiting camera, IMU and wheel encoder data. • Robust and computationally efficient online extrinsic calibration that improves the resulting accuracy observably. • Observability analysis under more general assumptions revealing the benefit of utilizing wheel encoder readings. • Better performance than the state-of-the-art monocular and stereo visual-inertial odometry algorithms. [ABSTRACT FROM AUTHOR]

In the last decade, numerous supervised deep learning approaches have been proposed for visual–inertial odometry (VIO) and depth map estimation, which require large amounts of labelled data. To overcome the data limitation, self-supervised learning has emerged as a promising alternative that exploits constraints such as geometric and photometric consistency in the scene. In this study, we present a novel self-supervised deep learning-based VIO and depth map recovery approach (SelfVIO) using adversarial training and self-adaptive visual–inertial sensor fusion. SelfVIO learns the joint estimation of 6 degrees-of-freedom (6-DoF) ego-motion and a depth map of the scene from unlabelled monocular RGB image sequences and inertial measurement unit (IMU) readings. The proposed approach is able to perform VIO without requiring IMU intrinsic parameters and/or extrinsic calibration between IMU and the camera. We provide comprehensive quantitative and qualitative evaluations of the proposed framework and compare its performance with state-of-the-art VIO, VO, and visual simultaneous localization and mapping (VSLAM) approaches on the KITTI, EuRoC and Cityscapes datasets. Detailed comparisons prove that SelfVIO outperforms state-of-the-art VIO approaches in terms of pose estimation and depth recovery, making it a promising approach among existing methods in the literature. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel robocentric formulation of the visual–inertial navigation system (VINS) within a sliding-window filtering framework and design an efficient, lightweight, robocentric visual–inertial odometry (R-VIO) algorithm for consistent motion tracking even in challenging environments using only a monocular camera and a six-axis inertial measurement unit (IMU). The key idea is to deliberately reformulate the VINS with respect to a moving local frame, rather than a fixed global frame of reference as in the standard world-centric VINS, in order to obtain relative motion estimates of higher accuracy for updating global pose. As an immediate advantage of this robocentric formulation, the proposed R-VIO can start from an arbitrary pose, without the need to align the initial orientation with the global gravitational direction. More importantly, we analytically show that the linearized robocentric VINS does not undergo the observability mismatch issue as in the standard world-centric counterparts that has been identified in the literature as the main cause of estimation inconsistency. Furthermore, we investigate in depth the special motions that degrade the performance in the world-centric formulation and show that such degenerate cases can be easily compensated for by the proposed robocentric formulation, without resorting to additional sensors as in the world-centric formulation, thus leading to better robustness. The proposed R-VIO algorithm has been extensively validated through both Monte Carlo simulation and real-world experiments with different sensing platforms navigating in different environments, and shown to achieve better (or competitive at least) performance than the state-of-the-art VINS, in terms of consistency, accuracy, and efficiency. [ABSTRACT FROM AUTHOR]

As mobile robots are being widely used, accurate localization of the robot counts for the system. Compared with position systems with a single sensor, multi-sensor fusion systems provide better performance and increase the accuracy and robustness. At present, camera and IMU (Inertial Measurement Unit) fusion positioning is extensively studied and many representative Visual–Inertial Odometry (VIO) systems have been produced. Multi-State Constraint Kalman Filter (MSCKF), one of the tightly coupled filtering methods, is characterized by high accuracy and low computational load among typical VIO methods. In the general framework, IMU information is not used after predicting the state and covariance propagation. In this article, we proposed a framework which introduce IMU pre-integration result into MSCKF framework as observation information to improve the system positioning accuracy. Additionally, the system uses the Helmert variance component estimation (HVCE) method to adjust the weight between feature points and pre-integration to further improve the positioning accuracy. Similarly, this article uses the wheel odometer information of the mobile robot to perform zero speed detection, zero-speed update, and pre-integration update to enhance the positioning accuracy of the system. Finally, after experiments carried out in Gazebo simulation environment, public dataset and real scenarios, it is proved that the proposed algorithm has better accuracy results while ensuring real-time performance than existing mainstream algorithms. [ABSTRACT FROM AUTHOR]

This paper presents a challenging panoramic vision and LiDAR dataset collected by an autonomous vehicle at Chungbuk National University campus to facilitate robotics research. The vehicle is equipped with a Point Grey Ladybug 3 camera, 3D-LiDAR, global positioning system (GPS) and inertial measurement unit (IMU). The data are collected while driving in an outdoor environment, which includes various scenes such as parking lot, semi-off road path and the campus road scene with traffic. The data from all sensors mounted on the vehicle are timely registered and synchronized. The dataset includes point clouds from 3D LiDAR, images, GPS and IMU measurement. The vision data contain multiple fisheye images covering 360 field-of-view from individual cameras of Ladybug3 at high resolution and accurately stitched spherical panoramic images. The availability of multiple-fisheye and accurate panoramic images may be used for development and validation of novel multi-fisheye, panoramic, and 3D LiDAR based simultaneous localization and mapping (SLAM) systems. The dataset is collected to target various applications such as odometry, SLAM, loop closure detection, deep learning based algorithms with vision, inertial, LiDAR and fusion of visual, inertial and 3D information. To evaluate the algorithm, high accuracy RTK GPS measurements are provided for testing and evaluation. [ABSTRACT FROM AUTHOR]

Most existing visual-inertial systems require control of extrinsic environment and extrinsic motion to obtain calibration results and simplify the initialization model. The former limits the usage scenarios of the method, while the latter reduces the estimation accuracy of the state parameters of the navigation system. Therefore, a visual-inertial calibration and initialization method is necessary for weak environmental and motion control conditions. Firstly, to overcome the reliance on environmental control, we establish a motion constraint equation based on the relative consistency of visual-inertial sensing spatial motion and implement a visual-inertial online calibration method. Secondly, to eliminate the reliance on motion control, a dual graph optimization structure is be constructed. The initial optimization outcome serves as the foundation for achieving improved calibration results in the refined optimization. Thirdly, to improve initialization accuracy, a complete initial state model is considered to decouple and calculate the translation extrinsic parameters. These parameters are compensated into subsequent initialization, enhancing the robustness of the initialization process. The proposed method is validated on the EuRoC dataset, and experimental results demonstrate that it achieves more accurate calibration results at a faster rate than VINS-Mono in scenarios without calibration target and with limited vehicle motion. The navigation results of different scenarios demonstrate that our method can more effectively integrate visual-inertial data due to improved calibration and initialization estimates. [ABSTRACT FROM AUTHOR]

In this article, we propose a novel optimization-based tightly coupled Direct Visual-Inertial Odometry (DVIO), which fuses the visual and inertial measurements to provide real-time full state estimation. Different from existing frameworks, the key novelty of the proposed method is to integrate the data association, state estimation, and outlier detection into a nonlinear optimization framework in a tightly coupled way. Specifically, by jointly minimizing the preintegration error of the inertial measurement unit and the photometric error of the camera, the data association is tightly coupled with the process of state estimation. Then, an iterative selection strategy is design to reject outliers during the data association and establish more visual constraints in the optimization. In addition, a hybrid weighting method is proposed to tightly integrate the iterative selection into the optimization by dynamically weighting residual terms. As a consequence, the proposed method aligns the image patches, estimates the motion and removes the outliers synchronously. Comparative experiments on the public dataset and extensive real-world experiments show that DVIO outperforms state-of-the-art visual-inertial odometries in terms of both the accuracy and the robustness. Thus, DVIO is highly applicable to the navigation or the simultaneous localization and mapping of mobile devices or agile robots like micro air vehicles. [ABSTRACT FROM AUTHOR]

This article presents a new dataset of ultra-wide field of view images with accurate ground truth, called PanoraMIS. The dataset covers a large spectrum of panoramic cameras (catadioptric, twin-fisheye), robotic platforms (wheeled, aerial, and industrial robots), and testing environments (indoors and outdoors), and it is well suited to rigorously validate novel image-based robot-motion estimation algorithms, including visual odometry, visual SLAM, and deep learning-based methods. PanoraMIS and the accompanying documentation is publicly available on the Internet for the entire research community. [ABSTRACT FROM AUTHOR]

In this article, we perform a thorough observability analysis for linearized inertial navigation systems (INS) aided by exteroceptive range and/or bearing sensors (such as cameras, LiDAR, and sonars) with different geometric features (points, lines, planes, or their combinations). In particular, by reviewing common representations of geometric features, we introduce two sets of unified feature representations, i.e., the quaternion and closest point (CP) parameterizations. While the observability of vision-aided INS (VINS) with point features has been extensively studied in the literature, we analytically show that the general aided INS with point features preserves the same observability property, i.e., four unobservable directions, corresponding to the global yaw and the global translation of the sensor platform. We further prove that there are at least five (or seven) unobservable directions for the linearized aided INS with a single line (plane) feature, and, for the first time, analytically derive the unobservable subspace for the case of multiple lines or planes. Building upon this analysis for homogeneous features, we examine the observability of the same system but with combinations of heterogeneous features, and show that, in general, the system preserves at least four unobservable directions, while if global measurements are available, as expected, the unobservable subspace will have lower dimensions. We validate our analysis in Monte–Carlo simulations using both EKF-based visual-inertial SLAM and visual-inertial odometry (VIO) with different geometric features. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new analytical preintegration theory for graph-based sensor fusion with an inertial measurement unit (IMU) and a camera (or other aiding sensors). Rather than using discrete sampling of the measurement dynamics as in current methods, we derive the closed-form solutions to the preintegration equations, yielding improved accuracy in state estimation. We advocate two new different inertial models for preintegration: (i) the model that assumes piecewise constant measurements; and (ii) the model that assumes piecewise constant local true acceleration. Through extensive Monte Carlo simulations, we show the effect that the choice of preintegration model has on estimation performance. To validate the proposed preintegration theory, we develop both direct and indirect visual–inertial navigation systems (VINSs) that leverage our preintegration. In the first, within a tightly coupled, sliding-window optimization framework, we jointly estimate the features in the window and the IMU states while performing marginalization to bound the computational cost. In the second, we loosely couple the IMU preintegration with a direct image alignment that estimates relative camera motion by minimizing the photometric errors (i.e., image intensity difference), allowing for efficient and informative loop closures. Both systems are extensively validated in real-world experiments and are shown to offer competitive performance to state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, a novel approach to ego-motion estimation is proposed based on visual and inertial sensors, named Omnidirectional Visual-Inertial Odometry (OVIO). The proposed approach combines omnidirectional visual features with inertial measurements within the Multi-State Constraint Kalman Filter (MSCKF). In contrast with other visual inertial odometry methods that use visual features captured by perspective cameras, the proposed approach utilizes spherical images obtained by an omnidirectional camera to obtain more accurate estimates of the position and orientation of the camera. Because the standard perspective model is unsuitable for omnidirectional cameras, a measurement model on a plane tangent to the unit sphere rather than on the image plane is defined. The key hypothesis of OVIO is that a wider field of view allows the incorporation of more visual features from the surrounding environment, thereby improving the accuracy and robustness of the ego-motion estimation. Moreover, by using an omnidirectional camera, a situation where there is not enough texture is less likely to arise. Experimental evaluation of OVIO using synthetic and real video sequences captured by a fish-eye camera in both indoor and outdoor environments shows the superior performance of the proposed OVIO as compared to the MSCKF using a perspective camera in both positioning and attitude estimation. [ABSTRACT FROM AUTHOR]

To address the problem of estimating camera trajectory and to build a structural three-dimensional (3D) map based on inertial measurements and visual observations, this paper proposes point–line visual–inertial odometry (PL-VIO), a tightly-coupled monocular visual–inertial odometry system exploiting both point and line features. Compared with point features, lines provide significantly more geometrical structure information on the environment. To obtain both computation simplicity and representational compactness of a 3D spatial line, Plücker coordinates and orthonormal representation for the line are employed. To tightly and efficiently fuse the information from inertial measurement units (IMUs) and visual sensors, we optimize the states by minimizing a cost function which combines the pre-integrated IMU error term together with the point and line re-projection error terms in a sliding window optimization framework. The experiments evaluated on public datasets demonstrate that the PL-VIO method that combines point and line features outperforms several state-of-the-art VIO systems which use point features only. [ABSTRACT FROM AUTHOR]

The paper presents a direct visual-inertial odometry system. In particular, a tightly coupled nonlinear optimization based method is proposed by integrating the recent development in direct dense visual tracking of camera and the inertial measurement unit (IMU) pre-integration. Then a factor graph optimization is adopted to estimate the pose and position of the camera, and a semi-dense map is created simultaneously. Two sliding windows are maintained in the proposed approach. The first one, based on direct sparse odometry (DSO), is to estimate the depths of candidate points for mapping and dense visual tracking. In the second one, measurements of both the IMU pre-integration and direct dense visual tracking are fused probabilistically based on a tightly-coupled, optimization-based sensor fusion framework. As a result, the scale drift of visual odometry is compensated by the constraints from the IMU pre-integration. Evaluations on real-world benchmark datasets show that the proposed method achieves competitive results in indoor scenes. [ABSTRACT FROM AUTHOR]

Simultaneous localization and mapping (SLAM) is emerging as a prominent issue in computer vision and next-generation core technology for robots, autonomous navigation and augmented reality. In augmented reality applications, fast camera pose estimation and true scale are important. In this paper, we present an adaptive monocular visual-inertial SLAM method for real-time augmented reality applications in mobile devices. First, the SLAM system is implemented based on the visual-inertial odometry method that combines data from a mobile device camera and inertial measurement unit sensor. Second, we present an optical-flow-based fast visual odometry method for real-time camera pose estimation. Finally, an adaptive monocular visual-inertial SLAM is implemented by presenting an adaptive execution module that dynamically selects visual-inertial odometry or optical-flow-based fast visual odometry. Experimental results show that the average translation root-mean-square error of keyframe trajectory is approximately 0.0617 m with the EuRoC dataset. The average tracking time is reduced by 7.8%, 12.9%, and 18.8% when different level-set adaptive policies are applied. Moreover, we conducted experiments with real mobile device sensors, and the results demonstrate the effectiveness of performance improvement using the proposed method. [ABSTRACT FROM AUTHOR]

This paper presents a visual-inertial odometry framework that tightly fuses inertial measurements with visual data from one or more cameras, by means of an iterated extended Kalman filter. By employing image patches as landmark descriptors, a photometric error is derived, which is directly integrated as an innovation term in the filter update step. Consequently, the data association is an inherent part of the estimation process and no additional feature extraction or matching processes are required. Furthermore, it enables the tracking of noncorner-shaped features, such as lines, and thereby increases the set of possible landmarks. The filter state is formulated in a fully robocentric fashion, which reduces errors related to nonlinearities. This also includes partitioning of a landmark's location estimate into a bearing vector and distance and thereby allows an undelayed initialization of landmarks. Overall, this results in a compact approach, which exhibits a high level of robustness with respect to low scene texture and motion blur. Furthermore, there is no timeconsuming initialization procedure and pose estimates are available starting at the second image frame. We test the filter on different real datasets and compare it with other state-of-the-art visual-inertial frameworks. Experimental results show that robust localization with high accuracy can be achieved with this filter-based framework. [ABSTRACT FROM AUTHOR]

RGB-D cameras such as RealSense and Structure Sensors have been widely used in most robotics systems. This paper presents a system for estimating the trajectory of an RGB-D camera and IMU in indoor environments. The system uses a novel relative pose estimation method that utilizes depth measurements and epipolar constraints for initialization. An adaptive depth estimation method is also proposed, which fuses a depth uncertainty model and multi-view triangulation. In the backend, a sliding window framework is used to optimize the system state by minimizing the residuals of pre-integrated IMU, 3D features re-projection, and 2D features epipolar constraint. The effectiveness of the system is evaluated using publicly available datasets with ground truth trajectories. • This paper uses the depth uncertainty model of the depth camera to divide feature point pairs into 3D–3D and 2D–2D pairs based on their depth measurement noise. A robust pose estimation method is then introduced to fuse 3D–2D and 2D–2D constraints for initialization. • An adaptive depth estimation method is proposed, which combines the depth uncertainty model and multi-view triangulation to achieve high-precision feature point depth estimation. • The paper presents a nonlinear optimization framework in the backend that combines epipolar and re-projection constraints. [ABSTRACT FROM AUTHOR]

In this paper, we present a stereo visual-inertial odometry algorithm assembled with three separated Kalman filters, i.e., attitude filter, orientation filter, and position filter. Our algorithm carries out the orientation and position estimation with three filters working on different fusion intervals, which can provide more robustness even when the visual odometry estimation fails. In our orientation estimation, we propose an improved indirect Kalman filter, which uses the orientation error space represented by unit quaternion as the state of the filter. The performance of the algorithm is demonstrated through extensive experimental results, including the benchmark KITTI datasets and some challenging datasets captured in a rough terrain campus. [ABSTRACT FROM AUTHOR]

This paper is concerned with real-time monocular visual–inertial simultaneous localization and mapping (SLAM). In particular a tightly coupled nonlinear-optimization-based solution that can match the global optimal result in real time is proposed. The methodology is motivated by the requirement to produce a scale-correct visual map, in an optimization framework that is able to incorporate relocalization and loop closure constraints. Special attention is paid to achieve robustness to many real world difficulties, including degenerate motions and unobservablity. A variety of helpful techniques are used, including: a relative manifold representation, a minimal-state inverse depth parameterization, and robust non-metric initialization and tracking. Importantly, to enable real-time operation and robustness, a novel numerical dog-leg solver is presented that employs multi-threaded, asynchronous, adaptive conditioning. In this approach, the conditioning edges of the SLAM graph are adaptively identified and solved for both synchronously and asynchronously. In this way one thread focuses on a small number of temporally immediate parameters and hence constitute a natural “front-end”; the other thread adaptively focuses on larger portions of the SLAM problem, and hence is able to re-estimate past parameters in the presence of new information: an ability that is useful for self-calibration, during degenerate motions, or when bias and the direction of gravity are poorly observed. Experiments with real and simulated data for both indoor and outdoor scenarios demonstrate that asynchronous adaptive conditioning is accurate, and able to closely track the batch SLAM maximum likelihood solution in real time. [ABSTRACT FROM AUTHOR]

Combining visual and inertial measurements has become popular in mobile robotics, since the two sensing modalities offer complementary characteristics that make them the ideal choice for accurate visual–inertial odometry or simultaneous localization and mapping (SLAM). While historically the problem has been addressed with filtering, advancements in visual estimation suggest that nonlinear optimization offers superior accuracy, while still tractable in complexity thanks to the sparsity of the underlying problem. Taking inspiration from these findings, we formulate a rigorously probabilistic cost function that combines reprojection errors of landmarks and inertial terms. The problem is kept tractable and thus ensuring real-time operation by limiting the optimization to a bounded window of keyframes through marginalization. Keyframes may be spaced in time by arbitrary intervals, while still related by linearized inertial terms. We present evaluation results on complementary datasets recorded with our custom-built stereo visual–inertial hardware that accurately synchronizes accelerometer and gyroscope measurements with imagery. A comparison of both a stereo and monocular version of our algorithm with and without online extrinsics estimation is shown with respect to ground truth. Furthermore, we compare the performance to an implementation of a state-of-the-art stochastic cloning sliding-window filter. This competitive reference implementation performs tightly coupled filtering-based visual–inertial odometry. While our approach declaredly demands more computation, we show its superior performance in terms of accuracy. [ABSTRACT FROM AUTHOR]

This paper presents a new deep visual-inertial odometry and depth estimation framework for improving the accuracy of depth estimation and ego-motion from image sequences and inertial measurement unit (IMU) raw data. The proposed framework predicts ego-motion and depth with absolute scale in a self-supervised manner. We first capture dense features and solve the pose by deep visual odometry (DVO), and then combine the pose estimation pipeline with deep inertial odometry (DIO) by the extended Kalman filter (EKF) method to produce the sparse depth and pose with absolute scale. We then join deep visual-inertial odometry (DeepVIO) with depth estimation by using sparse depth and the pose from DeepVIO pipeline to align the scale of the depth prediction with the triangulated point cloud and reduce image reconstruction error. Specifically, we use the strengths of learning-based visual-inertial odometry (VIO) and depth estimation to build an end-to-end self-supervised learning architecture. We evaluated the new framework on the KITTI datasets and compared it to the previous techniques. We show that our approach improves results for ego-motion estimation and achieves comparable results for depth estimation, especially in the detail area. [ABSTRACT FROM AUTHOR]

This work investigates the relationship between system observability properties and estimator inconsistency for a Vision-aided Inertial Navigation System (VINS). In particular, first we introduce a new methodology for determining the unobservable directions of nonlinear systems by factorizing the observability matrix according to the observable and unobservable modes. Subsequently, we apply this method to the VINS nonlinear model and determine its unobservable directions analytically. We leverage our analysis to improve the accuracy and consistency of linearized estimators applied to VINS. Our key findings are evaluated through extensive simulations and experimental validation on real-world data, demonstrating the superior accuracy and consistency of the proposed VINS framework compared to standard approaches. [ABSTRACT FROM AUTHOR]

In this paper, we focus on the problem of motion tracking in unknown environments using visual and inertial sensors. We term this estimation task visual–inertial odometry (VIO), in analogy to the well-known visual-odometry problem. We present a detailed study of extended Kalman filter (EKF)-based VIO algorithms, by comparing both their theoretical properties and empirical performance. We show that an EKF formulation where the state vector comprises a sliding window of poses (the multi-state-constraint Kalman filter (MSCKF)) attains better accuracy, consistency, and computational efficiency than the simultaneous localization and mapping (SLAM) formulation of the EKF, in which the state vector contains the current pose and the features seen by the camera. Moreover, we prove that both types of EKF approaches are inconsistent, due to the way in which Jacobians are computed. Specifically, we show that the observability properties of the EKF’s linearized system models do not match those of the underlying system, which causes the filters to underestimate the uncertainty in the state estimates. Based on our analysis, we propose a novel, real-time EKF-based VIO algorithm, which achieves consistent estimation by (i) ensuring the correct observability properties of its linearized system model, and (ii) performing online estimation of the camera-to-inertial measurement unit (IMU) calibration parameters. This algorithm, which we term MSCKF 2.0, is shown to achieve accuracy and consistency higher than even an iterative, sliding-window fixed-lag smoother, in both Monte Carlo simulations and real-world testing. [ABSTRACT FROM AUTHOR]

• The visual-inertial odometry with nonlinear optimization is proposed. • The depth estimation method based on quadtree is proposed. • The depth map is fused by the truncated symbol distance function (TSDF). Nowadays, 3D reconstruction mainly uses depth cameras. For the disadvantages that depth cameras cannot get depth value outdoor and requires careful scanning to obtain accurate reconstruction effects, we propose a monocular dense mapping system. Since vision-only methods and IMU sensors are both easy to drift, we optimize the propagation strategies during the IMU preintegration. We marginalize the frames in the sliding window to bound the computational complexity. The back-end minimizes the marginalized IMU measurement residuals and visual residuals together. The four-level quadtree is applied to estimate the depth and reconstruct the map. We compare our method with five advanced odometry systems and compare the developed method with and without the IMU sensor on public EuRoC datasets. The results show that the method which involves IMU performs better than the vision-only case in general. The reconstruction result is accurate and clear. [ABSTRACT FROM AUTHOR]

Based on the hypothesis of the Manhattan world, we propose a tightly-coupled monocular visual-inertial odometry (VIO) system that combines structural features with point features and can run on a mobile phone in real-time. The back-end optimization is based on the sliding window method to improve computing efficiency. As the Manhattan world is abundant in the man-made environment, this regular world can use structural features to encode the orthogonality and parallelism concealed in the building to eliminate the accumulated rotation error. We define a structural feature as an orthogonal basis composed of three orthogonal vanishing points in the Manhattan world. Meanwhile, to extract structural features in real-time on the mobile phone, we propose a fast structural feature extraction method based on the known vertical dominant direction. Our experiments on the public datasets and self-collected dataset show that our system is superior to most existing open-source systems, especially in the situations where the images are texture-less, dark, and blurry. [ABSTRACT FROM AUTHOR]

Mobile platform visual image sequence inevitably has large areas with various types of weak textures, which affect the acquisition of accurate pose in the subsequent platform moving process. The visual–inertial odometry (VIO) with point features and line features as visual information shows a good performance in weak texture environments, which can solve these problems to a certain extent. However, the extraction and matching of line features are time consuming, and reasonable weights between the point and line features are hard to estimate, which makes it difficult to accurately track the pose of the platform in real time. In order to overcome the deficiency, an improved effective point–line visual–inertial odometry system is proposed in this paper, which makes use of geometric information of line features and combines with pixel correlation coefficient to match the line features. Furthermore, this system uses the Helmert variance component estimation method to adjust weights between point features and line features. Comprehensive experimental results on the two datasets of EuRoc MAV and PennCOSYVIO demonstrate that the point–line visual–inertial odometry system developed in this paper achieved significant improvements in both localization accuracy and efficiency compared with several state-of-the-art VIO systems. [ABSTRACT FROM AUTHOR]

Robotic mapping and odometry are the primary competencies of a navigation system for an autonomous mobile robot. However, the state estimation of the robot typically mixes with a drift over time, and its accuracy is degraded critically when using only proprioceptive sensors in indoor environments. Besides, the accuracy of an ego-motion estimated state is severely diminished in dynamic environments because of the influences of both the dynamic objects and light reflection. To this end, the multi-sensor fusion technique is employed to bound the navigation error by adopting the complementary nature of the Inertial Measurement Unit (IMU) and the bearing information of the camera. In this paper, we propose a robust tightly-coupled Visual-Inertial Navigation System (VINS) based on multi-stage outlier removal using the Multi-State Constraint Kalman Filter (MSCKF) framework. First, an efficient and lightweight VINS algorithm is developed for the robust state estimation of a mobile robot by practicing a stereo camera and an IMU towards dynamic indoor environments. Furthermore, we propose strategies to deal with the impacts of dynamic objects by using multi-stage outlier removal based on the feedback information of estimated states. The proposed VINS is implemented and validated through public datasets. In addition, we develop a sensor system and evaluate the VINS algorithm in the dynamic indoor environment with different scenarios. The experimental results show better performance in terms of robustness and accuracy with low computation complexity as compared to state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Visual-Inertial Odometry (VIO) is subjected to additional unobservable directions under the special motions of ground vehicles, resulting in larger pose estimation errors. To address this problem, a tightly-coupled Ackermann visual-inertial odometry (ACK-MSCKF) is proposed to fuse Ackermann error state measurements and the Stereo Multi-State Constraint Kalman Filter (S-MSCKF) with a tightly-coupled filter-based mechanism. In contrast with S-MSCKF, in which the inertial measurement unit (IMU) propagates the vehicle motion and then the propagation is corrected by stereo visual measurements, we successively update the propagation with Ackermann error state measurements and visual measurements after the process model and state augmentation. This way, additional constraints from the Ackermann measurements are exploited to improve the pose estimation accuracy. Both qualitative and quantitative experimental results evaluated under real-world datasets from an Ackermann steering vehicle lead to the following demonstration: ACK-MSCKF can significantly improve the pose estimation accuracy of S-MSCKF under the special motions of autonomous vehicles, and keep accurate and robust pose estimation available under different vehicle driving cycles and environmental conditions. This paper accompanies the source code for the robotics community. [ABSTRACT FROM AUTHOR]

In the last two decades, the Visual–Inertial Odometry (VIO) has recently received much attention for efficient and accurate ego-motion estimation of unmanned vehicle systems. In particular, the VIO includes only an Inertial Measurement Unit (IMU) and a camera. In this paper, we present a novel calibration approach for accurate deployment of monocular VIO. For this purpose, the hybrid optimization algorithm is used for calibrating the camera intrinsic and camera–IMU extrinsic calibration, automatically and without knowing the mechanical configuration. It is a professional work to carefully calibrate the intrinsic and extrinsic parameters, and it is required to repeat this work when the mechanical configuration of the sensor suite changes. Quantitative comparisons our method with the offline conventional calibration method on the KITTI dataset verify the efficacy and accuracy of the proposed method. We also demonstrate the performance of the proposed approach in large scale outdoor experiments. [ABSTRACT FROM AUTHOR]

Owing to the nonlinearity in visual-inertial state estimation, sufficiently accurate initial states, especially the spatial and temporal parameters between IMU (Inertial Measurement Unit) and camera, should be provided to avoid divergence. Moreover, these parameters are required to be calibrated online since they are likely to vary once the mechanical configuration slightly changes. Recently, direct approaches have gained popularity for their better performance than feature-based approaches in little-texture or low-illumination environments, taking advantage of tracking pixels directly. Based on these considerations, we perform a direct version of monocular VIO (Visual-inertial Odometry), and propose a novel approach to initialize the spatial-temporal parameters and estimate them with all other variables of interest (IMU pose, point inverse depth, etc.). We highlight that our approach is able to perform robust and accurate initialization and online calibration for the spatial and temporal parameters without utilizing any prior information, and also achieves high-precision estimates even when large temporal offset occurs. The performance of the proposed approach was verified through the public UAV (Unmanned Aerial Vehicle) dataset. [ABSTRACT FROM AUTHOR]

Visual-inertial odometry (VIO) has recently received much attention for efficient and accurate ego-motion estimation of unmanned aerial vehicle systems (UAVs). Recent studies have shown that optimization-based algorithms achieve typically high accuracy when given enough amount of information, but occasionally suffer from divergence when solving highly non-linear problems. Further, their performance significantly depends on the accuracy of the initialization of inertial measurement unit (IMU) parameters. In this paper, we propose a novel VIO algorithm of estimating the motional state of UAVs with high accuracy. The main technical contributions are the fusion of visual information and pre-integrated inertial measurements in a joint optimization framework and the stable initialization of scale and gravity using relative pose constraints. To account for the ambiguity and uncertainty of VIO initialization, a local scale parameter is adopted in the online optimization. Quantitative comparisons with the state-of-the-art algorithms on the European Robotics Challenge (EuRoC) dataset verify the efficacy and accuracy of the proposed method. [ABSTRACT FROM AUTHOR]

Background: Animal-attached sensors are increasingly used to provide insights on behaviour and physiology. However, such tags usually lack information on the structure of the surrounding environment from the perspective of a study animal and thus may be unable to identify potentially important drivers of behaviour. Recent advances in robotics and computer vision have led to the availability of integrated depth-sensing and motion-tracking mobile devices. These enable the construction of detailed 3D models of an environment within which motion can be tracked without reliance on GPS. The potential of such techniques has yet to be explored in the field of animal biotelemetry. This report trials an animal-attached structured light depth-sensing and visual–inertial odometry motion-tracking device in an outdoor environment (coniferous forest) using the domestic dog (Canis familiaris) as a compliant test species. Results: A 3D model of the forest environment surrounding the subject animal was successfully constructed using point clouds. The forest floor was labelled using a progressive morphological filter. Trees trunks were modelled as cylinders and identified by random sample consensus. The predicted and actual presence of trees matched closely, with an object-level accuracy of 93.3%. Individual points were labelled as belonging to tree trunks with a precision, recall, and F β \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$_{\beta }$$\end{document} score of 1.00, 0.88, and 0.93, respectively. In addition, ground-truth tree trunk radius measurements were not significantly different from random sample consensus model coefficient-derived values. A first-person view of the 3D model was created, illustrating the coupling of both animal movement and environment reconstruction. Conclusions: Using data collected from an animal-borne device, the present study demonstrates how terrain and objects (in this case, tree trunks) surrounding a subject can be identified by model segmentation. The device pose (position and orientation) also enabled recreation of the animal's movement path within the 3D model. Although some challenges such as device form factor, validation in a wider range of environments, and direct sunlight interference remain before routine field deployment can take place, animal-borne depth sensing and visual–inertial odometry have great potential as visual biologging techniques to provide new insights on how terrestrial animals interact with their environments. [ABSTRACT FROM AUTHOR]