Search

3D multiple-view reconstruction is an important topic with applications in robotics, surveillance, augmented reality, and other fields. The ubiquity of reconstruction makes it a vital component in many systems, which need hardware and algorithms capable of processing the vast data found in reconstruction. In this dissertation, we first propose a method for performing a stage of reconstruction, triangulation, on the GPU that does not require second-order optimization and yields an order-of-magnitude speedup over a multi-core CPU processor. Next, we accelerate triangulation further on the GPU by discussing a method that leverages the path of a moving camera as a constraint and thus avoids doing non-linear optimization altogether. Subsequently, we shift to the problem of dense stereo, where we use GPU processing power to create more complete dense reconstructions while again leveraging scene constraints to keep runtime tractable. Finally, we devote a large portion of this dissertation to studying the use of multiple GPUs to accelerate the most expensive stage in reconstruction, bundle adjustment. The strategy is to partition the scene and optimize the subproblems in parallel. This approach minimizes communication across GPUs and removes the need for multi-GPU synchronization, which can be costly when there are many GPUs. We analyze multiple parallel, partitioned bundle adjustment strategies and their advantages and disadvantages. We develop an alternate method that parallelizes more efficiently and is scalable on large datasets with more GPUs. We compare the performance of the partitioned, parallel implementation against that of the original, full-problem implementation. We confirm our hypothesis that our approach can obtain large speedups with competitive accuracy for certain scenes but is less robust to ill-conditioned problems and the presence of local minima.

Stitching images with parallax for naturalness remains a challenging problem. This paper proposes an image stitching method which preserves the flatness of planes in the scene for a natural look. Our method formulates the alignment of images as the camera parameters and the normal vectors of planes. Given a set of feature point matches, a process of grouping points into different layers and rejecting outliers is introduced. According to the epipolar constraint of corresponding points in two images, the focal length and the pose change of the camera are recovered simultaneously. Then, the normal vectors are estimated from the point pairs. To achieve good alignment and guide the warping of images, the model is combined with the mesh deformation as a global similarity constraint. In addition, bundle adjustment is adopted to maintain the consistency for stitching multiple images. Experiment shows that the proposed approach outperforms some state-of-the-art warps on real-world scenes.

In this article, a novel method for simultaneous localization and mapping (SLAM) named surface elements (surfel) point SLAM (SP-SLAM) is proposed using an RGB-D camera. The key idea of SP-SLAM is to use not only keypoints but also surfels as features to cope with both high texture and low texture environments. By decomposing a surface into a small number of surfels, the method can represent spacious environments using a relatively small amount of memory. To optimize the poses of points, surfels, and cameras altogether, new objective functions are proposed, and a new bundle adjustment using these objective functions is developed. The proposed SP-SLAM runs in real time on a central processing unit as in other feature-based visual SLAM methods but works better than them not only in high texture but also in low texture environments, overcoming well-known drawbacks of feature-based visual SLAM with degradation in low texture environments. The proposed method is applied to benchmark datasets and its effectiveness is demonstrated by comparing against those of previous methods in terms of localization accuracy.

The indirect VSLAM is widely used in robot localization and navigation thanks to its potential to achieve high localization accuracy with the local feature observations. However, the existing local features are subject to drift and mismatches under various visual conditions, which causes a degrading in localization accuracy and tracking loss. This paper proposes a quantized self-supervised local feature for the indirect VSLAM to handle the environmental interference in robot localization tasks. A joint feature detection and description network is built in a lightweight manner to extract local features in real time. The network is iteratively trained by a self-supervised learning strategy, and the extracted local features are quantized by an orthogonal transformation for efficiency. We utilize frame-wise matching in Hamming space and bundle adjustment to establish a parallel indirect VSLAM. The proposed VSLAM demonstrates outstanding localization accuracy and tracking stability in the evaluation on multiple datasets and robustness in real-world experiments with the Realsense D435 RGB-D sensor. The efficiency experiment on Jetson TX2 indicates that the quantized self-supervised local feature is suitable for feature-based tasks on edge computing platforms.

This letter presents a robust, real-time visual odometry that efficiently exploits the available visual and geometry cues from RGB-D frames for both tracking and mapping. Together with a hybrid tracking algorithm based on a joint multi-objective formulation, we additionally incorporate the point-to-plane metrics into the photometric bundle adjustment (PBA) to constrain the iteration direction especially for those weakly-textured points. The relative pose constraints derived from optimized poses via PBA is then leveraged in combination with reprojection constraints retrieved from maintained feature map, to refine keyframe poses and feature locations. Moreover, the slanted support plane commonly used in multi-view stereo matching, is utilized for the adjustment of the semi-dense points to further enhance mapping accuracy that in turn benefits the front-end tracking. We extensively evaluate our algorithm on benchmark datasets, and those experimental results validate the advantage of our method in terms of overall tracking performance over other representative approaches.

Since visible cameras rely on ideal illumination to provide adequate environment information, visual simultaneous localization and mapping (SLAM) under extreme illumination remains a challenge. Therefore, we propose a direct near-infrared-depth visual SLAM method with an active near-infrared (NIR) light source. The NIR light source provides necessary lighting conditions without interference to human activities. The proposed method utilizes a direct bundle adjustment method jointly fusing near-infrared and depth images to optimize camera motion. In the front-end tracking, the albedo transformed images and depth maps are used in a direct image alignment. During the back-end optimization, the direct bundle adjustment method exploits the albedo-consistency clues to minimize the geometry and albedo residuals. The indoor experiments show that the proposed method can provide robust performances without being affected by visible illumination. Compared with leading SLAM methods using active light source, the proposed method still offers competitive and robust results. The results also show that NIR-based direct SLAM outperforms the same method based on visible light, indicating that the NIR-based implementation is a potential solution for robust pose estimation indoors.

Multiview registration is used to estimate Rigid Body Transformations (RBTs) from multiple frames and reconstruct a scene with corresponding scans. Despite the success of pairwise registration and pose synchronization, the concept of Bundle Adjustment (BA) has been proven to better maintain global consistency. So in this work, we make the multiview point-cloud registration more tractable from a different perspective in resolving range-based BA. We first analyse the optimal condition of the objective function of BA that unifies some previous approaches. Based on this analysis, we propose an objective function that takes both measurement noises and computational cost into account. For the feature parameter update, instead of calculating the global distribution parameters from the raw measurements, we aggregate the local distributions in a frame-wise fashion at each iteration. The computational cost of feature update is then only dependent on the number of scans. Finally, we develop a multiview registration system using voxel-based quantization that can be applied in real-world scenarios. The experimental results demonstrate our superiority over the baselines in terms of both accuracy and speed. Moreover, the results also show that our average positioning errors achieve the centimeter level. Related materials are available at our project page https://hyhuang1995.github.io/bareg/ .

Unmanned aerial vehicles (UAVs) equipped with integrated global navigation satellite systems/inertial navigation systems (GNSS/INS) together with cameras and/or LiDAR sensors are being widely used for topographic mapping in a variety of applications such as precision agriculture, coastal monitoring, and archaeological documentation. Integration of image-based and LiDAR point clouds can provide a comprehensive 3D model of the area of interest. For such integration, ensuring a good alignment between data from the different sources is critical. Although many works have been conducted on this topic, there is still a need for a rigorous integration approach that minimizes the discrepancy between camera and LiDAR data caused by inaccurate system calibration parameters and/or trajectory artifacts. This study proposes an automated tightly-coupled camera/LiDAR integration workflow for GNSS/INS-assisted UAV systems. The proposed strategy is conducted in three main steps. First, an image-based point cloud is generated using a LiDAR/GNSS/INS-assisted structure from motion (SfM) strategy. Then, feature correspondences between image-based and LiDAR point clouds are automatically identified. Finally, an integrated-bundle adjustment procedure including image points, LiDAR raw measurements, and GNSS/INS information is conducted to minimize the discrepancy between point clouds from different sensors while estimating system calibration parameters and refining the trajectory information. The proposed SfM strategy and integration framework are evaluated using five datasets. The SfM results show that using LiDAR data can facilitate feature matching and further increase the number of reconstructed 3D points. The experimental results also illustrate that the developed automated camera/LiDAR integration strategy is capable of accurately estimating system calibration parameters to achieve good alignment among camera/LiDAR data from single/multiple systems. An absolute accuracy in the range of 3–5 cm is achieved for the image/LiDAR point clouds after the integration process.

Existing projector-camera calibration methods typically warp keypoints from a camera image to a projector image using estimated homographies and often suffer from errors in camera parameters and noises due to imperfect planarity of the calibration target. This article proposes a practical and robust projector-camera calibration system that explicitly deals with these challenges. First, a graph-theory-based correspondence algorithm is built on top of a color-coded spatial structured light (SL) pattern. Such SL correspondences are then used for a coarse projector-camera calibration. To gain more robustness against noises from an imperfect planar calibration board, we develop a bundle adjustment algorithm to jointly optimize the estimated projector-camera parameters and the correspondences’ coordinates. Moreover, our system requires only one shot of an SL pattern for each calibration board pose, which is much more practical than multishot solutions. Comprehensive experimental validation is conducted on both synthetic and real data sets, and our method clearly outperforms the existing methods in all experiments. For the benefit of the society, a practical open-source software with graphical user interface (GUI) of the developed system is publicly available at https://github.com/bingyaohuang/single-shot-pro-cam-calib . Note to Practitioners —The proposed method is motivated by two challenges in industrial structured light (SL) system calibration: 1) robustness against imperfect planarity of the calibration target and 2) the number of SL projections per pose. In many industrial SL-based 3-D reconstruction systems, the calibration accuracy greatly affects the reconstruction reliability. Our SL calibration system explicitly deals with calibration target’s imperfect planarity and thus outperforms the existing methods in terms of system accuracy. Another advantage of our SL calibration system is single-shot-per-pose, allowing fast recalibration and reducing the decoding error due to slight pattern misalignment in multishot methods [37] . In addition, we release the open-source calibration software with a graphical user interface (GUI), with which calibration and sparse 3-D reconstruction can be easily performed without any further instructions. Moreover, considering the complex calibration environment and setup, we make the camera and projector imaging parameters, such as exposure, brightness, and contrast, adjustable through widgets and preview. Finally, a limitation of our color-coded SL system is its sensitivity to environment lighting and target texture. This problem may be solved by projector photometric compensation [16] , [18] , [19] , [39] .

Modern measurement technologies are commonly applied to monitor and preserve the cultural heritage as it is an integral part of modern societies. The Terrestrial Laser Scanning (TLS) method is one of the common technologies investigated by the researchers for accurate data acquisition and processing required for architectural documentation. In recent years, many methods were developed for TLS data registration to improve the processing time and accuracy of the bundle adjustment. The aim of this research is to compare the existing TLS target-based registration methods and compare them with the proposed novel method based on the reliability assessment- the robustness analysis. The novel feature-based approach also includes 2D detectors, which were applied to the TLS data converted into spherical images. Measurements were carried out at the Royal Castle in Warsaw using TLS Z+F 5006H and total station Leica TCRP1202. The collected data was analysed using existing software Z+F LaserControl, LupoScan and developed the application to perform 2D + 1H / 3D registration. The main results demonstrated that the proposed method for TLS registration removed the outliers that could not be eliminated by the deviation analysis on control and check points. The accuracy of TLS registration increased with a RMSE difference between 0.1 mm and 3.7 mm in comparison to existing methods. Furthermore, the accuracy of the results from 2D detectors was improved with relative orientation RMSE ≤ 2.1 mm and equivalent for control and check points for X, Y, and Z coordinates in comparison to target-based registration.

Multi-view camera systems are used more and more frequently for applications in close-range photogrammetry, engineering geodesy and autonomous navigation, since they can cover a large portion of the environment and are considerably cheaper than alternative sensors such as laser scanners. In many cases, the cameras do not have overlapping fields of view. In this paper, we report on the development of such a system mounted on a rigid aluminium platform, and focus on its geometric system calibration. We present an approach for estimating the exterior orientation of such a multi-camera system based on bundle adjustment. We use a static environment with ground control points, which are related to the platform via a laser tracker. In the experimental part, the precision and partly accuracy that can be achieved in different scenarios is investigated. While we show that the accuracy potential of the platform is very high, the mounting calibration parameters are not necessarily precise enough to be used as constant values after calibration. However, this disadvantage can be mitigated by using those parameters as observations and refining them on-the-job.

Uncontrolled refraction of optical rays in underwater photogrammetry is known to reduce its accuracy potential. Several strategies have been proposed aiming at restoring the accuracy to levels comparable with photogrammetry applied in air. These methods are mainly based on rigours modelling of the refraction phenomenon or empirical iterative refraction corrections. The authors of this contribution have proposed two mitigation strategies of image residuals systematic patterns in the image plane: (i) empirical weighting of image observations as function of their radial position; (ii) iterative look-up table corrections computed in a squared grid. Here, a novel approach is developed. It explicitly takes into account the object point-to-camera distance dependent error introduced by refraction in multimedia photogrammetry. A polynomial correction function is iteratively computed to correct the image residuals clustered in radial slices in the image plane as function of the point-to-camera distance. The effectiveness of the proposed method is demonstrated by simulations that allow to: (i) separate the geometric error under investigation from other effects not easily modellable and (ii) have reliable reference data against which to assess the accuracy of the result., International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2-2021, ISSN:1682-1750, ISSN:2194-9034, ISSN:1682-1777

For optical 3D measurements in close-range and UAV applications, the modelling of interior orientation is of superior importance in order to subsequently allow for high precision and accuracy in geometric 3D reconstruction. Nowadays, modern camera systems are often used for optical 3D measurements due to UAV payloads and economic purposes. They are constructed of aspheric and spherical lens combinations and include image pre-processing like low-pass filtering or internal distortion corrections that may lead to effects in image space not being considered with the standard interior orientation models. With a variety of structure-from-motion (SfM) data sets, four typical systematic patterns of residuals could be observed. These investigations focus on the evaluation of interior orientation modelling with respect to minimising systematics given in image space after bundle adjustment. The influences are evaluated with respect to interior and exterior orientation parameter changes and their correlations as well as the impact in object space. With the variety of data sets, camera/lens/platform configurations and pre-processing influences, these investigations indicate a number of different behaviours. Some specific advices in the usage of extended interior orientation models, like Fourier series, could be derived for a selection of the data sets. Significant reductions of image space systematics are achieved. Even though increasing standard deviations and correlations for the interior orientation parameters are a consequence, improvements in object space precision and image space reliability could be reached.

Very high-resolution satellite images (VHRSIs) with improved spatial resolution provide unprecedented opportunities to explore the geometric and semantic information of the world. Accordingly, to compensate for the bias of exterior orientation parameters, bundle adjustment of VHRSIs is required. Owing to the sparse points, the bundle adjustment of a few VHRSIs omits high-frequency errors and attitude jitters; therefore, the rational function model (RFM) can achieve an accuracy comparable to that of a rigorous sensor model (RSM), despite the significant difference between the RSM and RFM. In this study, we provide insight into the role of RSM in modeling attitude jitters for the Worldview-3 basic imagery product. Penalized splines were proposed to model the attitude jitters. After correcting the photogrammetric refraction using Saastamoinen’s model and light aberration, the RSM was built for agile satellites. The difference between RSM and RFM is consistent with attitude jitters, which is calculated using the penalized splines model and third-degree polyniomials. To fully explore the attitude jitters, a dense bundle adjustment was proposed to process 47 scenes of the Worldview-3 basic product imagery, which was provided by Johns Hopkins University Applied Physics Laboratory for the “Multi-View Stereo 3D Challenge.” Pairwise feature matching and feature tracking were adopted to generate over 16 000 tie-points (TPs), which were detected in an average of 9.38 images. The bundle adjustment residuals with RFM exhibited distortions similar to those of the attitude jitter. The experiments verified that the bundle adjustment with RFM introduced significant errors triggered by attitude jitters, with a maximum of over 5.0 pixels. The bundle adjustment with RSM could eliminate significant errors caused by attitude jitters and reduce the root mean square errors (RMSEs) from 1.12 pixels to 0.61 pixels. However, the basic product imagery of Worldview-3 exhibited errors in the interior orientation parameters. After analyzing the physical meanings of bias compensation, a self-calibration model was proposed. After comparing the shift compensation, affine compensation, self-calibration, temporal self-calibration, second-degree self-calibration, and second-degree polynomial models, the dense bundle adjustment with self-calibration was suggested because it could compensate for errors in the interior orientation parameters (IOPs) and obtain an accuracy similar to that of high-degree models. Thus, a self-calibration dense bundle adjustment with RSM compensates for the attitude jitters and errors in the IOPs, and achieves a remarkable accuracy of 0.49 pixels in the image coordinates.

Localisation is one of the key elements in navigation. Especially due to the development in automated driving, precise and reliable localisation becomes essential. In this paper, we report on different cooperation approaches in visual localisation with two vehicles driving in a convoy formation. Each vehicle is equipped with a multi-sensor platform consisting of front-facing stereo cameras and a global navigation satellite system (GNSS) receiver. In the first approach, the GNSS signals are used as excentric observations for the projection centres of the cameras in a bundle adjustment, whereas the second approach uses markers on the front vehicle as dynamic ground control points (GCPs). As the platforms are moving and data acquisition is not synchronised, we use time dependent platform poses. These time dependent poses are represented by trajectories consisting of multiple 6 Degree of Freedom (DoF) anchor points between which linear interpolation takes place. In order to investigate the developed approach experimentally, in particular the potential of dynamic GCPs, we captured data using two platforms driving on a public road at normal speed. As a baseline, we determine the localisation parameters of one platform using only data of that platform. We then compute a solution based on image and GNSS data from both platforms. In a third scenario, the front platform is used as a dynamic GCP which can be related to the trailing platform by markers observed in the images acquired by the latter. We show that both cooperative approaches lead to significant improvements in the precision of the poses of the anchor points after bundle adjustment compared to the baseline. The improvement achieved due to the inclusion of dynamic GCPs is somewhat smaller than the one due to relating the platforms by tie points. Finally, we show that for an individual vehicle, the use of dynamic GCPs can compensate for the lack of GNSS data.

Image orientation is a fundamental task in photogrammetric applications and it is performed by extracting keypoints with hand-crafted or learning-based methods, generating tie points among the images and running a bundle adjustment procedure. Nowadays, due to large number of extracted keypoints, tie point filtering approaches attempt to eliminate redundant tie points in order to increase accuracy and reduce processing time. This paper presents the results of an investigation concerning tie points impact on bundle adjustment results. Simulations and real data are processed in Australis and DBAT to evaluate different affecting factors, including tie point numbers, location accuracy, distribution and multiplicity. Achieved results show that increasing the amount of tie points improve the quality of bundle adjustment results, provided that the tie points are well-distributed on the image. Furthermore, bundle adjustment quality is improved as the multiplicity of tie points increases and their location uncertainty decrease. Based on simulation results, some suggestions for accurate tie points filtering in typical UAV photogrammetry blocks cases are derived.

Three-dimensional (3D) reconstruction, a popular topic in computer vision, has been researched extensively for more than three decades. Many practitioners have proposed several image-based Structure-from-Motion (SfM) and visual Simultaneous Localization and Mapping (SLAM) algorithms to improve the quality, accuracy, and efficiency of 3D reconstruction results. Nevertheless, the 3D reconstruction of human-made indoor structures remains one of the most challenging problems since indoor environments present specific challenges due to their distinctive properties such as lack of textures and dramatic viewpoint changes.This dissertation proposes a novel end-to-end approach for the 3D reconstruction of indoor scenes under the ``Manhattan World'' (MW) constraint, which assumes that visible planes intersect at orthogonal angles. My algorithm recovers planar structures by first computing multi-planar segmentation and motion estimation from consecutive image pairs using MW-constrained homographies. It then proceeds to recover the relative scale at each frame with respect to the global reference system to determine chains of clusters, where each cluster is associated with a plane in the scene. Camera motion and scene planar geometry are then optimized using a novel formulation of Bundle Adjustment. Compared with other state-of-the-art SfM/SLAM algorithms that maintain a representation of individual 3D points in space as in traditional reconstruction techniques, my algorithm estimates 3D plane locations that contain observed points. Using the estimated plane information, the algorithm renders input images using estimated 3D planes in a planar-patch-wise fashion. Thanks to the decreased number of parameters, this enables the generation of dense reconstructions with fewer images and facilitates real-world mobile applications.

Bundle adjustment is a fundamental problem in computer vision, with important applications such as 3D structure reconstruction from 2D images. This paper focuses on large-scale bundle adjustment tasks, e.g., city-wide 3D reconstruction, which require highly efficient solutions. For this purpose, it is common to apply the Levenberg-Marquardt algorithm, whose bottleneck lies in solving normal equations. The majority of recent methods focus on achieving scalability through modern hardware such as GPUs and distributed systems. On the other hand, the core of the solution, i.e., the math underlying the optimizer for the normal equations, remains largely unimproved since the proposal of the classic parallel bundle adjustment (PBA) algorithm, which increasingly becomes a major limiting factor for the scalability of bundle adjustment. This paper proposes parallel preconditioned conjugate gradient (PPCG) method, a novel parallel method for bundle adjustment based on preconditioned conjugate gradient, which achieves significantly higher efficiency and scalability than existing methods on the algorithmic level. The main idea is to exploit the sparsity of the Hessian matrix and reduce its structure parameters through an effective parallel Schur complement method; the result of this step is then fed into our carefully designed PPCG method which reduces matrix operations that are either expensive (e.g., large matrix reverse or multiplications) or scales poorly to multi-processors (e.g., parallel Reduce operators). Extensive experiments demonstrate that PPCG outperforms existing optimizers by large margins, on a wide range of datasets.

A local Bundle Adjustment (BA) on a sliding window of keyframes has been widely used in visual SLAM and proved to be very effective in lowering the drift. But in lidar SLAM, BA method is hardly used because the sparse feature points (e.g., edge and plane) make the exact point matching impossible. In this paper, we formulate the lidar BA as minimizing the distance from a feature point to its matched edge or plane. Unlike the visual SLAM (and prior plane adjustment method in lidar SLAM) where the feature has to be co-determined along with the pose, we show that the feature can be analytically solved and removed from the BA, the resultant BA is only dependent on the scan poses. This greatly reduces the optimization scale and allows large-scale dense plane and edge features to be used. To speedup the optimization, we derive the analytical derivatives of the cost function, up to second order, in closed form. Moreover, we propose a novel adaptive voxelization method to search feature correspondence efficiently. The proposed formulations are incorporated into a LOAM back-end for map refinement. Results show that, although as a back-end, the local BA can be solved very efficiently, even in real-time at 10Hz when optimizing 20 scans of point-cloud. The local BA also considerably lowers the LOAM drift. Our implementation of the BA optimization and LOAM are open-sourced to benefit the community., 8 pages, 9 figures, conference

High-precision measurement based on multi-view geometry benefits from aligning a template CAD model to the multi-view observations of a target object. It not only improves the multi-camera calibration but also assists the measurement by serving as a “scaffold.” A straightforward approach is to reconstruct the target object and perform a 3D registration with the CAD model. However, the accuracy of such 3D alignment cannot meet the high-precision requirement. We formulate the problem as a bundle adjustment where we jointly optimize the 6DoF poses of both the template model and the multiple cameras. To accommodate the manufacturing error of products, we propose a simple and robust solution based on a discrete–continuous optimization which interleaves between correspondence selection and pose optimization. In the discrete step, a robust RANSAC-based selection process selects well-matched 2D–3D feature points according to the current model/camera poses. In the continuous step, it jointly optimizes the 6D poses of the CAD model and the multiple cameras. This interleaving optimization constitutes a novel hybrid bundle adjustment (HBA). In HBA, the reprojection error of a feature point is measured either with the 2D–3D correspondence between the observation image and the CAD model or with the cross-view correspondence between multiple images, whichever is more reliable according to the discrete selection step. Through extensive evaluation on real-world data, we demonstrate that HBA achieves high-precision multi-camera calibration, outperforming alternative approaches significantly.

Most visual simultaneous localization and mapping (SLAM) algorithms rely on geometric features, such as points, lines, and planes, and these algorithms treated all features as equally important in the optimization process, ignoring the semantic information of these features. However, the robust and rich semantic features should play a more important role, which is similar to the mechanism of visual saliency or visual attention. Therefore, this article aims to mimic this mechanism in the visual SLAM framework by using the saliency prediction model. To solve the center bias of the generic salient dataset, we proposed a method for recomputing saliency map by considering both geometric and semantic information. Then, we proposed the salient bundle adjustment (SBA) algorithm by using the value of saliency prediction map as the weight of the feature points in the traditional bundle adjustment (BA) approach. Finally, exhaustive experiments conducted with the state-of-the-art algorithm show that our proposed algorithm outperforms existing algorithms such as direct sparse odometry (DSO) and oriented brief (ORB)-SLAM3 in both indoor and outdoor environments.

Multicamera systems have important applications in industrial online measurement, attracting wide interest due to their encouraging performance. However, how to develop a convenient and high-accuracy multicamera calibration model is a big challenge. Traditional calibration methods based on 2-D objects require the preparation of high-precision and large-scale calibration objects and have poor adaptability to the spatial distribution of cameras. To address these issues, a convenient and accurate multicamera calibration method is proposed based on an imperfect spherical object. Specifically, a special calibration object, i.e., an imperfect sphere with many coded targets is designed for calibrating the multicamera system, where the optical axes of the cameras are different in orientations and converge into the measurement field. Then, a Euclidean reconstruction method is employed to calculate the camera poses and the spatial coordinates of feature points in the local coordinate system after the adaptive grouping of cameras is completed. Moreover, a graph theory-based optimal path transformation algorithm is developed to obtain the camera poses and spatial point coordinates in the global coordinate system (GCS), and a spherical projection optimization algorithm is designed to spheroidize the spatial coordinates of feature points. In the end, the camera poses and the spatial coordinates of feature points in the GCS are optimized by a bundle adjustment algorithm. We build a multicamera system and conduct extensive experiments to demonstrate the superiority of the proposed method.

With the growing maturity of unmanned aerial vehicle (UAV) technology, its applications have widened to many spheres of life. The prerequisite for a UAV to perform air tasks smoothly is an accurate localization of its own position. Traditional UAV navigation relies on the Global Navigation Satellite System (GNSS) for localization; however, this system has disadvantages of instability and susceptibility to interference. Therefore, to obtain accuracy in UAV pose estimation in GNSS-denied environments, a UAV localization method that is assisted by deep learning features of satellite imagery is proposed. With the inclusion of a top-view optical camera to the UAV, localization is achieved based on satellite imageries with geographic coordinates and a digital elevation model (DEM). By utilizing the difference between the UAV frame and satellite imagery, the convolutional neural network is used to extract deep learning features between the two images to achieve stable registration. To improve the accuracy and robustness of the localization method, a local optimization method based on bundle adjustment (BA) is proposed. Experiments demonstrate that when the UAV's relative altitude is 0.5 km, the average localization error of this method under different trajectories is within 15 m.

This article proposes a novel synthetic aperture radar visual-inertial odometry (SAR-VIO) consisting of an SAR and an inertial measurement unit (IMU), which aims to enable the observation platform to complete successfully a continuous observation mission in the context of low-cost demand and lack of enough navigation information. First, we establish the observation models of the SAR in a continuous observation process based on the SAR frequency-domain imaging algorithm and the SAR time-domain imaging algorithm, respectively. With the preintegrated IMU data, we then propose a method for estimating the geographic locations of the matched targets in the SAR images and verify the condition and correctness of the method. The optimization of the track and the locations of the targets is achieved by bundle adjustment according to the minimum reprojection error criterion, and a sparse point-cloud map can be obtained. Finally, these methods and models are organized into a complete SAR-VIO framework, and the feasibility of the framework is verified through experiments.

In practical applications, how to use the complementary strengths of the direct and the feature-based methods for effective fusion may be the main challenge of simultaneous localization and mapping (SLAM). To solve this challenge, we propose the DO-SLAM, a novel fast and accurate semi-direct visual SLAM framework, which can maintain the direct method’s fast performance and the high precision and loop closure capability of the feature-based method. The direct method is used as the first half of the DO-SLAM to track the camera pose rapidly and robustly. The feature-based method is used as the second half of the DO-SLAM to refine the keyframe poses, perform loop closures, and build a globally consistent, long-term, sparse feature map that can be reused. The proposed pipeline fuses direct odometry and feature-based SLAM to perform three levels of parallel optimizations: (1) In the direct method module, the keyframe poses are estimated by minimizing the photometric error, (2) In the feature-based module, using the poses calculated by the inter-frame matching to correct and fuse the poses calculated by the direct method module as the initial poses, and the initial poses are optimized by the motion-only bundle adjustment, and (3) A pose graph optimization is used to achieve global map consistency in the presence of loop closures. Experimental evaluation on two benchmark datasets demonstrates that the proposed approach achieves higher accuracy and robustness on motion estimation compared to the other state-of-the-art methods.

Unmanned aerial vehicles (UAVs) equipped with integrated global navigation satellite systems/inertial navigation systems together with frame and/or line cameras are used for a variety of applications. Geometric system calibration is crucial for delivering accurate products from UAV-based imaging systems. This article presents automated geometric calibration strategies for UAV-based frame and line camera systems to estimate accurate system calibration parameters without the need for ground control points or manual measurements of tie points. The matching strategy used in this article to establish conjugate features among overlapping frame camera images is based on a traditional Structure from Motion technique augmented with several layers of matching outlier removal. On the other hand, a new strategy relying on ortho-rectified images is introduced for automated feature matching in line camera scenes. Then, a general bundle adjustment procedure with system calibration capabilities for frame and line cameras is presented, where the derived automated tie points are used for estimating accurate geometric system calibration parameters. The proposed approach is evaluated using four datasets—two datasets captured by frame cameras and two datasets captured by line cameras. The results show that the developed automated calibration strategy is capable of producing the same level of absolute accuracy when compared to using manually measured tie points for both frame camera and line camera systems. Results also indicate that the presented automated system calibration approach can be applied to systems even with significant deviation of actual system parameters from their nominal values, and still produce accurate estimates of calibration parameters.

In the study of RGB-D SLAM (Simultaneous Localization and Mapping), two types of primary visual features, point and line features, have been widely utilized to calculate the camera pose. As an RGB-D camera can capture RGB and depth information simultaneously, most RGB-D SLAM methods only utilize the 2D information within the point and line features. To obtain a higher accuracy camera pose and utilize the 2D and 3D information within points and lines better, a novel geometric constraint model of points and lines (PL-GM) using an RGB-D camera is proposed in this paper. Our contributions are threefold. Firstly, the 3D points and lines generated by an RGB-D camera combining with 2D point and line features are utilized to establish the PL-GM, which is different from most models of point-line SLAM (PL-SLAM). Secondly, in addition to the 2D re-projection error of point and line features, the constraint errors of 3D points and lines are constructed and minimized likewise, and then a unified optimization model based on PL-GM is extended to the bundle adjustment model (BA). Finally, extensive experiments have been performed on two public benchmark RGB-D datasets and a real scenario sequence. These experimental results demonstrate that our method achieves a comparable or better performance than the state-of-the-art SLAM methods based on point and line features, and point features.

In this article, the autonomous driving problem in an unstructured scene is addressed using a model predictive control (MPC) scheme. The lack of scene structure makes the sensing problem challenging, in particular when considered within a control loop. To circumvent this difficulty, the bundle adjustment (BA) algorithm from computer vision is used to detect obstacles and compute a sparse representation of the environment. In one of the main results of this article, it is shown how this sparse representation can be cast as additional safety constraints to the MPC optimization. The MPC/BA combination is intuitively appealing since they both solve quadratic problems and also because the BA estimations trend to be more accurate when the resulting constraints become active in the MPC solver. This article contains a theoretical presentation of the control scheme and discusses implementation details. An example of the overall approach at work can be seen in https://youtu.be/aU46vpzDHso .

This paper aims to correct camera pose estimation results acquired by bundle adjustment (BA) optimization in visual-inertial SLAM (VI-SLAM) approaches. By deriving the relationship between the pose...

Unmanned aerial vehicle (UAV) images have gained extensive attention in varying fields, and the Structure from Motion (SfM) technique has become the gold standard for aerial triangulation of UAV images. With increasing data volume caused by the use of multi-view and high-resolution imaging systems and the enhancement of UAV platform’s endurance, the capability for orientation of large-scale UAV images is becoming a prominent and necessary feature for SfM-based solutions. A classical SfM pipeline consists of three major steps, i.e., (i) feature extraction for an individual image, (ii) feature matching for each image pair, and (iii) parameter solving based on iterative bundle adjustment. Most of the time costs are consumed in the second and third steps. This can be explained from three main aspects. First, for feature matching the large number of images and high overlapping degrees cause high combinational complexity of match pairs. Second, the efficiency of commonly utilized techniques for outlier removal would be seriously degenerated because of high outlier ratios of initial matches. Third, for parameter solving of camera poses and scene structures, the iterative execution of bundle adjustment (BA) leads to high computational costs in the incremental SfM workflow. Thus, this paper gives a systematic survey of the state-of-the-art for match pair selection from both ordered and unordered datasets, for outlier removal of initial matches dominated by outliers, and for efficiency improvement of BA, and conducts an experimental evaluation for six well-known SfM-based software packages on UAV image orientation.

Industrial micro aerial vehicles (MAVs) with robotic manipulators have numerous applications in search and rescue tasks that reduce risks to human beings. However, such tasks distinctly require MAVs to have the capability of real-time autonomous navigation only with onboard sensors, especially in GPS-denied applications. This article introduces a new approach to onboard vision-based autonomous state estimation and mapping for MAVs’ navigation in unknown environments. The algorithms run on board and do not need an external positioning system to assist autonomous navigation. The state estimator is developed to provide MAV's current pose on the basis of the extended Kalman filter by using image patch features. Inverse depth convergence monitoring and local bundle adjustment are utilized to improve the accuracy. The mapping algorithm for navigation is developed according to a real-time stereo matching method for three-dimensional perception. Finally, we have performed several experiments to demonstrate the effectiveness of the proposed approach.

Forward-looking sonar (FLS) and camera are widely used sensors in multimodal systems for underwater environments. The extrinsic calibration is a precondition for this kind of multimodal systems to fuse optical and acoustic information. Due to the very different imaging formation mechanisms and models of 3-D-to-2-D projection for imaging sonar and optical camera, the multimodal correspondence and the sonar-pose recovery from a 2-D sonar image are two main issues in extrinsic calibration work. In this article, we propose an extrinsic calibration method for the opti-acoustic imaging system combining a camera and an FLS in underwater environment. The proposed method includes an estimation process yielding closed-form expressions to determine the sonar pose with respect to the calibration board, which enables the estimation of extrinsic parameters to be directly computed from sensor measurements and the knowledge of calibration-board structure. A calibration board with two sets of feature arrays is used to generate two 2-D-to-3-D correspondences, which circumvents the 2-D-to-2-D multimodal correspondence problem. An iterative optimization process is provided to further improve the performance, while the extra computation time is negligible. Both simulated and actual experiments show that the method outperforms the existing iterative method based on bundle adjustment.

This paper proposes an enhanced tightly-coupled sensor fusion scheme using a monocular camera and ultra-wideband (UWB) ranging sensors for the task of simultaneous localization and mapping. By leveraging UWB data, the method can achieve metric-scale, drift-reduced odometry and a map consisting of visual landmarks and UWB anchors without knowing the anchor positions. Firstly, the UWB configuration accommodates any degenerate cases with an insufficient number of anchors for 3D triangulation ( $$N\le 3$$ and no height data). Secondly, a practical model for UWB measurement is used, ensuring more accurate estimates for all the states. Thirdly, selected prior range measurements including the anchor-world origin and anchor–anchor ranges are utilized to alleviate the requirement of good initial guesses for anchor position. Lastly, a monitoring scheme is introduced to appropriately fix the scale factor to maintain a smooth trajectory as well as the UWB anchor position to fuse camera and UWB measurement in the bundle adjustment. Extensive experiments are carried out to showcase the effectiveness of the proposed system.

Generally, the distributed bundle adjustment (DBA) method uses multiple worker nodes to solve the bundle adjustment problems and overcomes the computation and memory storage limitations of a single computer. However, the performance considerably degrades owing to the overhead introduced by the additional block partitioning step and synchronous waiting. Therefore, we propose a low-overhead consensus framework. A partial barrier based asynchronous method is proposed to early achieve consensus with respect to the faster worker nodes to avoid waiting for the slower ones. A scene summarization procedure is designed and integrated into the block partitioning step to ensure that clustering can be performed on the small summarized scene. Experiments conducted on public datasets show that our method can improve the worker node utilization rate and reduce the block partitioning time. Also, sample applications are demonstrated using our large-scale culture heritage datasets.

Visual odometry (VO) is a technique applied to track the dynamic positioning and orientation of a moving platform with one or more cameras taking image sequences. The determination relies on the estimation of relative orientation parameters (ROPs) of time adjacent images. The idea of stereo VO to develop a dual-camera system is adopted in this study. By taking advantage of the calibrated stereo camera, this system is able to recover the true scale of relative translation without the need from additional sensors. However, the scale might not be very accurate, and the error also could exist in the orientation including rotation and translation due to environmental factors such as the illumination and texture. Therefore, the primary objective of this study is to find the optimized theory and method of stereo VO. Through the analysis of the geometric relationship of the time adjacent stereo image pairs, locally optimized network adjustment is developed to improve the accuracy of ROPs.The proposed network adjustment model is verified by the simulation data and experiment data both. ROPs are adopted as observations that would update the states of the image sequence further. Besides, exterior orientation parameters (EOPs) of the dual-camera system could be optimized obviously during the whole operation. In this study, it is worth mentioning that 3D coordinates of object points matched in each image pair are not necessary to be calculated. The conventional bundle adjustment is not adopted, but more accurate EOPs still have been generated automatically during the process.

In this paper, we present a new technique of tridimensional reconstruction from a sequence of uncalibrated stereo images taken with cameras having varying parameters. At first, our system allows to recover initial coordinates of a set of 3D points. In this context, we have used our method of self-calibration based on the use of unknown 3D scene with its image projections and genetic algorithms to estimate all intrinsic parameters. After that extrinsic parameters are estimated based on classical pose estimation algorithms. Matching points and estimated value of intrinsic and extrinsic parameters are used to estimate initial 3D model that helps us in the initialization step. In order to have a reliable and relevant 3D reconstruction the proposed method is based on good and new exploitation of bundle adjustment (without camera poses initialization) technique based on Levenberg-Marquardt optimization with the aim to estimate our optimal 3D model that has special features compared to the classical case because it masks the pose parameters estimation in the optimization process. Finally, 3D mesh of the 3D scene is constructed with Delaunay algorithm and the 2D image is projected on the 3D model to generate the texture mapping. Experiments is conducted on real data to achieve demonstrate the validity and the performance of the proposed approach in terms of convergence, simplicity, stability and reconstruction quality.

Localization is one of the first steps in navigation. Especially due to the rapid development in automated driving, a precise and reliable localization becomes essential. In this paper, we report an investigation of the usage of dynamic ground control points (GCP) in visual localization in an automotive environment. Instead of having fixed positions, dynamic GCPs move together with the camera. As a measure of quality, we employ the precision of the bundle adjustment results. In our experiments, we simulate and investigate different realistic traffic scenarios. After investigating the role of tie points, we compare an approach using dynamic GCPs to an approach with static GCPs to answer the question how a comparable precision can be reached for visual localization. We show, that in our scenario, where two dynamic GCPs move together with a camera, similar results are indeed obtained to using a number of static GCPs distributed over the whole trajectory. In another experiment, we take a closer look at sliding window bundle adjustments. Sliding windows make it possible to work with an arbitrarily large number of images and to still obtain near real-time results. We investigate this approach in combination with dynamic GCPs and vary the no. of images per window.

The proper function of rail-based transport networks relies on the accurate positioning of the tracks. Regular control and maintenance intervals are in place to guarantee safe and reliable operation. This also holds for the crane rails of the storage cranes in the container terminal in the Hamburg harbour. Especially in the terminal “Altenwerder” the geomorphological conditions of the soil lead to a permanent subsidence of the tracks and thus ask for intensive surveying and maintenance activities. The allowed tolerances are in the range of 10mm in the XY-plane on a stretch of 300m. In the daily practice, the measurements are done using traditional tachymetric survey, in combination with a rail car carrying a reflector. This method is reliable but comes with the disadvantage that the operation of cranes needs to be interrupted. In this paper we present an alternative, automatic approach which employs state-of-the-art UAV-based photogrammetry to measure the actual location of the rail. The mid-format camera system combined with a 150mm tele-lens results in a GSD of 0.9mm at 35m flying height. Challenges addressed concern the proper setup and installation of the ground control network, the flight planning and bundle adjustment. Furthermore, an automated rail delineation in the derived surface model was developed. First experiments show that an automatic workflow is possible, including the delineation task. Remaining obstacles concern, for instance, the compliance with the requirements regarding absolute positional accuracy, since the inner block geometry is theoretically much more accurate than the realised control point network.

In recent years, the demand for inexpensive, simple, and highly accurate 3D measurement has been increasing. Representative methods, photogrammetry, and shape from focus (SfF) have limitations in terms of measurement time and labour. In order to solve them, computational photography (CP) has been proposed. A light field camera, based on CP, has also been developed. It has a feature to acquire multi-view and multi-focus images simultaneously in one shot. It is possible to perform 3D measurements with less time and labour for photographing and calculation processing using these images. In this study, we combined the photogrammetry as applied to multi-view images with the SfF as applied to multi-focus images using a light field camera. We applied the proposed method to a rigid body and verified its accuracy. We confirmed that the proposed method achieved more accurate results than the photogrammetry and the SfF method. Furthermore, we applied the proposed method to screws and cracks on walls of buildings and affirmed its applicability. Finally, we suggested future work on the developed method.

This paper deals with bundle adjustment with constrained cameras, i.e. where the orientation of certain cameras is expressed relatively to others, and these relative orientations are part of the unknowns. Despite the remarkable interest for oblique multi-camera systems, an empirical study on the effect of enforcing relative orientation constraints in bundle adjustment is still missing. We provide experimental evidence that indeed these constraints improve the accuracy of the results, while reducing the computational load as well. Moreover, we report for the first time in the literature the complete derivation of the Jacobian matrix for bundle adjustment with constrained cameras, to foster other implementations.

The global approaches solve SfM problems by independently inferring relative motions, followed be a sequential estimation of global rotations and translations. It is a fast approach but not optimal because it relies only on pairs and triplets of images and it is not a joint optimisation. In this publication we present a methodology that increases the quality of global solutions without the usual computational burden tied to the bundle adjustment. We propose an efficient structure approximation approach that relies on relative motions known upfront. Using the approximated structure, we are capable of refining the initial poses at very low computational cost. Compared to different benchmark datasets and software solutions, our approach improves the processing times while maintaining good accuracy.

Obtaining accurate image interior and exterior orientations is the key to improve 3D measurement accuracy besides reliable and accurate image matching. A majority of cameras used for those tasks are non-metric cameras. Non-metric cameras commonly suffer various distortions. Generally, there are two ways to remove these distortions: 1) conducting prior camera calibration in a controlled environment; 2) applying self-calibrating bundle adjustment in the application environment. Both approaches have their advantages and disadvantages but one thing is common that there is no universal calibration model available so far which can remove all sorts of distortions on images and systemic errors of image orientations. Instead of developing additional calibration models for camera calibration and self-calibrating adjustment, this paper presents a novel approach which applies self-calibrating bundle adjustment in an iterative fashion: after performing a conventional self-calibrating bundle adjustment, the image coordinates of tie points are re-calculated using the newly obtained self-calibration model coefficients, and the self-calibrating bundle adjustment is applied again in the hope that the remaining distortions and systematic errors will be reduced further within next a few iterations. Using a “virtual image” concept this iterative approach does not require to resample images or/and re-measure tie points during iterations, only costs a few additional iterations computational resource. Several trails under various application environments are conducted using this proposed iterative approach and the results indicate that not only the distortions can be reduced further but also image orientations become much stable after a few iterations.

Advances in digitalization technologies lead to rapid and massive changes in infrastructure management. New collaborative processes and workflows require detailed, accurate and up-to-date 3D geodata. Image-based web services with 3D measurement functionality, for example, transfer dangerous and costly inspection and measurement tasks from the field to the office workplace. In this contribution, we introduced an image-based backpack mobile mapping system and new georeferencing methods for capture previously inaccessible outdoor locations. We carried out large-scale performance investigations at two different test sites located in a city centre and in a forest area. We compared the performance of direct, SLAM-based and image-based georeferencing under demanding real-world conditions. Both test sites include areas with restricted GNSS reception, poor illumination, and uniform or ambiguous geometry, which create major challenges for reliable and accurate georeferencing. In our comparison of georeferencing methods, image-based georeferencing improved the median precision of coordinate measurement over direct georeferencing by a factor of 10–15 to 3 mm. Image-based georeferencing also showed a superior performance in terms of absolute accuracies with results in the range from 4.3 cm to 13.2 cm. Our investigations showed a great potential for complementing 3D image-based geospatial web-services of cities as well as for creating such web services for forest applications. In addition, such accurately georeferenced 3D imagery has an enormous potential for future visual localization and augmented reality applications.

In this paper we propose a new SLAM algorithm that is robust to the changing environment of the countryside. The hardware part consists of two separate machine vision cameras, joined in stereo, and can be supplemented with LiDAR, IMU and GPS. We introduce a method that can be used to reliably calculate the position of a vehicle in natural environments. To estimate the pose and produce a three-dimensional reconstruction we use a stereo camera rig, inertial measurement unit and the global positioning system. While solving the problem of visual odometry in outdoor scenes we faced a number of difficulties, arising from high dynamic range, as well as the presence of a large number of "similar elements", such as leaves, grass, trees and etc. Under these conditions, it becomes difficult to match feature points in image sequences. HOG-based methods, such as SIFT, SURF and others often do not obtain good matching due to noise, lack of a sufficient number of gradients, and the presence of identical domains. Using neighborhood-based detectors such as DAISY often allows to identify the correct matches, but using them is worth it too expensive. These methods are very demanding on the computational resources and prone to drift. We needed a method that is less expensive, but at the same time provides sufficient accuracy in the trajectory estimation. Direct methods, such as optical flow calculating or direct image matching allow us to map point-to-point in these conditions with high reliability. They also have disadvantages that can be eliminated by using an IMU and modern algorithms. To improve the quality of the algorithms, we solve the reconstruction problem for several frames using the Levenberg-Marquardt optimization method for bundle adjustment. Each pass optimizes frames that are directly related to the last one, we use two threads that perform partial and full optimization of the entire trajectory using graphs to significantly increase the performance of the method.

The article presents with the use of archive aerial photographs. The first task was to search and identify drainage detail from archive aerial photographs. The second task is to create procedures for processing aerial reconnaissance images (from WWII) to identify sites with potential pyrotechnic load. Both of these tasks are connecting by the effort to determine the internal orientation parameters of the cameras for using and easier calculation of exterior parameters by image correlation. Complete automation process searching of fiducial mark (FM) identification was implemented. The coordinates of all FM are calculated automatically from archive aerial photographs. In addition, the edges of the photographs are automatically found and a program was created to minimize of the cropping of the archive aerial photographs. The next part of the paper describes the procedures of averaging the values of the relative position of FM and transforming archive aerial photographs to a uniform dimension from a set of images taken with the same camera. The second part of the paper describes the process of creating a historical ortophoto with the standard calculation of bundle adjustment performed by an external process in the background of the OrthoEngine module using the Celery library installed as a python service. Finding of external image orientation parameters through bundle adjustment calculation are parameters, in the first, defined in the local system and then transformed into the national geodetic system of the Czech Republic. This entire section is available and free to use for on the internet. The third part of the article describes the practical procedure of the interpretation of archive and wartime photographs with aim of identification of the drainage detail and the procedures leading to the interpretation, identification, location and calculation of the position of unexploded air ammunition.

Augmented reality is a technology that visualizes and synthesizes virtual objects and information in the real world. Augmented reality can provide useful information to the real world realistically. As the investment and demand for augmented reality increase, many researches have been conducted to visualize objects based on ground coordinate system such as cadastral information and underground facilities using augmented reality. In order to visualize objects based on the ground coordinate system on a smartphone using augmented reality, high accuracy of the position and attitude of the smartphone is essential. Accordingly, this study proposed position and attitude correction method for the first image using reference images and single photo resection. Then, the absolute position of the entire image was estimated using the first calibrated image and the tracking algorithm. When the reference image-based correction method was used, the accuracy was 0.74m, 0.94m. And when the single photo resection correction method was used, the accuracy was 3.13m, 1.24m. The method presented in this study showed higher accuracy when only the sensor was used, but it was confirmed that errors in position and attitude occur as the tracking error accumulates.

Due to the increasing range of work carried out with UAV in recent years, the importance of final product accuracy appreciates. However, obtaining survey-grade accuracy requires to perform bundle adjustment processes that could be affected by multiple factors like unstable camera calibration, a correlation between interior and exterior orientation, insufficient georeferenced information, and software settings. During the project, multi-variant flight over the test field was conducted. The flights were performed with a fixed-wing airframe equipped with PPK receiver on-board. Based on the conducted flights, the database for multifactorial data sets has been prepared. The database containing hundreds of independent adjustment variants which differ as follows: georeferencing method, flight configuration, additional camera calibration corrections, tie points filtering, and a priori accuracy settings. The database allowed to investigate the separate influence of each factor on the final results using ANOVA statistical models.

In this article, a structure from motion (SfM) framework for oblique aerial images of man-made environments is proposed, covering the issues of determining overlapping images, feature extraction, i...