Search

As urbanisation accelerates, the urban landscape reshapes at a fast pace. Consequently, the urban built environment continuously evolves horizontally as well as vertically. However, more attention in the field of spatial analysis is given to horizontal dynamics, despite the importance of geoinformation in the vertical dimension. 3D modelling methods have gained popularity due to their powerful capability of capturing and analysing geoinformation in the vertical dimension and visualising objects lifelike in the urban built environment. Various urban applications with diverse 3D modelling methods at different research scales and purposes have emerged. However, there is no systematic overview of these different modelling methods. Therefore, it is imperative to provide an up-to-date review of these advances. In this paper, we aim to review urban 3D modelling methods widely used in the prior 5-year period (2015–2020). Our analysis focuses on five attributes, i.e., basic characteristics, data requirements, technical requirements, users requirements and ethical considerations. The discussion presents the current status of 3D modelling methods – a wide range of applications yet with substantial development potential. This paper closes with insights for future work regarding the necessities of 3D data structure support as well as interdisciplinary research, specifically for big data management and integration.

This paper presents our contribution to the development of a standardized 3D input data model for solar photovoltaic potential estimation. Presently, different input data and processing steps influence the calculation for estimating the potential of solar energy in the Netherlands. The variety in characteristics of input data and issues with temporal accuracy extracted from the national registers and databases makes it challenging to obtain a consistent and reliable result. To address this issue, we created a point cloud dataset that integrated from LiDAR point cloud and dense image matching which is complete, recent and positionally accurate. Furthermore, we made a 3D building model from the integrated point cloud and identified the effect of finer resolution in the photovoltaic potential analysis.

Precise and accurate three-dimensional geospatial data has become increasingly available thanks to advances in both Terrestrial Laser Scanning (TLS) and Structure-from-Motion Photogrammetry (SfM). These tools provide valuable information for mapping geomorphological features and detect surface changes in mountainous environments. The exploitation of 3D point-clouds has been proven tremendously useful in the field of geosciences. It remains, however, controversial whether cost efficient photogrammetry can provide as accurate and reliable geospatial information as the significantly more expensive laser scanning or not. In this study, a rockfall case site in the territory of Obergurgl, Austria, is investigated in order to provide answers to the above question in a complex environment. The analysis includes different terrestrial photogrammetry configurations aiming to comprehensively define the strengths and limitations of terrestrial photogrammetry over TLS. The latter constitutes an optimized methodology that provides guidelines for costly future assessments as part of the site investigation phase in geohazard management. There are no doubts that compared to traditional and conventional surveying methods TLS and Photogrammetry both offer products much faster and with a much higher data density. In the current study, we show that when photogrammetry is applied following a well-defined optimized strategy, it can be potentially an adequate alternative to more costly TLS datasets for mass movement assessment and monitoring purposes.

Deep learning methods based on Fully convolution networks (FCNs) have shown an impressive progress in building outline delineation from very high resolution (VHR) remote sensing (RS) imagery. Common issues still exist in extracting precise building shapes and outlines, often resulting in irregular edges and over smoothed corners. In this paper, we use PolyMapper, a recently introduced deep-learning framework that is able to predict object outlines in a vector representation directly. We have introduced two main modifications to this baseline method. First, we introduce EffcientNet as backbone feature encoder to our network, which uses compound coefficient to scale up all dimensions of depth/width/resolution uniformly, to improve the processing speed with fewer parameters. Second, we integrate a boundary refinement block (BRB) to strengthen the boundary feature learning and to further improve the accuracy of corner prediction. The results demonstrate that the end-to-end learnable model is capable of delineating polygons of building outlines that closely approximate the structure of reference labels. Experiments on the crowdAI building instance segmentation datasets show that our model outperforms PolyMapper in all COCO metrics, for instance showing a 0.13 higher mean Average Precision (AP) value and a 0.60 higher mean Average Recall value. Also qualitative results show that our method segments building instances of various shapes more accurately.

One of the major challenges in precision viticulture in Europe is the detection and mapping of flavescence dorée (FD) grapevine disease to monitor and contain its spread. The lack of effective cures and the need for sustainable preventive measures are nowadays crucial issues. Insecticides and the plants uprooting are commonly employed to withhold disease infection, even if these solutions imply serious economic consequences and a strong environmental impact. The development of a rapid strategy to identify the disease is required to cover large portions of the crop and thus to limit damages in a time-effective way. This paper investigates the use of Unmanned Aerial Vehicles (UAVs), a cost-effective approach to early detection of diseased areas. We address this task with an object detection deep network, Faster R-CNN, instead of a traditional pixel-wise classifier. This work tests Faster R-CNN performance on this specific application through a comparative analysis with a pixel-wise classification algorithm (Random Forest). To take advantage of the full image resolution, the experimental analysis is performed using the original UAV imagery acquired in real conditions (instead of the derived orthomosaic). The first result of this paper is the definition of a new dataset for FD disease identification by UAV original imagery at the canopy scale. Moreover, we demonstrate the feasibility of applying Faster-R-CNN as a quasi-real-time alternative solution to semantic segmentation. The trained Faster-R-CNN achieved an average precision of 82% on the test set.

Dynamic Time Warping (DTW) has been successfully used for crops mapping due to its capability to achieve good classification results when a reduced number of training samples and irregular satellite image time series is available. Despite its recognized advantages, DTW does not account for the duration and seasonality of crops and local differences when assessing the similarity between two temporal sequences. In this study, we implemented a Weighted Derivative modification of DTW (WDDTW) and compared it with DTW and Time Weighted Dynamic Time Warping (TWDTW) for crops mapping. We show that WDDTW outperformed DTW achieving an overall accuracy of 67 %, whereas DTW obtained an accuracy of 57%. Yet, TWDTW performed better than both methods obtaining an accuracy of 88%.

Land use is an important variable in remote sensing which describes the functions carried out on a piece of land in order to obtain benefits and is especially useful to the personnel working in the fields of urban management and planning. The land use information is maintained by national mapping agencies in geo-spatial databases. Commonly, land use data is stored in the form of polygon objects; the label of the object indicates land use. The main goal of classification of land use objects is to update an existing database in an automatic process. Recently, Convolutional Neural Networks (CNN) have been widely used to tackle this task utilizing high resolution aerial images (and derived data such as digital surface model). One big challenge classifying polygons is to deal with the large variation in their geometrical extent. For this challenge, we adopt the method of Yang et al. (2019) to decompose polygons into regular patches of fixed size. The decomposition leads to two sets of polygons: small and large, where the former suffers from a lower identification rate. In this paper, we propose CNN methods which incorporate dense connectivity and integrate it with intermediate information via global average pooling to improve land use classification, mainly focusing on small polygons. We present different network variants by incorporating intermediate information via global average pooling from different stages of the network. We test our methods on two sites; our experiments show that the dense connectivity and integration of intermediate information has a positive effect not only on the classification accuracy on the whole but also on the identification of small polygons. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

Crop lodging – the bending of crop stems from the vertical – is a major yield-reducing factor in cereal crops and causes deterioration in grain quality. Accurate assessment of crop lodging is important for improving estimates of crop yield losses, informing insurance loss adjusters and influencing management decisions for subsequent seasons. The role of remote sensing data, particularly synthetic aperture radar (SAR) data has been emphasized in the recent literature for crop lodging assessment. However, the effect of lodging on SAR scattering mechanisms is still unknown. Therefore, this research aims to understand the possible change in scattering mechanisms due to lodging by investigating SAR image pairs before and after lodging. We conducted the study in 26 wheat fields in the Bonifiche Ferraresi farm, located in Jolanda di Savoia, Ferrara, Italy. We measured temporal crop biophysical (e.g. crop angle) parameters and acquired multi-incidence angle RADARSAT-2 (R-2 FQ8-27° and R-2 FQ21-41°) and Sentinel-1 (S-1 40°) images corresponding to the time of field observations. We extracted metrics of SAR scattering mechanisms from RADARSAT-2 and Sentinel-1 image pairs in different zones using the unsupervised H/α decomposition algorithm and Wishart classifier. Contrasting results were obtained at different incidence angles. Bragg surface scattering increased in the case of S-1 (6.8%), R-2 FQ8 (1.8%) while at R-2 FQ21, it decreased (8%) after lodging. The change in double bounce scattering was more prominent at low incidence angle. These observations can guide future use of SAR-based information for operational crop lodging assessment in particular, and sustainable agriculture in general.

Deforestation is a wide-reaching problem, responsible for serious environmental issues, such as biodiversity loss and global climate change. Containing approximately ten percent of all biomass on the planet and home to one tenth of the known species, the Amazon biome has faced important deforestation pressure in the last decades. Devising efficient deforestation detection methods is, therefore, key to combat illegal deforestation and to aid in the conception of public policies directed to promote sustainable development in the Amazon. In this work, we implement and evaluate a deforestation detection approach which is based on a Fully Convolutional, Deep Learning (DL) model: the DeepLabv3+. We compare the results obtained with the devised approach to those obtained with previously proposed DL-based methods (Early Fusion and Siamese Convolutional Network) using Landsat OLI-8 images acquired at different dates, covering a region of the Amazon forest. In order to evaluate the sensitivity of the methods to the amount of training data, we also evaluate them using varying training sample set sizes. The results show that all tested variants of the proposed method significantly outperform the other DL-based methods in terms of overall accuracy and F1-score. The gains in performance were even more substantial when limited amounts of samples were used in training the evaluated methods. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

With the rapid urbanization and the dynamics in land transactions, Turkish cadastre, as in many other countries, is facing challenges in the daily recording of sales, donations, taxations, and mortgages on land. Especially with the modern constructions and complex interrelationships of rights, restrictions, and responsibilities (RRR), the third dimension should not be forgotten. In this context, the design of a new cadastre data model for Turkey is essential. Inspired by this need, with this study, we aim in proposing a new cadastre model based on international standards such as Land Administration Data Model (LADM) and CityGML. LADM represents the legal and administrative aspects of the cadastral objects but does not show the semantic and 3D geometrical representation of physical cadastral objects which are required for the process of 3D cadastre. Therefore, in the paper, we propose an Application Domain Extension (ADE) for the cadastral objects that expands the integration of LADM and CityGML data model with the legal and administrative concepts defined in the Turkish Law. The study presents a detailed overview of the Turkish legal cadastre system and a proposal for its physical realization based on international standards. In addition, the developed ADE is also valuable for cadastral services undertaken by the General Directorate of Land Registry and Cadastre (GDLRC). It could also be used as a basis of a 3D national data standard for cadastral information systems.

This study investigates the ongoing postseismic deformation induced by two moderate mainshocks of Mw 6.1 and Mw 6.0, 2017 Hojedk earthquake in Southern Iran. Available Sentinel-1 TOPS C-band Synthetic Aperture Radar (SAR) images over about one year after the earthquakes are used to analyze the postseismic activities. An adaptive method incorporating Independent Component Analysis (ICA) and multi-temporal Small BAseline Subset (SBAS) Interferometric SAR (InSAR) techniques is proposed and implemented to recover the rapid deformation. This method is applied to the series of interferograms generated in a fully constructed SBAS network to retrieve the postseismic deformation signal. ICA algorithm uses a linear transformation to decompose the input mixed signal to its source components, which are non-Gaussian and mutually independent. This analysis allows extracting the low rate postseismic deformation signal from a mixture of interferometric phase components. The independent sources recovered from the multi-temporal InSAR dataset are then analyzed using a group clustering test aiming to identify and enhance the undescribed deformation signal. Analysis of the processed interferograms indicates a promising performance of the proposed method in determining tectonic deformation. The proposed method works well, mainly when the tectonic signal is dominated by the undesired signals, including atmosphere or orbital/unwrapping noise that counts as temporally uncorrelated components.In contrast to the standard SBAS time series method, the ICA-based time series analysis estimates the cumulative deformation with no prior assumption about elevation dependence of the interferometric phase or temporal nature of the tectonic signal. Application of the method to 433 Sentinel-1 pairs within the dataset reports two distinct deformation patches corresponding to the postseismic deformation. Besides the performance of the ICA-based analysis, the proposed method automatically detects rapid or low rate tectonic processes in unfavorable conditions.

Recently, great progress has been made in formulating dense disparity estimation as a pixel-wise learning task to be solved by deep convolutional neural networks. However, most resulting pixel-wise disparity maps only show little detail for small structures. In this paper, we propose a two-stage architecture: we first learn initial disparities using an initial network, and then employ a disparity refinement network, guided by the initial results, which directly learns disparity corrections. Based on the initial disparities, we construct a residual cost volume between shared left and right feature maps in a potential disparity residual interval, which can capture more detailed context information. Then, the right feature map is warped with the initial disparity and a reconstruction error volume is constructed between the warped right feature map and the original left feature map, which provides a measure of correctness of the initial disparities. The main contribution of this paper is to combine the residual cost volume and the reconstruction error volume to guide training of the refinement network. We use a shallow encoder-decoder module in the refinement network and do learning from coarse to fine, which simplifies the learning problem. We evaluate our method on several challenging stereo datasets. Experimental results demonstrate that our refinement network can significantly improve the overall accuracy by reducing the estimation error by 30% compared with our initial network. Moreover, our network also achieves competitive performance compared with other CNN-based methods. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

Updated information on urban land use allows city planners and decision makers to conduct large scale monitoring of urban areas for sustainable urban growth. Remote sensing data and classification methods offer an efficient and reliable way to update such land use maps. Features extracted from land cover maps are helpful on performing a land use classification task. Such prior information can be embedded in the design of a deep learning based land use classifier by applying a multitask learning setup—simultaneously solving a land use and a land cover classification task. In this study, we explore a fully convolutional multitask network to classify urban land use from very high resolution (VHR) imagery. We experimented with three different setups of the fully convolutional network and compared it against a baseline random forest classifier. The first setup is a standard network only predicting the land use class of each pixel in the image. The second setup is a multitask network that concatenates the land use and land cover class labels in the same output layer of the network while the other setup accept as an input the land cover predictions, predicted by a subpart of the network, concatenated to the original input image patches. The two deep multitask networks outperforms the other two classifiers by at least 30% in average F1-score.

Modern machine learning, especially deep learning, which is used in a variety of applications, requires a lot of labelled data for model training. Having an insufficient amount of training examples leads to models which do not generalize well to new input instances. This is a particular significant problem for tasks involving aerial images: often training data is only available for a limited geographical area and a narrow time window, thus leading to models which perform poorly in different regions, at different times of day, or during different seasons. Domain adaptation can mitigate this issue by using labelled source domain training examples and unlabeled target domain images to train a model which performs well on both domains. Modern adversarial domain adaptation approaches use unpaired data. We propose using pairs of semantically similar images, i.e., whose segmentations are accurate predictions of each other, for improved model performance. In this paper we show that, as an upper limit based on ground truth, using semantically paired aerial images during training almost always increases model performance with an average improvement of 4.2% accuracy and .036 mean intersection-over-union (mIoU). Using a practical estimate of semantic similarity, we still achieve improvements in more than half of all cases, with average improvements of 2.5% accuracy and .017 mIoU in those cases.

Land use (LU) is an important information source commonly stored in geospatial databases. Most current work on automatic LU classification for updating topographic databases considers only one category level (e.g. residential or agricultural) consisting of a small number of classes. However, LU databases frequently contain very detailed information, using a hierarchical object catalogue where the number of categories differs depending on the hierarchy level. This paper presents a method for the classification of LU on the basis of aerial images that differentiates a fine-grained class structure, exploiting the hierarchical relationship between categories at different levels of the class catalogue. Starting from a convolutional neural network (CNN) for classifying the categories of all levels, we propose a strategy to simultaneously learn the semantic dependencies between different category levels explicitly. The input to the CNN consists of aerial images and derived data as well as land cover information derived from semantic segmentation. Its output is the class scores at three different semantic levels, based on which predictions that are consistent with the class hierarchy are made. We evaluate our method using two test sites and show how the classification accuracy depends on the semantic category level. While at the coarsest level, an overall accuracy in the order of 90% can be achieved, at the finest level, this accuracy is reduced to around 65%. Our experiments also show which classes are particularly hard to differentiate. © 2020 Copernicus GmbH. All rights reserved.

In recent years, the determination of global image orientation, i.e. global SfM, has gained a lot of attentions from researchers, mainly due to its time efficiency. Most of the global methods take relative rotations and translations as input for a two-step strategy comprised of global rotation averaging and global translation averaging. This paper by contrast presents a hybrid approach that aims to solve global rotations and translations simultaneously, but hierarchically. We first extract an optimal minimum cover connected image triplet set (OMCTS) which includes all available images with a minimum number of triplets, all of them with the three related relative orientations being compatible to each other. For non-collinear triplets in the OMCTS, we introduce some basic characterizations of the corresponding essential matrices and solve for the image pose parameters by averaging the constrained essential matrices. For the collinear triplets, on the other hand, the image pose parameters are estimated by relative orientation using the depth of object points from individual local spatial intersection. Finally, all image orientations are estimated in a common coordinate frame by traversing every solved triplet using a similarity transformation. We show results of our method on different benchmarks and demonstrate the performance and capability of the proposed approach by comparing with other global SfM methods.

Large-scale mapping of the Brazilian Savanna (Cerrado) vegetation using remote sensing images is still a challenge due to the high spatial variability and spectral similarity of the different characteristic vegetation types (physiognomies). In this paper, we report on semantic segmentation of the three major groups of physiognomies in the Cerrado biome (Grasslands, Savannas and Forests) using a fully convolutional neural network approach. The study area, which covers a Brazilian conservation unit, was divided into three regions to enable testing the approach in regions that were not used in the training phase. A WorldView-2 image was used in cross validation experiments, in which the average overall accuracy achieved with the pixel-wise classifications was 87.0%. The F-1 score values obtained with the approach for the classes Grassland, Savanna and Forest were of 0.81, 0.90 and 0.88, respectively. Visual assessment of the semantic segmentation outcomes was also performed and confirmed the quality of the results. It was observed that the confusion among classes occurs mainly in transition areas, where there are adjacent physiognomies if a scale of increasing density is considered, which agrees with previous studies on natural vegetation mapping for the Cerrado biome.

Many countries were the target of air strikes during World War II. Numerous unexploded bombs still exist in the ground. These duds can be tracked down with the help of bomb craters, indicating areas where unexploded bombs may be located. Such areas are documented in so-called impact maps based on detected bomb craters. In this paper, a stochastic approach based on marked point processes (MPPs) for the automatic detection of bomb craters in aerial images taken during World War II is presented. As most areas are covered by multiple images, the influence of redundant image information on the object detection result is investigated: we compare the results generated based on single images with those obtained by our new approach that combines the individual detection results of multiple images covering the same location. The object model for the bomb craters is represented by circles. Our MPP approach determines the most likely configuration of objects within the scene. The goal is reached by minimizing an energy function that describes the conformity with a predefined model by Reversible Jump Markov Chain Monte Carlo sampling in combination with simulated annealing. Afterwards, a probability map is generated from the automatic detections via kernel density estimation. By setting a threshold, areas around the detections are classified as contaminated or uncontaminated sites, respectively, which results in an impact map. Our results show a significant improvement with respect to its quality when redundant image information is used.

Deep learning classification models require large amounts of labeled training data to perform properly, but the production of reference data for most Earth observation applications is a labor intensive, costly process. In that sense, transfer learning is an option to mitigate the demand for labeled data. In many remote sensing applications, however, the accuracy of a deep learning-based classification model trained with a specific dataset drops significantly when it is tested on a different dataset, even after fine-tuning. In general, this behavior can be credited to the domain shift phenomenon. In remote sensing applications, domain shift can be associated with changes in the environmental conditions during the acquisition of new data, variations of objects' appearances, geographical variability and different sensor properties, among other aspects. In recent years, deep learning-based domain adaptation techniques have been used to alleviate the domain shift problem. Recent improvements in domain adaptation technology rely on techniques based on Generative Adversarial Networks (GANs), such as the Cycle-Consistent Generative Adversarial Network (CycleGAN), which adapts images across different domains by learning nonlinear mapping functions between the domains. In this work, we exploit the CycleGAN approach for domain adaptation in a particular change detection application, namely, deforestation detection in the Amazon forest. Experimental results indicate that the proposed approach is capable of alleviating the effects associated with domain shift in the context of the target application. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

Pixel-based land cover classification of aerial images is a standard task in remote sensing, whose goal is to identify the physical material of the earth’s surface. Recently, most of the well-performing methods rely on encoder-decoder structure based convolutional neural networks (CNN). In the encoder part, many successive convolution and pooling operations are applied to obtain features at a lower spatial resolution, and in the decoder part these features are up-sampled gradually and layer by layer, in order to make predictions in the original spatial resolution. However, the loss of spatial resolution caused by pooling affects the final classification performance negatively, which is compensated by skip-connections between corresponding features in the encoder and the decoder. The most popular ways to combine features are element-wise addition of feature maps and 1x1 convolution. In this work, we investigate skip-connections. We argue that not every skip-connections are equally important. Therefore, we conducted experiments designed to find out which skip-connections are important. Moreover, we propose a new cosine similarity loss function to utilize the relationship of the features of the pixels belonging to the same category inside one mini-batch, i.e. these features should be close in feature space. Our experiments show that the new cosine similarity loss does help the classification. We evaluated our methods using the Vaihingen and Potsdam dataset of the ISPRS 2D semantic labelling challenge and achieved an overall accuracy of 91.1% for both test sites.

At 48 years of age CORONA-KH-4B images are important for actual urban planning in Bangladesh, where no old maps or aerial images exist, indicating locations of former water courses that are no longer visible, but causing problems for the stability of building ground. CORONA KH-4B images are available for very low cost or even free of charge. A forward and a backward-looking panoramic camera with a nadir angle of approximately 15° enabled a stereoscopic coverage. Taken from a height of 154km, an image covers approximately 220km × 14km up to 17km with a height to base relation of 1:1.85. The ground sampling distance (GSD) varies with the incidence angle from 1.77m in the centre of the image up to 2.18m in Y-direction and 2.69m in the X-direction at the ends of the images. These nice conditions of the old images are affected by geometric problems of the panoramic images. A geometric bending can be improved by a correction based on the available sidelines, nevertheless deformations dominantly in the longitudinal direction only can be determined based on ground control points. A group of neighboured images have similar deformations, allowing the determination of a correction grid, describing the systematic image errors. This improved the geometry strongly, but could not eliminate all individual and local geometric problems. The high morphologic quality of the CORONA images, but limited absolute height accuracy has been improved by merging with the highly accurate TDM90 height model. For ortho images a fitting of neighboured images was required. © 2020 Copernicus GmbH. All rights reserved.

The 3rd edition of the international summer school “Close-range Sensing Techniques in Alpine terrain” took place in Obergurgl, Austria, in June 2019. This article reports on results from the training and seminar activities and the outcome of student questionnaire survey. Comparison between the recent edition and the past edition in 2017 shows no significant differences on the level of satisfaction on organizational and training aspects. Gender balance was present both in candidates and in the outcome of selections. Selection was based on past research activities and on topic relevance. The majority of trainees were therefore doctoral candidates and postdoctoral researchers, but also motivated master students participated. The training took place through keynotes, lectures, seminars, in the field with hands-on surveys followed by data analysis in the lab, and teamwork for preparing a final team presentation over different assignments.

Fully convolutional neural networks (FCN) are successfully used for the automated pixel-wise classification of aerial images and possibly additional data. However, they require many labelled training samples to perform well. One approach addressing this issue is semi-supervised domain adaptation (SSDA). Here, labelled training samples from a source domain and unlabelled samples from a target domain are used jointly to obtain a target domain classifier, without requiring any labelled samples from the target domain. In this paper, a two-step approach for SSDA is proposed. The first step corresponds to a supervised training on the source domain, making use of strong data augmentation to increase the initial performance on the target domain. Secondly, the model is adapted by entropy minimization using a novel weighting strategy. The approach is evaluated on the basis of five domains, corresponding to five cities. Several training variants and adaptation scenarios are tested, indicating that proper data augmentation can already improve the initial target domain performance significantly resulting in an average overall accuracy of 77.5%. The weighted entropy minimization improves the overall accuracy on the target domains in 19 out of 20 scenarios on average by 1.8%. In all experiments a novel FCN architecture is used that yields results comparable to those of the best-performing models on the ISPRS labelling challenge while having an order of magnitude fewer parameters than commonly used FCNs.

Photogrammetric methods and sensors like LIDAR, RADAR and cameras are becoming more and more important for new applications like highly automatic driving, since they enable capturing relative information of the ego vehicle w.r.t its environment. Integrity measure the trust that we can put in the navigation information of a system. The concept of integrity was first developed for civil aviation and is linked to reliability concepts well known in geodesy and photogrammetry. Currently, the navigation community is discussing how to guarantee integrity for car navigation and multi-sensor systems. In this paper, we will give a short review on integrity concepts and on the current discussion of how to apply it to car navigation. We will discuss which role photogrammetry could play to solve the open issues in the integrity definition and monitoring for multi-sensor systems. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

This paper proposes several methods for training a Convolutional Neural Network (CNN) for learning the similarity between images of silk fabrics based on multiple semantic properties of the fabrics. In the context of the EU H2020 project SILKNOW (http://silknow.eu/), two variants of training were developed, one based on a Siamese CNN and one based on a triplet architecture. We propose different definitions of similarity and different loss functions for both training strategies, some of them also allowing the use of incomplete information about the training data. We assess the quality of the trained model by using the learned image features in a k-NN classification. We achieve overall accuracies of 93–95% and average F1-scores of 87–92%.

State-of-the-art object detection approaches such as Fast/Faster R-CNN, SSD, or YOLO have difficulties detecting dense, small targets with arbitrary orientation in large aerial images. The main reason is that using interpolation to align RoI features can result in a lack of accuracy or even loss of location information. We present the Local-aware Region Convolutional Neural Network (LR-CNN), a novel two-stage approach for vehicle detection in aerial imagery. We enhance translation invariance to detect dense vehicles and address the boundary quantization issue amongst dense vehicles by aggregating the high-precision RoIs' features. Moreover, we resample high-level semantic pooled features, making them regain location information from the features of a shallower convolutional block. This strengthens the local feature invariance for the resampled features and enables detecting vehicles in an arbitrary orientation. The local feature invariance enhances the learning ability of the focal loss function, and the focal loss further helps to focus on the hard examples. Taken together, our method better addresses the challenges of aerial imagery. We evaluate our approach on several challenging datasets (VEDAI, DOTA), demonstrating a significant improvement over state-of-the-art methods. We demonstrate the good generalization ability of our approach on the DLR 3K dataset., Comment: 8 pages

Pixel-wise classification of remote sensing imagery is highly interesting for tasks like land cover classification or change detection. The acquisition of large training data sets for these tasks is challenging, but necessary to obtain good results with deep learning algorithms such as convolutional neural networks (CNN). In this paper we present a method for the automatic generation of a large amount of training data by combining satellite imagery with reference data from an available geospatial database. Due to this combination of different data sources the resulting training data contain a certain amount of incorrect labels. We evaluate the influence of this so called label noise regarding the time difference between acquisition of the two data sources, the amount of training data and the class structure. We combine Sentinel-2 images with reference data from a geospatial database provided by the German Land Survey Office of Lower Saxony (LGLN). With different training sets we train a fully convolutional neural network (FCN) and classify four land cover classes (code Building, Agriculture, Forest, Water/code). Our results show that the errors in the training samples do not have a large influence on the resulting classifiers. This is probably due to the fact that the noise is randomly distributed and thus, neighbours of incorrect samples are predominantly correct. As expected, a larger amount of training data improves the results, especially for the less well represented classes. Other influences are different illuminations conditions and seasonal effects during data acquisition. To better adapt the classifier to these different conditions they should also be included in the training data. © 2020 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives.

Automatic feature matching is a crucial step in Structure-from-Motion (SfM) applications for 3D reconstruction purposes. From an historical perspective we can say now that SIFT was the enabling technology that made SfM a successful and fully automated pipeline. SIFT was the ancestor of a wealth of detector/descriptor methods that are now available. Various research activities have tried to benchmark detector/descriptors operators, but a clear outcome is difficult to be drawn. This paper presents an ISPRS Scientific Initiative aimed at providing the community with an educational open-source tool (called PhotoMatch) for tie point extractions and image matching. Several enhancement and decolorization methods can be initially applied to an image dataset in order to improve the successive feature extraction steps. Then different detector/descriptor combinations are possible, coupled with different matching strategies and quality control metrics. Examples and results show the implemented functionality of PhotoMatch which has also a tutorial for shortly explaining the implemented methods.

We propose a new approach for the automatic detection of network structures in raster data. The model for the network structure is represented by a graph whose nodes and edges correspond to junction-points and to connecting line segments, respectively; nodes and edges are further described by certain parameters. We embed this model in the probabilistic framework of marked point processes and determine the most probable configuration of objects by stochastic sampling. That is, different graph configurations are constructed randomly by modifying the graph entity parameters, by adding and removing nodes and edges to/ from the current graph configuration. Each configuration is then evaluated based on the probabilities of the changes and an energy function describing the conformity with a predefined model. By using the Reversible Jump Markov Chain Monte Carlo sampler, a global optimum of the energy function is determined. We apply our method to the detection of river and tidal channel networks in digital terrain models. In comparison to our previous work, we introduce constraints concerning the flow direction of water into the energy function. Our goal is to analyse the influence of different parameter settings on the results of network detection in both, synthetic and real data. Our results show the general potential of our method for the detection of river networks in different types of terrain.

This paper proposes a novel approach for linear feature detection. The contribution is twofold: a novel model for spatial point processes and a new method for linear feature detection. It describes a linear feature as a string of points, represents all features in an image as a configuration of a spatial point process, and formulates feature detection as finding the optimal configuration of a spatial point process. Further, a prior term is proposed to favor straight linear configurations, and a data term is constructed to superpose the points on linear features. The proposed approach extracts straight linear features in a global framework. The paper reports ongoing work. As demonstrated in preliminary experiments, globally optimal linear features can be detected. National Natural Science Foundation of China/41071263 National Natural Science Foundation of China/41571335 Zhejiang Provincial Natural Science Foundation of China/LY13D010003 Key Laboratory for National Geographic Census and Monitoring National Administration of Surveying, Mapping and Geoinformation/2014NGCM

We propose a novel hierarchical approach for the classification of airborne 3D lidar points. Spatial and semantic context is incorporated via a two-layer Conditional Random Field (CRF). The first layer operates on a point level and utilises higher order cliques. Segments are generated from the labelling obtained in this way. They are the entities of the second layer, which incorporates larger scale context. The classification result of the segments is introduced as an energy term for the next iteration of the point-based layer. This framework iterates and mutually propagates context to improve the classification results. Potentially wrong decisions can be revised at later stages. The output is a labelled point cloud as well as segments roughly corresponding to object instances. Moreover, we present two new contextual features for the segment classification: the distance and the orientation of a segment with respect to the closest road. It is shown that the classification benefits from these features. In our experiments the hierarchical framework improve the overall accuracies by 2.3% on a point-based level and by 3.0% on a segment-based level, respectively, compared to a purely point-based classification.

Online multi-person tracking in image sequences is commonly guided by recursive filters, whose predictive models define the expected positions of future states. When a predictive model deviates too much from the true motion of a pedestrian, which is often the case in crowded scenes due to unpredicted accelerations, the data association is prone to fail. In this paper we propose a novel predictive model on the basis of Gaussian Process Regression. The model takes into account the motion of every tracked pedestrian in the scene and the prediction is executed with respect to the velocities of all interrelated persons. As shown by the experiments, the model is capable of yielding more plausible predictions even in the presence of mutual occlusions or missing measurements. The approach is evaluated on a publicly available benchmark and outperforms other state-of-the-art trackers.

In this paper we describe learning of a descriptor based on the Siamese Convolutional Neural Network (CNN) architecture and evaluate our results on a standard patch comparison dataset. The descriptor learning architecture is composed of an input module, a Siamese CNN descriptor module and a cost computation module that is based on the L2 Norm. The cost function we use pulls the descriptors of matching patches close to each other in feature space while pushing the descriptors for non-matching pairs away from each other. Compared to related work, we optimize the training parameters by combining a moving average strategy for gradients and Nesterov's Accelerated Gradient. Experiments show that our learned descriptor reaches a good performance and achieves state-of-art results in terms of the false positive rate at a 95 % recall rate on standard benchmark datasets.

In this paper we present an approach for a weighted rotation averaging to estimate absolute rotations from relative rotations between two images for a set of multiple overlapping images. The solution does not depend on initial values for the unknown parameters and is robust against outliers. Our approach is one part of a solution for a global image orientation. Often relative rotations are not free from outliers, thus we use the redundancy in available pairwise relative rotations and present a novel graph-based algorithm to detect and eliminate inconsistent rotations. The remaining relative rotations are input to a weighted least squares adjustment performed in the Lie algebra of the rotation manifold SO(3) to obtain absolute orientation parameters for each image. Weights are determined using the prior information we derived from the estimation of the relative rotations. Because we use the Lie algebra of SO(3) for averaging no subsequent adaptation of the results has to be performed but the lossless projection to the manifold. We evaluate our approach on synthetic and real data. Our approach often is able to detect and eliminate all outliers from the relative rotations even if very high outlier rates are present. We show that we improve the quality of the estimated absolute rotations by introducing individual weights for the relative rotations based on various indicators. In comparison with the state-of-the-art in recent publications to global image orientation we achieve best results in the examined datasets.

Tracking-by-detection is a widely used practice in recent tracking systems. These usually rely on independent single frame detections that are handled as observations in a recursive estimation framework. If these observations are imprecise the generated trajectory is prone to be updated towards a wrong position. In contrary to existing methods our novel approach uses a Dynamic Bayes Network in which the state vector of a recursive Bayes filter, as well as the location of the tracked object in the image are modelled as unknowns. These unknowns are estimated in a probabilistic framework taking into account a dynamic model, and a state-of-the-art pedestrian detector and classifier. The classifier is based on the Random Forest-algorithm and is capable of being trained incrementally so that new training samples can be incorporated at runtime. This allows the classifier to adapt to the changing appearance of a target and to unlearn outdated features. The approach is evaluated on a publicly available benchmark. The results confirm that our approach is well suited for tracking pedestrians over long distances while at the same time achieving comparatively good geometric accuracy.

In order to tackle the problem of consistently integrating 2D vector data and a DTM, we presented an approach for the adaptation of 2D GIS road objects to airborne laser scanning (ALS) data using active contours (snakes) in (Göpfert et al., 2011). In this paper the algorithm is modified for the integration of stereo images as an alternative data source for area-wide height information. For that reason, a new image energy is developed that exploits geometric and radiometric features derived from the image data. Afterwards, we compare the applicability of our method with respect to the ALS data and stereo images as input. In addition, a new approach is suggested that analyses the different energy terms of active contours after the optimisation process in order to automatically detect contour parts that did not reach a suitable position in the sensor data. This concept of an internal evaluation is able to guide the user during post processing. Experiments show that the snake approach with an image energy based on stereo images is generally able to adapt GIS road centrelines to the sensor data and thus to improve the quality of the 2D vector data. However, the comparison to the results for ALS data demonstrates that the algorithm perform slightly worse for image data in the high precision level.

In this paper, we propose a new framework for segmenting feature-based moving objects under affine subspace model. Since the feature trajectories in practice are high-dimensional and contain a lot of noise, we firstly apply the sparse PCA to represent the original trajectories with a low-dimensional global subspace, which consists of the orthogonal sparse principal vectors. Subsequently, the local subspace separation will be achieved via automatically searching the sparse representation of the nearest neighbors for each projected data. In order to refine the local subspace estimation result, we propose an error estimation to encourage the projected data that span a same local subspace to be clustered together. In the end, the segmentation of different motions is achieved through the spectral clustering on an affinity matrix, which is constructed with both the error estimation and sparse neighbors optimization. We test our method extensively and compare it with state-of-the-art methods on the Hopkins 155 dataset. The results show that our method is comparable with the other motion segmentation methods, and in many cases exceed them in terms of precision and computation time.

The stereo matching method semi-global matching (SGM) relies on consistency constraints during the cost aggregation which are enforced by so-called penalty terms. This paper proposes new and evaluates four penalty functions for SGM. Due to mutual dependencies, two types of matching cost calculation, census and rank transform, are considered. Performance is measured using original and degenerated images exhibiting radiometric changes and noise from the Middlebury benchmark. The two best performing penalty functions are inversely proportional and negatively linear to the intensity gradient and perform equally with 6.05% and 5.91% average error, respectively. The experiments also show that adaptive penalty terms are mandatory when dealing with difficult imaging conditions. Consequently, for highest algorithmic performance in real-world systems, selection of a suitable penalty function and thorough parametrization with respect to the expected image quality is essential.

driving. While much progress has been made in recent years, imaging conditions in natural outdoor environments are still very challenging for current reconstruction and recognition methods. In this paper, we propose a novel unified approach which reasons jointly about 3D scene flow as well as the pose, shape and motion of vehicles in the scene. Towards this goal, we incorporate a deformable CAD model into a slanted-plane conditional random field for scene flow estimation and enforce shape consistency between the rendered 3D models and the parameters of all superpixels in the image. The association of superpixels to objects is established by an index variable which implicitly enables model selection. We evaluate our approach on the challenging KITTI scene flow dataset in terms of object and scene flow estimation. Our results provide a prove of concept and demonstrate the usefulness of our method.

Complex activity modeling and identification of anomaly is one of the most interesting and desired capabilities for automated video behavior analysis. A number of different approaches have been proposed in the past to tackle this problem. There are two main challenges for activity modeling and anomaly detection: 1) most existing approaches require sufficient data and supervision for learning; 2) the most interesting abnormal activities arise rarely and are ambiguous among typical activities, i.e. hard to be precisely defined. In this paper, we propose a novel approach to model complex activities and detect anomalies by using non-parametric Gaussian Process (GP) models in a crowded and complicated traffic scene. In comparison with parametric models such as HMM, GP models are nonparametric and have their advantages. Our GP models exploit implicit spatial-temporal dependence among local activity patterns. The learned GP regression models give a probabilistic prediction of regional activities at next time interval based on observations at present. An anomaly will be detected by comparing the actual observations with the prediction at real time. We verify the effectiveness and robustness of the proposed model on the QMUL Junction Dataset. Furthermore, we provide a publicly available manually labeled ground truth of this data set.

Land cover and land use exhibit strong contextual dependencies. We propose a novel approach for the simultaneous classification of land cover and land use, where semantic and spatial context is considered. The image sites for land cover and land use classification form a hierarchy consisting of two layers: a land cover layer and a land use layer. We apply Conditional Random Fields (CRF) at both layers. The layers differ with respect to the image entities corresponding to the nodes, the employed features and the classes to be distinguished. In the land cover layer, the nodes represent super-pixels; in the land use layer, the nodes correspond to objects from a geospatial database. Both CRFs model spatial dependencies between neighbouring image sites. The complex semantic relations between land cover and land use are integrated in the classification process by using contextual features. We propose a new iterative inference procedure for the simultaneous classification of land cover and land use, in which the two classification tasks mutually influence each other. This helps to improve the classification accuracy for certain classes. The main idea of this approach is that semantic context helps to refine the class predictions, which, in turn, leads to more expressive context information. Thus, potentially wrong decisions can be reversed at later stages. The approach is designed for input data based on aerial images. Experiments are carried out on a test site to evaluate the performance of the proposed method. We show the effectiveness of the iterative inference procedure and demonstrate that a smaller size of the super-pixels has a positive influence on the classification result.

In this paper we propose a new method for the automatic extraction of tidal channels in digital terrain models (DTM) using a sampling approach based on marked point processes. In our model, the tidal channel system is represented by an undirected, acyclic graph. The graph is iteratively generated and fitted to the data using stochastic optimization based on a Reversible Jump Markov Chain Monte Carlo (RJMCMC) sampler and simulated annealing. The nodes of the graph represent junction points of the channel system and the edges straight line segments with a certain width in between. In each sampling step, the current configuration of nodes and edges is modified. The changes are accepted or rejected depending on the probability density function for the configuration which evaluates the conformity of the current status with a pre-defined model for tidal channels. In this model we favour high DTM gradient magnitudes at the edge borders and penalize a graph configuration consisting of non-connected components, overlapping segments and edges with atypical intersection angles. We present the method of our graph based model and show results for lidar data, which serve of a proof of concept of our approach. Ministry of Environment, Energy, and Climate Protection Ministry of Science and Culture of Lower Saxony

In this paper we present a robust orientation approach for an imaging sensor flown on a micro-UAV based on image triplets. Our aim is to have the orientation available online, i.e. during image acquisition. The resulting point cloud and sensor orientations can then for instance be evaluated for navigation purposes of the UAV or to analyse the completeness of the point cloud. We use low quality imagery extracted from the downlink of an onboard PAL-camera. Trilinear constraints and cross-checked matches allow for a high robustness of the sensor orientation and the sparse 3D point cloud. In order to reach the goal of on-line processing given the large number of observations and unknowns, we make use of an incremental bundle adjustment. Estimated parameters are incrementally improved without explicitly considering previous observations. Our approach combines linear projective geometry for obtaining initial values using the trifocal tensor with non-linear perspective geometry for the estimation of the unknowns. This combination allows for a high precision of the estimation, while eliminating the need for initial values. We evaluate the performance of our approach by means of imagery we acquired of the facade of theWelfenschloss in Hannover, collected with a Microdrones md4-200 micro-UAV. The results are the orientation parameters of the images and a sparse 3D point cloud representing the object. They are compared to those from a commercial bundle adjustment software and analysed in terms of geometric precision.

Domain adaptation techniques in transfer learning try to reduce the amount of training data required for classification by adapting a classifier trained on samples from a source domain to a new data set (target domain) where the features may have different distributions. In this paper, we propose a new technique for domain adaptation based on logistic regression. Starting with a classifier trained on training data from the source domain, we iteratively include target domain samples for which class labels have been obtained from the current state of the classifier, while at the same time removing source domain samples. In each iteration the classifier is re-trained, so that the decision boundaries are slowly transferred to the distribution of the target features. To make the transfer procedure more robust we introduce weights as a function of distance from the decision boundary and a new way of regularisation. Our methodology is evaluated using a benchmark data set consisting of aerial images and digital surface models. The experimental results show that in the majority of cases our domain adaptation approach can lead to an improvement of the classification accuracy without additional training data, but also indicate remaining problems if the difference in the feature distributions becomes too large.

Figure-ground image segmentation has been a challenging problem in computer vision. Apart from the difficulties in establishing an effective framework to divide the image pixels into meaningful groups, the notions of figure and ground often need to be properly defined by providing either user inputs or object models. In this paper, we propose a novel graph-based segmentation framework, called superpixel cut. The key idea is to formulate foreground segmentation as finding a subset of superpixels that partitions a graph over superpixels. The problem is formulated as Min-Cut. Therefore, we propose a novel cost function that simultaneously minimizes the inter-class similarity while maximizing the intra-class similarity. This cost function is optimized using parametric programming. After a small learning step, our approach is fully automatic and fully bottom-up, which requires no high-level knowledge such as shape priors and scene content. It recovers coherent components of images, providing a set of multiscale hypotheses for high-level reasoning. We evaluate our proposed framework by comparing it to other generic figure-ground segmentation approaches. Our method achieves improved performance on state-of-the-art benchmark databases.

The classification of building facade images is a challenging problem that receives a great deal of attention in the photogrammetry community. Image classification is critically dependent on the features. In this paper, we perform an empirical feature evaluation task for building facade images. Feature sets we choose are basic features, color features, histogram features, Peucker features, texture features, and SIFT features. We present an approach for region-wise labeling using an efficient randomized decision forest classifier and local features. We conduct our experiments with building facade image classification on the eTRIMS dataset, where our focus is the object classes building, car, door, pavement, road, sky, vegetation, and window.

Matching and fusion of 3D point clouds, such as close range laser scans, is important for creating an integrated 3D model data infrastructure. The Iterative Closest Point algorithm for alignment of point clouds is one of the most commonly used algorithms for matching of rigid bodies. Evidently, scans are acquired from different positions and might present different data characterization and accuracies, forcing complex data-handling issues. The growing demand for near real-time applications also introduces new computational requirements and constraints into such processes. This research proposes a methodology to solving the computational and processing complexities in the ICP algorithm by introducing specific performance enhancements to enable more efficient analysis and processing. An Octree data structure together with the caching of localized Delaunay triangulation-based surface meshes is implemented to increase computation efficiency and handling of data. Parallelization of the ICP process is carried out by using the Single Instruction, Multiple Data processing scheme – based on the Divide and Conquer multi-branched paradigm – enabling multiple processing elements to be performed on the same operation on multiple data independently and simultaneously. When compared to the traditional non-parallel list processing the Octree-based SIMD strategy showed a sharp increase in computation performance and efficiency, together with a reliable and accurate alignment of large 3D point clouds, contributing to a qualitative and efficient application.

The fully automated analysis of 3D point clouds is of great importance in photogrammetry, remote sensing and computer vision. For reliably extracting objects such as buildings, road inventory or vegetation, many approaches rely on the results of a point cloud classification, where each 3D point is assigned a respective semantic class label. Such an assignment, in turn, typically involves statistical methods for feature extraction and machine learning. Whereas the different components in the processing workflow have extensively, but separately been investigated in recent years, the respective connection by sharing the results of crucial tasks across all components has not yet been addressed. This connection not only encapsulates the interrelated issues of neighborhood selection and feature extraction, but also the issue of how to involve spatial context in the classification step. In this paper, we present a novel and generic approach for 3D scene analysis which relies on (i) individually optimized 3D neighborhoods for (ii) the extraction of distinctive geometric features and (iii) the contextual classification of point cloud data. For a labeled benchmark dataset, we demonstrate the beneficial impact of involving contextual information in the classification process and that using individual 3D neighborhoods of optimal size significantly increases the quality of the results for both pointwise and contextual classification.

In this paper we address the problem of classification of remote sensing images in the framework of transfer learning with a focus on domain adaptation. The main novel contribution is a method for transductive transfer learning in remote sensing on the basis of logistic regression. Logistic regression is a discriminative probabilistic classifier of low computational complexity, which can deal with multiclass problems. This research area deals with methods that solve problems in which labelled training data sets are assumed to be available only for a source domain, while classification is needed in the target domain with different, yet related characteristics. Classification takes place with a model of weight coefficients for hyperplanes which separate features in the transformed feature space. In term of logistic regression, our domain adaptation method adjusts the model parameters by iterative labelling of the target test data set. These labelled data features are iteratively added to the current training set which, at the beginning, only contains source features and, simultaneously, a number of source features are deleted from the current training set. Experimental results based on a test series with synthetic and real data constitutes a first proof-of-concept of the proposed method.

We present an interactive system to manipulate a virtual object by tracking multiple hands in 3D space using a Kinect device. The system segments hand shapes from a captured 3D scene by using depth information and active contours. A hand shape is recognized by a trained naive Bayes classifier, whether it belongs to a palm, a pointing hand form or both hands with simple occlusion. A plane is approximated by using RANSAC for a palm hand form, while a vector from the hand centroid to the fingertip is obtained for a pointing hand form by using ICP as tracking method. A shape of simple occluded two hands is split into a palm and an incomplete pointing hand form, whose missing data is estimated by using PCA. The system works in semi-real time.