Search

ALS data are essential data sources used to generate digital terrain models (DTM), 3D city models, landscape models and high precision maps. Semantic segmentation aiming to assign every point with a semantic label of ALS point clouds is of importance when generating those 3D products that have multiple categories and ask for detailed object geometry. Motivated by the top performance of deep learning algorithms on scene understanding tasks, this Ph.D. thesis investigates the semantic segmentation of ALS point clouds based on deep learning algorithms. We first explore how to learn representative features from ALS point clouds (Chapter 2). Then we focus on how to reduce the manual labelling efforts to train a deep learning model for semantic segmentation. We investigate active learning (Chapter 3) to select and annotate informative points, and weak supervision (Chapter 4) to annotate only weak labels for the pointwise prediction task.

3D change detection draws more and more attention in recent years due to the increasing availability of 3D data. It can be used in the fields of land use / land cover (LULC) change detection, 3D geographic information updating, terrain deformation analysis, urban construction monitoring et al. Our motivation to study 3D change detection is mainly related to the practical need to update the outdated point clouds captured by Airborne Laser Scanning (ALS) with new point clouds obtained by dense image matching (DIM). The thesis has three main parts. The first part, chapter 1, explains the motivation, providing a review of current ALS and airborne photogrammetry techniques. It also presents the research objectives and questions. The second part including chapter 2 and chapter 3 evaluates the quality of photogrammetric products and investigates their potential for change detection. The third part including chapter 4 and chapter 5 proposes two methods for change detection that meet different requirements. To investigate the potential of using point clouds derived by dense matching for change detection, we propose a framework for evaluating the quality of 3D point clouds and DSMs generated by dense image matching. Our evaluation framework based on a large number of square patches reveals the distribution of dense matching errors in the whole photogrammetric block. Robust quality measures are proposed to indicate the DIM accuracy and precision quantitatively. The overall mean offset to the reference is 0.1 Ground Sample Distance (GSD); the maximum mean deviation reaches 1.0 GSD. We also find that the distribution of dense matching errors is homogenous in the whole block and close to a normal distribution based on many patch-based samples. However, in some locations, especially along narrow alleys, the mean deviations may get worse. In addition, the profiles of ALS points and DIM points reveal that the DIM profile fluctuates around the ALS profile. We find that the accuracy of DIM point cloud improves and that the noise level decreases on smooth ground areas when oblique images are used in dense matching together with nadir images. Then we evaluate whether the standard LiDAR filters are effective to filter dense matching points in order to derive accurate DTMs. Filtering results on a city block show that LiDAR filters perform well on the grassland, along bushes and around individual trees if the point cloud is sufficiently precise. When a ranking filter is used on the point clouds before filtering, the filtering will identify fewer but more reliable ground points. However, some small objects on the terrain will be filtered out. Since we aim at obtaining accurate DTMs, the ranking filter shows its value in identifying reliable ground points. Based on the previous findings in DIM quality, we propose a method to detect building changes between ALS and photogrammetric data. Firstly, the ALS points and DIM points are split out and concatenated with the orthoimages. The multimodal data are normalized to feed into a pseudo-Siamese Neural network for change detection. Then, the changed objects are delineated through per-pixel classification and artefact removal. The change detection module based on a pseudo-Siamese CNN can quickly localize the changes and generate coarse change maps. The next module can be used in precise mapping of change boundaries. Experimental results show that the proposed pseudo-Siamese Neural network can cope with the DIM errors and output plausible change detection results. Although the point cloud quality from dense matching is not as fine as laser scanning points, the spectral and textural information provided by the orthoimages serve as a supplement. Considering that the tasks of semantic segmentation and change detection are correlated, we propose SiamPointNet++ model to combine the two tasks in one framework. The method outputs a pointwise joint label for each ALS point. If an ALS point is unchanged, it is assigned a semantic label; If an ALS point is changed, it is assigned a change label. The sematic and change information are included in the joint labels with minimum information redundancy. The combined Siamese network learns both intra-epoch and inter-epoch features. Intra-epoch features are extracted at multiple scales to embed the local and global information. Inter-epoch features are extracted by Conjugated Ball Sampling (CBS) and concatenated to make change inference. Experiments on the Rotterdam data set indicate that the network is effective in learning multi-task features. It is invariant to the permutation and noise of inputs and robust to the data difference between ALS and DIM data. Compared with a sophisticated object-based method and supervised change detection, this method requires much less hyper-parameters and human intervention but achieves superior performance. As a conclusion, the thesis evaluates the quality of dense matching points and investigates its potential of updating outdated ALS points. The two change detection methods developed for different applications show their potential in the automation of topographic change detection and point cloud updating. Future work may focus on improving the generalizability and interpretability of the proposed models.

Indoor mobile mapping is important for a wide range of applications such as indoor navigation and positioning, mapping hazardous sites, facility management, virtual tourism and interior design. State-of-the-art indoor mobile mapping systems use a combination of light detection and ranging (LIDAR) scanners, cameras and/or inertial measurement units (IMUs) mounted on movable platforms and allow for capturing 3D data of buildings’ interiors. As global navigation satellite system (GNSS) positioning does not work inside buildings, the extensively investigated simultaneous localisation and mapping (SLAM) algorithms seem to offer a suitable solution for the problem. In this dissertation, a SLAM-based backpack mobile mapping system (ITC-Backpack) was developed for mapping buildings’ interiors. The configuration of the ITC-Backpack consists of three 2D LIDAR scanners and an IMU. The employed SLAM is planar feature-based SLAM algorithm that exploits the LIDAR scanners and the IMU to estimate the 3D pose and plane parameters. Representing the SLAM map by planes is advantageous for multiple reasons. First, the planar features are typically large and spatially distinct and therefore distinguishable from one another. Second, they are abundant in indoor man-made environments. Third, storing planar features takes less data space than storing the captured point clouds. Finally, the representation by planar shapes is a popular format for the state-of-the-art indoor 3D reconstruction methods. The developed SLAM in this dissertation performs loop closure detection and correction using these planar features. This enables the backpack system to recognize an already visited place and correct for the accumulated drift. The outputs of ITC-Backpack are reconstructed 3D planes, 3D point clouds as well as a trajectory of the system’s motion in a local coordinate system. A combination of the point cloud and the trajectory represents an advantageous supplementary information for some indoor modelling problems such as semantic interpretation and space partitioning. The ITC-Backpack system is validated on various indoor environments that differ in terms of geometry, architecture and clutter. Moreover, we evaluate the performance of the system by comparing the obtained point clouds against those obtained from a commercial indoor mobile mapping system, Viametris1 iMS3D, and ground truth obtained from a terrestrial laser scanner (TLS).

Scene understanding is an important and fundamental research field in computer vision, which is quite useful for many applications in photogrammetry and remote sensing. It focuses on locating and classifying objects in images, understanding the relationships between them. The higher goal is to interpret what event happens in the scene, when it happens and why it happens, and what should we do based on the information. Dynamic scene understanding is to use information from different time to interpret scenes and answer the above related questions. For modern scene understanding technology, deep learning has shown great potential for such task. "Deep" in deep learning refers to the use of multiple layers in the neural networks. Deep neural networks are powerful as they are highly non-linear function that possess the ability to map from one domain to another quite different domain after proper training. It is the best solution for many fundamental research tasks regarding scene understanding. This ph.D. research also takes advantage of deep learning for dynamic scene understanding. Temporal information plays an important role for dynamic scene understanding. Compared with static scene understanding from images, information distilled from the time dimension provides values in many different ways. Images across consecutive frames have very high correlation, i.e., objects observed in one frame have very high chance to be observed and identified in nearby frames as well. Such redundancy in observation could potentially reduce the uncertainty for object recognition with deep learning based methods, resulting in more consistent inference. High correlation across frames could also improve the chance for recognizing objects correctly. If the camera or the object moves, the object could be observed in multiple different views with different poses and appearance. The information captured for object recognition would be more diverse and complementary, which could be aggregated to jointly inference the categories and the properties of objects. This ph.D. research involves several tasks related to the dynamic scene understanding in computer vision, including semantic segmentation for aerial platform images (chapter 2, 3), video object segmentation and video object detection for common objects in natural scenes (chapter 4, 5), and multi-object tracking and segmentation for cars and pedestrians in driving scenes(chapter 6). Chapter2 investigates how to establish the semantic segmentation benchmark for the UAV images, which includes data collection, data labeling, dataset construction, and performance evaluation with baseline deep neural networks and the proposed multi-scale dilation net. Conditional random field with feature space optimization is used to achieve consistent semantic segmentation prediction in videos. Chapter3 investigates how to better extract the scene context information for etter object recognition performance by proposing the novel bidirectional multiscale attention networks. It achieves better performance by inferring features and attention weights for feature fusing from both higher level and lower level branches. Chapter4 investigates how to simultaneously segment multiple objects across multiple frames by combining memory modules with instance segmentation networks. Our method learns to propagate the target object labels without auxiliary data, such as optical flow, which simplifies the model. Chapter5 investigates how to improve the performance of well-trained object detectors with a light weighted and efficient plug&play tracker for object detection in video. This chapter also investigates how the proposed model performs when lacking video training data. Chapter6 investigates how to improve the performance of detection, segmentation, and tracking by jointly considering top-down and bottom-up inference. The whole pipeline follows the multi-task design, i.e., a single feature extraction backbone with multiple heads for different sub-tasks. Overall, this manuscript has delved into several different computer vision tasks, which share fundamental research problems, including detection, segmentation, and tracking. Based on the research experiments and knowledge from literature review, several reflections regarding dynamic scene understanding have been discussed: The range of object context influence the quality for object recognition; The quality of video data affect the method choice for specific computer vision task; Detection and tracking are complementary for each other. For future work, unified dynamic scene understanding task could be a trend, and transformer plus self-supervised learning is one promising research direction. Real-time processing for dynamic scene understanding requires further researches in order to put the methods into usage for real-world applications.

For several years, digital 3D city models have taken a central role in a number of different tasks. These models are used in urban and regional planning, surveying, and navigation and telecommunication technology. They also enable in the environmental field precise analyses and simulations of pollutant, flood, and noise propagation. For many applications, realistic building models are an essential component of a 3D city model. Their manual reconstruction provides good results but is associated with a very high time expenditure. In order to meet the high demand for building models and the increased demands on their level of detail, research has been recently conducted to develop various (semi-)automatic reconstruction methods for the time-efficient and cost-effective generation of 3D building models. The goal of designing an automatic building reconstruction method, which fully meets the high demands of the present day, has, however, not yet been achieved. The reasons for this are manifold; a major reason is, for example, the large number of different and complex roof shapes. In this thesis, a new approach to the fully automatic reconstruction of semantic 3D building models based on airborne LiDAR data is presented. The approach is characterized by a strong integration of building knowledge, which is automatically derived during the reconstruction. The derivation of building knowledge is carried out by employing a grammar whose production rules are applied in several bottom-up and top-down phases. In the selection process of the applicable production rules, methods of reinforcement learning from the field of machine learning are utilized. Thereby, it is taken into account that in complex roof structures and neighboring buildings similar roof elements or roof structures often occur multiple times. Through the application of the grammar, knowledge about the buildings is derived. This knowledge is modeled for each building in a separate multi-scale knowledge graph. The use of the grammar, together with the multi-scale knowledge graphs, in alternating bottom-up and top-down phases provides a reliable and robust derivation of further building knowledge. The grammar-guided method for deriving building knowledge can generally be integrated also into already existing data-driven reconstruction methods to improve their performance. This procedure is demonstrated exemplarily for a data-driven reconstruction based on binary space partitioning. In order to reduce the search space for the automatic derivation of building knowledge, a method for the automatic segmentation of roof surfaces is presented. It uses the building points of the underlying data set and employs the principles of surface growing. However, not only the measured height values of the point cloud are incorporated in this procedure, but additionally also virtual points, which are automatically generated during segmentation. As a result, segments can also grow below other roof surfaces, whereby the derivation of building knowledge and the construction of 3D building models are considerably simplified. For the construction of 3D building models, a method based on half-space modeling is presented. It utilizes the building knowledge in the multi-scale knowledge graphs to define and combine half-spaces, whose hyperplanes result from the segments of the point cloud. In this way, buildings of arbitrary complexity are described in canonical form. The resulting building models have the property that they are always both geometrically and topologically correct. The building geometry of the reconstructed 3D building models can be enriched by the semantic information of the multi-scale knowledge graphs. Thereby, also the need for semantic 3D building models is satisfied. In general, the requirements for reconstructed building models depend on the individual application. To take this into account, two methods are presented which can optionally be executed during the reconstruction. The first method is concerned with the reconstruction of small roof elements (e.g., dormers or chimneys) in order to increase the level of detail of insufficiently detailed building models. It does not reconstruct multiple occurrences of roof elements of similar shape individually from one another, but jointly by means of an ICP (iterative closest point) based method. As a result, also those roof elements are recognized and reconstructed which taken individually would not have been reconstructed. The second method recognizes the frequently occurring regularities in buildings and renders them more strongly in the reconstructed models. For this, divisive clustering methods are applied, both in a local and a global context, to the hyperplanes of the half-spaces that are used to define the half-space models in canonical form. The resulting well-shaped building models are particularly suitable for visualization purposes. Finally, the presented reconstruction method together with the resulting building models is evaluated using a benchmark data set and compared with other current state-of-the-art building reconstruction approaches. The results show that the presented method is robust and transferable to different building shapes and regions, and that the resulting 3D building models have a high quality with regard to their completeness and correctness. Seit mehreren Jahren nehmen digitale 3D-Stadtmodelle in verschiedenen Aufgabenstellungen eine zentrale Rolle ein. Sie werden unter anderem in der Stadt- und Raumplanung, im Vermessungswesen und in der Navigations- und Telekommunikationstechnik eingesetzt. Zudem ermöglichen sie im Umweltbereich präzise Analysen und Simulationen zur Ausbreitung von Hochwasser, Lärm und Schadstoffen. Ein für viele Anwendungszwecke wesentlicher Bestandteil von 3D-Stadtmodellen sind realitätsgetreue Gebäudemodelle. Deren manuelle Rekonstruktion bietet gute Ergebnisse, ist aber mit einem sehr hohen Zeitaufwand verbunden. Um der hohen Nachfrage nach Gebäudemodellen und den gestiegenen Anforderungen an deren Detailierungsgrad nachzukommen, hat man sich in den letzten Jahren mit ihrer zeiteffizienten und kostengünstigen Erzeugung beschäftigt und einige (semi-) automatische Rekonstruktionsmethoden entwickelt. Das Ziel ein Verfahren zur automatischen Gebäuderekonstruktion zu entwerfen, das den hohen Ansprüchen der heutigen Zeit gerecht wird, wurde bislang jedoch noch nicht vollständig erreicht. Die Ursachen hierfür sind vielfältig; z.B. ist ein wesentlicher Grund die große Anzahl unterschiedlicher und komplexer Dachformen. In der vorliegenden Arbeit wird ein neuer Ansatz zur vollautomatischen Rekonstruktion von semantischen 3D-Gebäudemodellen auf der Basis von luftgestützten LiDAR-Daten vorgestellt. Der Ansatz zeichnet sich durch eine starke Integration von Gebäudewissen aus, welches während der Rekonstruktion automatisch abgeleitet wird. Die Ableitung von Gebäudewissen erfolgt über eine Grammatik, deren Produktionsregeln in mehreren Bottom-Up- und Top-Down-Phasen angewendet werden. Bei der Selektierung der anzuwendenden Produktionsregeln werden Methoden des bestärkenden Lernens aus dem Bereich des maschinellen Lernens eingesetzt. Dadurch wird berücksichtigt, dass in komplexen Dachstrukturen und benachbarten Gebäuden oftmals ähnliche Dachelemente oder Dachstrukturen mehrfach vorkommen. Das durch die Anwendung der Grammatik abgeleitete Gebäudewissen wird für jedes Gebäude in jeweils einem multiskaligen Wissensgraphen modelliert. Der Einsatz der Grammatik zusammen mit den Wissensgraphen in sich abwechselnden Bottom-Up- und Top-Down-Phasen ermöglicht eine zuverlässige und robuste Ableitung von weiterem Gebäudewissen. Das durch eine Grammatik geleitete Verfahren zur Ableitung von Gebäudewissen lässt sich generell auch in bereits existierende datengetriebene Rekonstruktionsverfahren zu deren Vorteil integrieren. Demonstriert wird dies exemplarisch anhand einer datengetriebenen Rekonstruktion, die auf einer binären Raumpartitionierung (engl. binary space partitioning) basiert. Um den Suchraum für die automatische Ableitung von Gebäudewissen zu verringern, wird eine Methode zur automatischen Segmentierung von Dachflächen vorgestellt. Sie verwendet die Gebäudepunkte des zugrundeliegenden Datensatzes und die Prinzipien des Oberflächenwachstums (engl. surface growing). Es werden jedoch nicht nur die gemessenen Höhenwerte der Punktwolke in diesem Prozess berücksichtigt, sondern zusätzlich auch virtuelle Punkte, die während der Segmentierung automatisch erzeugt werden. Dies ermöglicht das Wachstum von Segmenten auch unterhalb anderer Dachflächen, wodurch die Ableitung von Gebäudewissen und die Konstruktion der 3D-Gebäudemodelle wesentlich vereinfacht werden. Für die Konstruktion der 3D-Gebäudemodelle wird ein Verfahren basierend auf der Halbraummodellierung vorgestellt. Es nutzt das Gebäudewissen in den multiskaligen Wissensgraphen für die Definition und Kombination von Halbräumen, deren Hyperebenen sich aus den Segmenten der Punktwolke ergeben. Auf diese Weise werden Gebäude beliebiger Komplexität in kanonischer Form beschrieben. Die daraus abgeleiteten Gebäudemodelle haben die Eigenschaft, dass sie sowohl geometrisch als auch topologisch immer korrekt sind. Die Gebäudegeometrie der rekonstruierten 3D-Gebäudemodelle lässt sich durch die semantischen Informationen der multiskaligen Wissensgraphen anreichern. Dadurch wird auch der Bedarf an semantischen 3D-Gebäudemodellen befriedigt. Generell hängen die Anforderungen an rekonstruierten Gebäudemodellen von der individuellen Anwendung ab. Um dies zu berücksichtigen werden zwei Verfahren vorgestellt, die optional während der Rekonstruktion ausgeführt werden können. Das erste Verfahren beschäftigt sich mit der Rekonstruktion von kleinen Dachelementen (z.B. Gauben oder Schornsteinen), um den Detailierungsgrad von nicht ausreichend detaillierten Gebäudemodellen zu erhöhen. Es rekonstruiert mehrfach und in gleicher Form vorkommende Dachelemente nicht unabhängig voneinander, sondern gemeinsam mittels einer ICP (engl. iterative closest point) basierten Methode. Dadurch werden auch solche Dachelemente erkannt und rekonstruiert, die einzeln für sich genommen nicht rekonstruiert worden wären. Das zweite Verfahren erkennt die in Gebäuden häufig auftretenden Regularitäten und gibt sie in den rekonstruierten Modellen verstärkt wieder. Dazu werden divisive Clusterverfahren sowohl im lokalen als auch im globalen Kontext auf die Hyperebenen der Halbräume der in der kanonischen Form definierten Halbraummodelle angewandt. Die daraus resultierenden formschönen Gebäudemodelle eignen sich insbesondere für Visualisierungszwecke. Schließlich wird das vorgestellte Rekonstruktionsverfahren mit den daraus resultierenden Gebäudemodellen anhand eines Benchmark-Datensatzes evaluiert und mit anderen aktuellen Gebäuderekonstruktionsverfahren verglichen. Die Ergebnisse zeigen, dass das vorgestellte Verfahren sowohl robust als auch auf unterschiedliche Gebäudeformen und Regionen übertragbar ist, und dass die daraus resultierenden 3D-Gebäudemodelle eine hohe Qualität in Bezug auf deren Vollständigkeit und Korrektheit aufweisen.

In a recent contribution Ferretti and co-workers (Ferretti, A., Fumagalli, A., Novali, F., Prati, C., Rocca, F., Rucci, A., 2011. A new algorithm for processing interferometric data-stacks: SqueeSAR IEEE Transactions on Geoscience and Remote Sensing 49(9), pp. 3460�3470) have proposed the SqueeSAR method, a way to exploit temporally coherent distributed scatterers in coherent SAR data stacks. Elevation and deformation or subsidence estimates are obtained with accuracy similar as in the well known persistent scatterer interferometry (PSI). In this paper we propose an alternative approach and provide a first demonstration of the optimal estimation of distributed scatterers� phase histories in urban areas. Different to SqueeSAR, we derive phase histories for each distributed scatterer pixel rather than for groups of pixels. We use the Anderson�Darling statistical test to identify neighboring samples of the same distribution. Prior to covariance matrix estimation required for maximum likelihood estimation we apply a multi-resolution defringe technique. By using TerraSAR-X high resolution spotlight data, it is demonstrated that we are able to retrieve reliable phase histories and motion parameter estimates from distributed scatterers with signal-to-noise-ratio far below the common range.

Virtual reconstruction of historic sites, planning of restorations and attachments of new building parts, as well as forest inventory are few examples of fields that benefit from the application of 3D surveying data. Originally using 2D photo based documentation and manual distance measurements, the 3D information obtained from multi camera and laser scanning systems realizes a noticeable improvement regarding the surveying times and the amount of generated 3D information. The 3D data allows a detailed post processing and better visualization of all relevant spatial information. Yet, for the extraction of the required information from the raw scan data and for the generation of useable visual output, time-consuming, complex user-based data processing is still required, using the commercially available 3D software tools. In this context, the automatic object recognition from 3D point cloud and depth data has been discussed in many different works. The developed tools and methods however, usually only focus on a certain kind of object or the detection of learned invariant surface shapes. Although the resulting methods are applicable for certain practices of data segmentation, they are not necessarily suitable for arbitrary tasks due to the varying requirements of the different fields of research. This thesis presents a more widespread solution for automatic scene reconstruction from 3D point clouds, targeting street scenarios, specifically for the task of traffic accident scene analysis and documentation. The data, obtained by sampling the scene using a mobile scanning system is evaluated, segmented, and finally used to generate detailed 3D information of the scanned environment. To realize this aim, this work adapts and validates various existing approaches on laser scan segmentation regarding the application on accident relevant scene information, including road surfaces and markings, vehicles, walls, trees and other salient objects. The approaches are therefore evaluated regarding their suitability and limitations for the given tasks, as well as for possibilities concerning the combined application together with other procedures. The obtained knowledge is used for the development of new algorithms and procedures to allow a satisfying segmentation and reconstruction of the scene, corresponding to the available sampling densities and precisions. Besides the segmentation of the point cloud data, this thesis presents different visualization and reconstruction methods to achieve a wider range of possible applications of the developed system for data export and utilization in different third party software tools.