Back to Search
Start Over
Juggling with representations: On the information transfer between imagery, point clouds, and meshes for multi-modal semantics
- Source :
- ISPRS Journal of Photogrammetry and Remote Sensing. 176:55-68
- Publication Year :
- 2021
- Publisher :
- Elsevier BV, 2021.
-
Abstract
- The automatic semantic segmentation of the huge amount of acquired remote sensing data has become an important task in the last decade. Images and Point Clouds (PCs) are fundamental data representations, particularly in urban mapping applications. Textured 3D meshes integrate both data representations geometrically by wiring the PC and texturing the surface elements with available imagery. We present a mesh-centered holistic geometry-driven methodology that explicitly integrates entities of imagery, PC and mesh. Due to its integrative character, we choose the mesh as the core representation that also helps to solve the visibility problem for points in imagery. Utilizing the proposed multi-modal fusion as the backbone and considering the established entity relationships, we enable the sharing of information across the modalities imagery, PC and mesh in a twofold manner: (i) feature transfer and (ii) label transfer. By these means, we achieve to enrich feature vectors to multi-modal feature vectors for each representation. Concurrently, we achieve to label all representations consistently while reducing the manual label effort to a single representation. Consequently, we facilitate to train machine learning algorithms and to semantically segment any of these data representations – both in a multi-modal and single-modal sense. The paper presents the association mechanism and the subsequent information transfer, which we believe are cornerstones for multi-modal scene analysis. Furthermore, we discuss the preconditions and limitations of the presented approach in detail. We demonstrate the effectiveness of our methodology on the ISPRS 3D semantic labeling contest (Vaihingen 3D) and a proprietary data set (Hessigheim 3D).
- Subjects :
- FOS: Computer and information sciences
Information transfer
010504 meteorology & atmospheric sciences
Computer science
Computer Vision and Pattern Recognition (cs.CV)
Feature vector
Computer Science - Computer Vision and Pattern Recognition
0211 other engineering and technologies
Point cloud
02 engineering and technology
Semantics
01 natural sciences
Polygon mesh
Computer vision
Computers in Earth Sciences
Representation (mathematics)
Engineering (miscellaneous)
021101 geological & geomatics engineering
0105 earth and related environmental sciences
business.industry
Sensor fusion
Atomic and Molecular Physics, and Optics
Computer Science Applications
Feature (computer vision)
Artificial intelligence
business
Subjects
Details
- ISSN :
- 09242716
- Volume :
- 176
- Database :
- OpenAIRE
- Journal :
- ISPRS Journal of Photogrammetry and Remote Sensing
- Accession number :
- edsair.doi.dedup.....ebba9b90e956d14fbe8555cf0e8b3b2e