Search

This paper focuses on indoor semantic segmentation based on RGB-D images. Semantic segmentation is a pixel-level classification task that has made steady progress based on fully convolutional networks (FCNs). However, we find there is still room for improvements in the following three aspects. The first is related to multi-scale feature extraction. Recent state-of-the-art works forcibly concatenate multi-scale feature representations extracted by spatial pyramid pooling, dilated convolution or other architectures, regardless of the spatial extent for each pixel. The second is regarding RGB-D modal fusion. Most successful methods treat RGB and depth as two separate modalities and force them to be joined together regardless of their different contributions to the final prediction. The final aspect is about the modeling ability of extracted features. Due to the “local grid” defined by the receptive field, the learned feature representation lacks the ability to model spatial dependencies. In addition to these modules, we design a depth estimation module to encourage the RGB network to extract more effective features. To solve the above challenges, we propose four modules to address them: scale-aware module, modality-aware module, attention module and depth estimation module. Extensive experiments on the NYU-Depth v2 and SUN RGB-D datasets demonstrate that our method is effective against RGB-D indoor semantic segmentation.

This paper proposes LinkNet, a 2D-3D linked multi-modal network served for online semantic segmentation of RGB-D videos, which is essential for real-time applications such as robot navigation. Existing methods for RGB-D semantic segmentation usually work in the regular image domain, which allows efficient processing using convolutional neural networks (CNNs). However, RGB-D videos are captured from a 3D scene, and different frames can contain useful information of the same local region from different views. Working solely in the image domain fails to utilize such crucial information. Our novel approach is based on joint 2D and 3D analysis. The online process is realized simultaneously with 3D scene reconstruction, from which we set up 2D-3D links between continuous RGB-D frames and 3D point cloud. We combine image color and view-insensitive geometric features generated from the 3D point cloud for multi-modal semantic feature learning. Our LinkNet further uses a recurrent neural network (RNN) module to dynamically maintain the hidden semantic states during 3D fusion, and refines the voxel-based labeling results. The experimental results on SceneNet [1] and ScanNet [2] demonstrate that the semantic segmentation results of our framework are stable and effective.

We present an efficient method to propagate sparse user edits indicated by strokes on 360 ∘ panoramas. Our algorithm first projects each equirectangular pixel to its corresponding position on a 3D unit sphere, so each pixel can be characterized by a feature vector consisting of its 3D coordinates and RGB color values. We formulate edit propagation as an optimization problem that aims to satisfy the user edit constraints while preserving the manifold structure of the image at the same time. To solve the problem using a linear system efficiently, we first construct the K-D tree structure in the feature space to cluster pixels. Then we optimize the manifold structure where both the number of nearest neighbors and their corresponding weights are determined by the feature distributions. We further apply a multiresolution strategy to speedup the edit propagation. Our method is the first to perform interactive edit propagation on 360 ∘ panoramas. Experiments show that our method is able to generate seam-free and visually pleasing results, and users can receive instant feedback during interactive editing.

3D face reconstruction from a single image is a classic computer vision problem with many applications. However, most works achieve reconstruction from face photos, and little attention has been paid to reconstruction from other portrait forms. In this paper, we propose a learning-based approach to reconstruct a 3D face from a single face sketch. To overcome the problem of no paired sketch-3D data for supervised learning, we introduce a photo-to-sketch synthesis technique to obtain paired training data, and propose a dual-path architecture to achieve synergistic 3D reconstruction from both sketches and photos. We further propose a novel line loss function to refine the reconstruction with characteristic details depicted by lines in sketches well preserved. Our method outperforms the state-of-the-art 3D face reconstruction approaches in terms of reconstruction from face sketches. We also demonstrate the use of our method for easy editing of details on 3D face models.

Accurate edge reconstruction is critical for depth map super resolution (SR). Therefore, many traditional SR methods utilize edge maps to guide depth SR. However, it is difficult to predict accurate edge maps from low resolution (LR) depth maps. In this paper, we propose a deep edge map guided depth SR method, which includes an edge prediction subnetwork and an SR subnetwork. The edge prediction subnetwork takes advantage of the hierarchical representation of color and depth images to produce accurate edge maps, which promote the performance of SR subnetwork. The SR subnetwork is a disentangling cascaded network to progressively upsample SR result, where every level is made up of a weight sharing module and an adaptive module. The weight sharing module extracts the general features in different levels, while the adaptive module transfers the general features to the specific features to adapt to different degraded inputs. Quantitative and qualitative evaluations on various datasets with different magnification factors demonstrate the effectiveness and promising performance of the proposed method. In addition, we construct a benchmark dataset captured by Kinect-v2 to facilitate research on real-world depth map SR.

Automatic segmentation of 3D tooth models into individual teeth is an important step in orthodontic CAD systems. 3D tooth segmentation is a mesh instance segmentation task. Complex geometric features on the surface of 3D tooth models often lead to failure of tooth boundary detection, so it is difficult to achieve automatic and accurate segmentation by traditional mesh segmentation methods. We propose a novel solution to address this problem. We map a 3D tooth model isomorphically to a 2D harmonic parameter space and convert it into an image. This allows us to use a CNN to learn a highly robust image segmentation model to achieve automated and accurate segmentation of 3D tooth models. Finally, we map the image segmentation mask back to the 3D tooth model and refine the segmentation result using an improved Fuzzy Clustering-and-Cuts algorithm. Our method has been incorporated into an orthodontic CAD system, and performs well in practice.

This paper presents a deep CNN approach for point-based semantic scene labeling. This is challenging because 3D point clouds do not have a canonical domain and can have complex geometry and substantial variation of sampling densities. We propose a novel framework where the convolution operator is defined on depth maps around sampled points, which captures characteristics of local surface regions. We introduce Depth Mapping (DM) and Reverse Depth Mapping (RDM) operators to transform between the point domain and the depth map domain. Our depth map based convolution is computationally efficient, robust to scene scales and sampling densities, and can capture rich surface characteristics. We further propose to augment each point with feature encoding of the local geometric patches resulted from multi-method through patch pooling network (PPN). The patch features provide complementary information and are fed into our classification network to achieve semantic segmentation.

Image colorization is an important and difficult problem in image processing with various\ud applications including image stylization and heritage restoration. Most existing\ud image colorization methods utilize feature matching between the reference color image\ud and the target grayscale image. The effectiveness of features is often significantly\ud affected by the characteristics of the local image region. Traditional methods usually\ud combine multiple features to improve the matching performance. However, the same\ud set of features is still applied to the whole images. In this paper, based on the observation\ud that local regions have different characteristics and hence different features may\ud work more effectively, we propose a novel image colorization method using automatic\ud feature selection with the results fused via a Markov Random Field (MRF) model for\ud improved consistency. More specifically, the proposed algorithm automatically classifies\ud image regions as either uniform or non-uniform, and selects a suitable feature\ud vector for each local patch of the target image to determine the colorization results.\ud For this purpose, a descriptor based on luminance deviation is used to estimate the\ud probability of each patch being uniform or non-uniform, and the same descriptor is\ud also used for calculating the label cost of the MRF model to determine which feature\ud vector should be selected for each patch. In addition, the similarity between the luminance\ud of the neighborhood is used as the smoothness cost for the MRF model which enhances the local consistency of the colorization results. Experimental results on a variety\ud of images show that our method outperforms several state-of-the-art algorithms,\ud both visually and quantitatively using standard measures and a user study.

The objective of our work is to enable the reading of fragile scrolled historical parchments without the need to physically unravel them, thus providing valuable information to a wide range of scholarly disciplines. This problem has not been investigated by the computer vision community properly yet due to the need for parchment scanning technology: standard X-ray equipment is not sufficient as there is a requirement to extract out parchment ink in addition to the parchment's underlying structure. Effective data recovery is also compromised as content from historical scrolled documents is inaccessible due to the deterioration of the parchment. We create a 3D volumetric model of a scrolled parchment's underlying geometry and perform digital unwrapping of the parchment, producing a readable image of the text as an output. The proposed recovery framework consists of structure preserving anisotropic filtering in combination with robust segmentation, surface modelling and ink projection. We demonstrate with real examples how our algorithm is able to recover the underlying text and to solve the major challenge for scrolled parchment analysis, namely segmentation of connected layers and processing the data without user interaction. HighlightsA pioneering attempt to automate virtual reading of a real historical parchment.The presented framework does not require user interaction.We propose a flexible framework which is controlled by several parameters.Our model is able to reveal a scrolled parchment's content if its ink contains enough iron.We demonstrate performance of the framework with five different real scrolls.

Solid textures require large storage and are computationally expensive to synthesize. In this\ud paper, we propose a novel solid representation called gradient solids to compactly represent\ud solid textures, including a tricubic interpolation scheme of colors and gradients for\ud smooth variation and a region-based approach for representing sharp boundaries. We further\ud propose a novel approach to directly synthesize gradient solid textures from exemplars.\ud Compared to existing methods, our approach avoids the expensive step of synthesizing the\ud complete solid textures at voxel level and produces optimized solid textures using our representation.\ud This avoids significant amount of unnecessary computation and storage involved\ud in the voxel-level synthesis while producing solid textures with comparable quality to the\ud state of the art. The algorithm is much faster than existing approaches for solid texture\ud synthesis and makes it feasible to synthesize high-resolution solid textures in full. We also\ud propose a novel application — instant editing propagation on full solids.

3D models are now widely available for use in various applications. The demand for automatic model analysis and understanding is ever increasing. Model segmentation is an important step towards model understanding, and acts as a useful tool for different model processing applications, e.g. reverse engineering and modeling by example. We extend a random walk method used previously for image segmentation to give algorithms for both interactive and automatic model segmentation. This method is extremely efficient, and scales almost linearly with the number of faces, and the number of regions. For models of moderate size, interactive performance is achieved with commodity PCs. We demonstrate that this method can be applied to both triangle meshes and point cloud data. It is easy-to-implement, robust to noise in the model, and yields results suitable for downstream applications for both graphical and engineering models.