Search

The paper presents a novel solution to the issue of incomplete regions in 3D meshes obtained through digitization. Traditional methods for estimating the surface of missing geometry and topology often yield unrealistic outcomes for intricate surfaces. To overcome this limitation, the paper proposes a neural network-based approach that generates points in areas where geometric information is lacking. The method employs 2D inpainting techniques on color images obtained from the original mesh parameterization and curvature values. The network used in this approach can reconstruct the curvature image, which then serves as a reference for generating a polygonal surface that closely resembles the predicted one. The paper's experiments show that the proposed method effectively fills complex holes in 3D surfaces with a high degree of naturalness and detail. This paper improves the previous work in terms of a more in-depth explanation of the different stages of the approach as well as an extended results section with exhaustive experiments. [Display omitted] • We perform 3D surface reconstructions using generative inpainting techniques. • 2D representation of a 3D surface geometry based on its curvature. • Application of a general purpose neural network for inpainting. • Our approach does not require dataset nor training time. • Results outperform state-of-the-art quality and naturalness of the reconstructions. [ABSTRACT FROM AUTHOR]

In recent times, there has been notable and swift advancement in the field of image dehazing. Several deep learning techniques have demonstrated remarkable proficiency in resolving homogeneous dehazing issues. Nonetheless, the current dehazing approaches are generally formulated to deal with homogeneous haze, which is often undermined in real-world scenarios due to the uncertain haze dispersion. In this paper, we propose a dehazing model named RepDehazeNet by combining a structurally Reparameterization Encoder-Decoder subnet and a Full-Resolution Attention subnet. To be specific, the structural reparameterization idea is introduced into the encoder–decoder subnet to strengthen the feature extraction of dehazed images and improve the feature extraction speed. RepDehazeNet is compared with seven SOTA models on different datasets in terms of PSNR, SSIM, parameter quantity, and inference time. Compared to the DW-GAN model, the proposed RepDehazeNet model reduces the number of parameters by 2.7 million, and improves the inference speed by 90.3%, while achieving a higher PSNR of 0.5 dB on the NH-Haze2021 dataset. The experimental results demonstrate that the proposed RepDehazeNet model can effectively improve the real-time performance, accuracy of dehazing synthesized and nonhomogeneous haze images. [Display omitted] • Structural reparameterization dehazenet: outstanding performance, faster speed. • Replacing Tanh with ReLU leads to better results. • Transfer learning addresses the problem of insufficient samples. • Dual subnets method proves highly effective in datasets of different scales. [ABSTRACT FROM AUTHOR]

Search and retrieval remains a major research topic in several domains, including computer graphics, computer vision, engineering design, etc. A search engine requires primarily an input search query and a database of items to search from. In engineering, which is the primary context of this paper, the database consists of 3D CAD models, such as washers, pistons, connecting rods, etc. A query from a user is typically in the form of a sketch, which attempts to capture the details of a 3D model. However, sketches have certain typical defects such as gaps, over-drawn portions (multi-strokes), etc. Since the retrieved results are only as good as the input query, sketches need cleaning-up and enhancement for better retrieval results. In this paper, a deep learning approach is proposed to improve or clean the query sketches. Initially, sketches from various categories are analysed in order to understand the many possible defects that may occur. A dataset of cleaned-up or enhanced query sketches is then created based on an understanding of these defects. Consequently, an end-to-end training of a deep neural network is carried out in order to provide a mapping between the defective and the clean sketches. This network takes the defective query sketch as the input and generates a clean or an enhanced query sketch. Qualitative and quantitative comparisons of the proposed approach with other state-of-the-art techniques show that the proposed approach is effective. The results of the search engine are reported using both the defective and enhanced query sketches, and it is shown that using the enhanced query sketches from the developed approach yields improved search results. [Display omitted] • The first learning-based strategy to clean rough query sketches of 3D CAD models • Introduces SketchCleanNet — an end-to-end image translation scheme • SketchCleanNet aims to understand the mapping between rough sketches and clean query images • A novel scheme to calculate the loss is introduced • Dataset Contribution: The resulting enhanced query sketch dataset is made available publicly. • This paper will significantly contribute to the research community and give researchers opportunities to develop new algorithms for search and retrieval of 3D mechanical components. [ABSTRACT FROM AUTHOR]

The research topic of estimating hand pose from the images of hand-object interaction has the potential for replicating natural hand behavior in many practical applications of virtual reality and robotics. However, the intricacy of hand-object interaction combined with mutual occlusion, and the need for physical plausibility, brings many challenges to the problem. This paper provides a comprehensive survey of the state-of-the-art deep learning-based approaches for estimating hand pose (joint and shape) in the context of hand-object interaction. We discuss various deep learning-based approaches to image-based hand tracking, including hand joint and shape estimation. In addition, we review the hand-object interaction dataset benchmarks that are well-utilized in hand joint and shape estimation methods. Deep learning has emerged as a powerful technique for solving many problems including hand pose estimation. While we cover extensive research in the field, we discuss the remaining challenges leading to future research directions. [Display omitted] • Deep learning is effectively used for estimating hand pose from images. • The correlation between a hand and an object helps in estimating hand-object pose. • Hand model helps estimate hand shape, but it restricts within the model's prior. • Implicit function methods have emerged in hand-object pose estimation. [ABSTRACT FROM AUTHOR]

Increasingly complex 3D CAD models are essential during different life-cycle stages of modern engineering projects. Even though these models contain several repeated geometries, instancing information is often not available, resulting in increased requirements for storage, transmission, and rendering. Previous research have successfully applied shape matching techniques to identify repeated geometries and thus reduce memory requirements and improve rendering performance. However, these approaches require consistent vertex topology, prior knowledge about the scene, and/or the laborious creation of labeled datasets. In this paper, we present an unsupervised deep-learning method that overcomes these limitations and is capable of identifying repeated geometries and computing their instancing transformations. The method also guarantees a maximum visual error and preserves intrinsic characteristics of surfaces. Results on real-world 3D CAD models demonstrate the effectiveness of our approach: the datasets are reduced by up to 83.93% in size. Our approach achieves better results than previous work that does not rely on supervised learning. The proposed method is applicable to any kind of 3D scene and geometry. [Display omitted] • Unsupervised deep learning method for 3D shape registration. • Does not require any previous knowledge of the 3D geometries. • Does not require a labeled dataset for any supervised training. • Guarantees an upper bound on any visual errors. • Generalizes for any 3D scene and geometry. [ABSTRACT FROM AUTHOR]

Multi-View Stereo (MVS) is a crucial technique for reconstructing the geometric structure of a scene, given the known camera parameters. Previous deep learning-based MVS methods have mainly focused on improving the reconstruction quality but overlooked the running efficiency during the actual algorithm deployment. For example, deformable convolutions have been introduced to improve the accuracy of the reconstruction results further, however, its inability for parallel optimization caused low inference speed. In this paper, we propose AdaptMVSNet which is device-friendly and reconstruction-efficient, while preserving the original results. To this end, adaptive convolution is introduced to significantly improve the efficiency in speed and metrics compared to current methods. In addition, an attention fusion module is proposed to blend features from adaptive convolution and the feature pyramid network. Our experiments demonstrate that our proposed approach achieves state-of-the-art performance and is almost 2 × faster than the recent fastest MVS method. We will release our source code. [Display omitted] [ABSTRACT FROM AUTHOR]

Raw point clouds captured by sensing devices are often contaminated with noise, which perturbs the fidelity of the original geometric information. Point cloud denoising is therefore an inseparable post-processing step, aiming to remove the noise in the point clouds. Existing point cloud denoising approaches are typically trained on datasets that have uniform point distributions and densities, making them unsuitable for effectively denoising point clouds with severe noise or irregular point distributions. In this paper, we introduce a novel random screening-based feature aggregation method for point cloud denoising. Our key insight is that merging features of dense and sparse points assists with enhancing the quality of point cloud denoising results. In specific, our approach involves randomly screening the features of local point patches and fusing richer geometric information of denser points into sparser point representations. Comprehensive experiments demonstrate that our method achieves state-of-the-art performance in the point cloud denoising task on both synthetic and real-world datasets. [Display omitted] • Random screening-based point feature aggregation is useful for point cloud denoising. • Merging features of dense points into sparser points boosts point cloud denoising. • Our pipeline demonstrates robustness to noise and saves processing time. • Our technique achieves impressive results on severe noise and irregular points. [ABSTRACT FROM AUTHOR]

Images taken under poor lighting conditions generally present problems such as low brightness, noise, and color distortion, so dealing with low-light images is challenging. To solve this problem, current mainstream methods focus on designing special deep networks. For example, many methods are based on Retinex-based models. However, the solution of the Retinex model is an ill-conditioned problem in mathematics, so these methods are difficult to obtain the expected result. In this paper, following the new paradigm of embedding prior knowledge in deep learning, we design a simple yet surprisingly effective neural network, called Multi-Resolution Edge-aware Lighting Enhancement Network (MELENet). Since resolution and acutance are two of the key factors related to the visual quality of natural images, MELENet with a dual-branch structure aims to incorporate multi-resolution edge information into deep learning for guiding lighting enhancement. Additionally, by adopting special loss functions, the model suppresses noise and prevents color distortion while obtaining sound lighting enhancement effects. Experiments have verified that such a simple solution exhibits outstanding performance compared to existing mainstream lighting enhancement methods. We also directly apply the proposed model to image de-hazing and obtain state-of-the-art results. [Display omitted] • The proposed algorithm focuses on embedding prior knowledge into the network to assist the low-light image enhancement. • An effective loss function is designed to balance the color distortion and noise suppression. • Our model can be directly applied to other image processing tasks such as image de-hazing and gain competitive results. [ABSTRACT FROM AUTHOR]

Shadow detection is a common computer vision task that identifies the location of shadows. It has been used as a key primitive for many high-level tasks such as object segmentation and video tracking. Due to the complexity and variety of shadows, it is challenging to design a one-size-fits-all shadow detection algorithm in real-world scenarios. Instead, it is often necessary in practice to pick the best shadow detection algorithm for a particular vision application. While shadow detection algorithms can be assessed by comparing the detected shadow with the ground truth, such ground truth is often unavailable in real-world vision systems consistently generating new data. In this paper, we propose a no-reference shadow detection quality assessment network (NSDQA-Net) to rank shadow detection algorithms without needing the ground truth. For a given image, the proposed assessment network first learns a reference map that compensates the false positives and false negatives of multiple input shadow detection algorithms. The reference map, as well as the original image and the input shadow detection results, are then used to regress the quality scores of the shadow detection methods. Our framework supports the assessment of one, two, or multiple algorithms in a single run, making it easier for deployment in practical scenarios. Experiments results show that the proposed network achieves optimal accuracy in no-reference shadow detection assessment. Relying on it for adaptive algorithm selection also improves the performance of shadow detection using a fixed algorithm on benchmark datasets. [Display omitted] • We identify a task of no-reference quality assessment in shadow detection. • We design a novel no-reference shadow detection quality assessment network (NSDQA-Net). • We experimentally demonstrate the state-of-the-art performance of the proposed network. [ABSTRACT FROM AUTHOR]

In this paper, we address the problem of capturing both the shape and the pose of a human character using a single depth sensor. Some previous works proposed to fit a parametric generic human template into the depth image, while others developed deep learning (DL) approaches to find the correspondence between depth pixels and vertices of the template. We designed a hybrid approach, combining the advantages of both methods, and conducted extensive experiments on the SURREAL Varol et al. (2017), DFAUST datasets Bogo etal (2017) and a subset of AMASS Mahmood et al. (2019). Results show that this hybrid approach enables us to enhance pose and shape estimation compared to using DL or model fitting separately. We also evaluated the ability of the DL-based dense correspondence method to segment also the background — not only the body parts. We also evaluated 4 different methods to perform the model fitting based on a dense correspondence, where the number of available 3D points differs from the number of corresponding template vertices. These two results enabled us to better understand how to combine DL and model fitting, and the potential limits of this approach to deal with real-depth images. Future works could explore the potential of taking temporal information into account, which has proven to increase the accuracy of pose and shape reconstruction based on a unique depth or RGB image. [Display omitted] • Dense correspondence and model fitting enhance 3D shape and pose reconstruction. • Bodypart and background color embedding enables character segmentation in depth map. • Averaging 3D candidate points before model fitting offers the best reconstruction. [ABSTRACT FROM AUTHOR]

This paper explores the potential of self-supervised learning as an alternative to supervised learning in the context of geometry-based 3D object retrieval. With the ongoing digitalization of many industries, an exponentially increasing number of 3D objects are processed by retrieval systems. In order to support new shapes, modern deep learning-based retrieval systems require retraining. The dominant paradigm for optimizing neural networks in this field is supervised classification training. Supervised learning requires time-consuming and expensive data annotation. Moreover, training neural networks for classification introduces a bias towards the classes in the training data, which is undesirable for retrieval systems encountering unseen object types in the wild. Through extensive experiments, we make a direct comparison between supervised and self-supervised learning on four datasets from three different domains (household, manufacturing and medical). For object classes seen during training, self-supervised and supervised learning are competitive. For unseen classes, self-supervised learning outperforms supervised learning in many cases. We conclude that self-supervised learning provides a powerful tool for circumventing labeling costs and providing more robust retrieval systems. [Display omitted] • Self-supervised learning is on par with supervised learning for 3D object retrieval. • Self-supervised object retrieval outperforms supervised learning out-of-distribution. • Object alignment is highly impactful on the performance of retrieval systems. • Non-contrastive siamese networks are a practical option in 3D deep learning. [ABSTRACT FROM AUTHOR]

Scene understanding has made tremendous progress over the past few years, as data acquisition systems are now providing an increasing amount of data of various modalities (point cloud, depth, RGB...). However, this improvement comes at a large cost on computation resources and data annotation requirements. To analyze geometric information and images jointly, many approaches rely on both a 2D loss and 3D loss, requiring not only 2D per pixel-labels but also 3D per-point labels. However, obtaining a 3D groundtruth is challenging, time-consuming and error-prone. In this paper, we show that image segmentation can benefit from 3D geometric information without requiring a 3D groundtruth, by training the geometric feature extraction and the 2D segmentation network jointly, in an end-to-end fashion, using only the 2D segmentation loss. Our method starts by extracting a map of 3D features directly from a provided point cloud by using a lightweight 3D neural network. The 3D feature map, merged with the RGB image, is then used as an input to a classical image segmentation network. Our method can be applied to many 2D segmentation networks, improving significantly their performance with only a marginal network weight increase and light input dataset requirements, since no 3D groundtruth is required. • Geometry. • Computer vision. • 2D segmentation. • Neural network. • multi-modalities. [ABSTRACT FROM AUTHOR]

• The goal of this paper is to create a sketch dataset that is suitable for developing deep learning-based solutions to the problem of search and retrieval in 3D CAD models. • A sketch dataset of query images, called 'CADSketchNet' has been built using the available CAD datasets. For the 58,696 CAD models of the MCB Dataset, we obtain computer-generated sketches. We call this as Dataset-A. We obtain hand-drawn sketch data for all the 801 CAD models from ESB dataset. This is the Dataset-B. • For obtaining Dataset-A, many computer sketch generation methods, including the current state-of-the art methods are experimented. We justify the proposed weighted canny technique based on the comparison. • An analysis of the quality of the hand-drawn sketches in Dataset-B is also performed. • Various deep learning architectures have been tested on the developed CADSketchNet dataset, and a detailed analysis of their performance have been reported. • Among the many experiments conducted, it is shown that a Siamese Network approach yielded the best results as compared to other features/networks. The class-wise results of the approach on both Datasets - A & B. • We also justify the need for a dedicated dataset for CAD over the regular 3D graphical shapes by testing the existing methods on the developed CADSketchNet. [Display omitted] Ongoing advancements in the fields of 3D modelling and digital archiving have led to an outburst in the amount of data stored digitally. Consequently, several retrieval systems have been developed depending on the type of data stored in these databases. However, unlike text data or images, performing a search for 3D models is non-trivial. Among 3D models, retrieving 3D Engineering/CAD models or mechanical components is even more challenging due to the presence of holes, volumetric features, presence of sharp edges etc., which make CAD a domain unto itself. The research work presented in this paper aims at developing a dataset suitable for building a retrieval system for 3D CAD models based on deep learning. 3D CAD models from the available CAD databases are collected, and a dataset of computer-generated sketch data, termed 'CADSketchNet', has been prepared. Additionally, hand-drawn sketches of the components are also added to CADSketchNet. Using the sketch images from this dataset, the paper also aims at evaluating the performance of various retrieval system or a search engine for 3D CAD models that accepts a sketch image as the input query. Many experimental models are constructed and tested on CADSketchNet. These experiments, along with the model architecture, choice of similarity metrics are reported along with the search results. [ABSTRACT FROM AUTHOR]

• The current 3D human body reconstruction technology requires a large number of 3D scanning equipment and a complex dynamic capture process, which significantly limits its rapid adoption in these application domains. The purpose of this article is to simplify the process of 3D reconstruction. • By combining deep learning methods, an end-to-end 3D human body reconstruction framework is proposed to achieve the goal of reconstructing a high-precision 3D human body model based on RGB images. [Display omitted] In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications. [ABSTRACT FROM AUTHOR]

• The iterative adaptive depth estimation for learning-based MVS to avoid excessive computation on well-estimated regions during each refinement stage. • A location selection strategy based on geometric consistency to select locations where depth hypotheses are likely to be incorrect. • A depth candidate construction strategy for the selected location based on geometric information aggregation from multiple views. • The pixelwise depth estimation module which can estimate depth value for a single location independently and be utilized for sparse depth estimation. [Display omitted] In this paper, we propose a Multi-View Stereo (MVS) network which can perform high-quality depth estimation with low memory consumption. The proposed MVS network is constructed based on the pyramid architecture to gradually refine and upsample the depth map to the desired resolution. Instead of estimating depth hypotheses for all pixels in the depth map, our method only performs prediction at adaptively selected locations, alleviating excessive computation on well-estimated positions. To estimate depth hypotheses for sparse selected locations, we propose the lightweight pixelwise depth estimation module, which can estimate accurate depth value for each selected location independently. In this paper, we propose a Multi-View Stereo (MVS) network which can perform efficient high-resolution depth estimation with low memory consumption. Classical learning-based MVS approaches typically construct 3D cost volumes to regress depth information, making the output resolution rather limited as the memory consumption grows cubically with the input resolution. Although recent approaches have made significant progress in scalability by introducing the coarse-to-fine fashion or sequential cost map regularization, the memory consumption still grows quadratically with input resolution and is not friendly for commodity GPU. Observing that the surfaces of most objects in real world are locally smooth, we assume that most of the depth hypotheses upsampled from a well-estimated depth map are accurate. Based on the assumption, we propose a pyramid MVS network based on the adaptive depth estimation, which gradually refines and upsamples the depth map to the desired resolution. Instead of estimating depth hypotheses for all pixels in the depth map, our method only performs prediction at adaptively selected locations, alleviating excessive computation on well-estimated positions. To estimate depth hypotheses for sparse selected locations, we propose the lightweight pixelwise depth estimation network, which can estimate depth value for each selected location independently. Experiments demonstrate that our method can generate results comparable with the state-of-the-art learning-based methods while reconstructing more geometric details and consuming less GPU memory. [ABSTRACT FROM AUTHOR]

Omnidirectional images and videos are widespread due to the popularization of devices for capturing and visualization. Unlike pinhole-based imagery, omnidirectional media lie on the surface of a sphere, have a 360° × 180°field of view, and store the light intensities from an entire environment. Notably, applications involving immersive augmented, mixed, and virtual reality experiences benefit from the 360° content. Although defined on the spherical domain, omnidirectional images are often mapped to a (multi-)planar representation, which results in distorted images and degrades the performance of most traditional visual computing algorithms designed to work on the plane. This paper reviews the spherical camera model, the most common capturing devices, and popular (multi-)planar representations of omnidirectional media. It also approaches the main challenges of omnidirectional visual computing, focusing on the deep learning paradigm, and tackles four visual computing applications that strongly explore the potential of omnidirectional imagery. [Display omitted] • We review the foundations of the spherical camera model. • We review standard 360°capturing devices and format representations. • We discuss the main challenges of 360°visual computing and possible solutions. • We showcase the recent advances on four selected applications that explore 360°media. [ABSTRACT FROM AUTHOR]

The existing 3D deep learning methods adopt either individual point-based features or local-neighboring voxel-based features, and demonstrate great potential for processing 3D data. However, the point-based models are inefficient due to the unordered nature of point clouds and the voxel-based models suffer from large information loss. Motivated by the success of recent point-voxel representation, such as PVCNN and DRINet, we propose a new convolutional neural network, called Multi Point-Voxel Convolution (MPVConv), for deep learning on point clouds. Integrating both the advantages of voxel and point-based methods, MPVConv can effectively increase the neighboring collection between point-based features and also promote independence among voxel-based features. Extensive experiments on benchmark datasets such as ShapeNet Part, S3DIS and KITTI for various tasks show that MPVConv improves the accuracy of the backbone (PointNet) by up to 36% , and achieves higher accuracy than the voxel-based model with up to 34 × speedups. In addition, MPVConv outperforms the state-of-the-art point-based models with up to 8 × speedups. Also, our MPVConv only needs 65% of the GPU memory required by the latest point-voxel-based model (DRINet). The source code of our method is attached in https://github.com/NWUzhouwei/MPVConv. [Display omitted] • In this paper, we propose Multi Point-Voxel Convolution (MPVConv) which takes the advantages of both the voxel- and point-based methods, and can work with different backbones for a wide range of 3D tasks. • Our MPVConv increases not only the neighboring collection for point-based features, but also the independence among voxel-based features. • 1×1×1 3D CNN is adopted in fusion module to strengthen the information association and nonlinearity between point-based and voxel-based features. • We replace the tri-linear interpolation with nearest-neighbor interpolation to reduce the computation cost of voxel feature interpolation. [ABSTRACT FROM AUTHOR]

• In this paper, we present a deep learning approach for cross-domain retrieving 3D shape and 2D sketch image. • We propose a new Cycle CNNs, which can construct the cross-domain mapping between the feature space of 2D sketches and the one of 3D shapes. • The core of our method is that the mapping relationship can be directly generated through the proposed Cycle CNNs, without explicit common feature space construction. • Our proposed Cycle CNNs works well on both sketch-based shape retrieval and shape-based sketch retrieval. In this paper, we present a deep learning approach for cross-domain retrieval of 3D shape and 2D sketch image. Cross-domain retrieval has received significant attention to flexibly find information across different modalities of data. Effective measuring the similarity between different modalities of data is the key of cross-domain retrieval. Different modalities such as shape and sketch have imbalanced and complementary relationships, which contain unequal amount of information when describing the same semantics. Existing methods based on deep learning networks mostly construct one common space for different modalities, and these nets usually loss exclusive modality-specific characteristics. To address this problem, we propose a novel Cycle CNNs to estimate the cross-domain mapping between the space of 3D shape descriptors and the one of 2D sketch features. First, we employ the existing networks to construct independent feature spaces for each modality. For each feature space, modality-specific properties within one modality are fully exploited. Next, we use the designed Cycle CNNs to learn the mapping function between different feature spaces. This network can capture the mapping relationship between 3D shape feature space and 2D sketch feature domain. Finally, we use the explored mapping between the feature spaces of different modalities to perform cross-domain retrieval. We demonstrate a variety of promising results, where our method achieves better retrieval accuracy than existing state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

In augmented reality (AR) applications, detecting and tracking real-world objects remains a challenge. Another challenge is making the loaded virtual objects truly reflect the lighting and shadow conditions of the actual environment in which they are located. In this paper, we proposed an improved method for real-time light estimation of environmental lighting in AR applications, tracking the movement of physical objects, and making the corresponding virtual objects reflect the lighting and shadows of the actual environment. First, we proposed an effective method for estimating light in an AR scene by training a deep neural network once on different scenes from the input of RGB-D images to determine the direction of the brightest light. Second, we experimented with advanced methods of detecting and tracking different types of real objects (flat surfaces, simple geometries, and complex geometries) for visualizing virtual materials on the top of these surfaces, where virtual materials will acquire the conditions of the real place in terms of lighting, reflection, and shadows depending on our proposed method for light estimation. Our method of light estimation was tested via experiments on the three types of real objects mentioned above. Results indicate that our method achieves higher resolution and lower error rate compared with other work. Our methods have achieved very good results in terms of detection accuracy, tracking accuracy, realistic visualization of materials, matching real shade, and lighting conditions. [Display omitted] [ABSTRACT FROM AUTHOR]

Image segmentation is one of the most classical computer vision tasks. Segmentation tasks yield a set of classes attributed to individual pixels instead of sparsely predicted images or patches, such as in classification or detection tasks. However, creating annotation sets for pixelwise tasks is a very costly task, often requiring hours for labeling single samples in images with multiple classes of objects. In this context, unsupervised learning can be leveraged either to expedite the annotation procedure and/or to guide the segmentation algorithms altogether without the need for manual annotations. Classical unsupervised segmentation methods leveraged techniques from areas as graph theory, image processing, clustering or supervised classifiers in order to achieve "shallow" pixelwise classification. These techniques usually aim to achieve superpixel over-segmentations by grouping similar pixels that should pertain to the same object. Modern deep unsupervised approaches for image segmentation aimed to group pixels in a data-driven way by using the capabilities of deep architectures to process unstructured data such as images. Later, self-supervised learning bypassed the need for labels via pretext tasks, compelling deep architectures to learn more generic features capable of enhancing downstream tasks, including segmentation. The generalized representations produced by unsupervised models have propelled the recent progress in self-supervised, few- and zero-shot learning and even general-purpose foundational models in computer vision, yielding state-of-the-art results across diverse tasks and datasets. This paper provides an overview of unsupervised and generalizable approaches for image segmentation, introduces key concepts and terminology, and discusses the main aspects of state-of-the-art methods. Additionally, we highlight prominent applications in various domains such as remote sensing, medical imaging, and geology. Finally, we discuss trends and future directions for state-of-the-art unsupervised image segmentation. [Display omitted] [ABSTRACT FROM AUTHOR]

Foundation models, such as ULIP-2 (Xue et al., 2023) recently projected forward the field of 3D deep learning. These models are trained with significantly more data and show superior representation learning capacity in many downstream tasks like 3D shape classification and few-shot part segmentation. A particular characteristic of the recent 3D foundation models is that they are typically multi-modal , and involve image (2D) as well as caption (text) branches. This leads to an intricate interplay that benefits all modalities. At the same time, the nature of the 3D encoders alone, involved in these foundation models is not well-understood. Specifically, there is little analysis on the utility of both pre-trained 3D features provided by these models, or their capacity to adapt to new downstream 3D data. Furthermore, existing studies typically focus on label-oriented downstream tasks, such as shape classification, and ignore other critical applications, such as 3D content-based object retrieval. In this paper, we fill this gap and show, for the first time, how 3D foundation models can be leveraged for strong 3D-to-3D retrieval performance on seven different datasets, on par with state-of-the-art view-based architectures. We evaluate both the pre-trained foundation models, as well as their fine-tuned versions using downstream data. We compare supervised fine-tuning using classification labels against two self-supervised label-free fine-tuning methods. Importantly, we introduce and describe a methodology for fine-tuning, as we found this to be crucial to make transfer learning from 3D foundation models work in a stable manner. [Display omitted] • Self-supervised 3D object retrieval is competitive with supervised learning on the same data distribution. • Self-supervised 3D object retrieval outperforms supervised learning out-of-distribution. • Object alignment is highly impactful on the out-of-distribution performance of retrieval systems. • Non-contrastive siamese networks are a practical option in 3D deep learning. [ABSTRACT FROM AUTHOR]

Text-conditioned image synthesis methods such as DALLE-2, IMAGEN, and Stable Diffusion are gaining strong attention from deep learning and art communities recently. Meanwhile, Image-to-Image (Img2Img) synthesis applications that emerged from the pioneering Neural Style Transfer (NST) approach have swiftly transitioned towards the feed-forward Automatic Style Transfer (AST) methods, due to numerous constraints inherent in the former method, including inconsistent synthesis outcomes and sluggish optimization-based synthesis process. However, NST holds significant potential yet remains relatively underexplored within this research domain. In this paper, we revisited the original NST method and uncovered its potential to attain image quality comparable to the AST synthesis methods across a diverse range of artistic styles. We propose a two-stage Feature-guided Style Transfer (FeaST) which consists (a) pre-stylization step called Sketching to address the poor initialization issue, and (b) Finetuning to guide the synthesis process based on high-frequency (HF) and low-frequency (LF) guidance channels. By addressing the issues of inconsistent synthesis and slow convergence inherent in the original method, FeaST unlocks the full capabilities of NST and significantly enhances its efficiency. [Display omitted] • We revisit the Neural Style Transfer (NST) framework for high-fidelity art generation. • We proposed a feature-guided reconstruction based on separated guidance channels for high and low-frequency feature regions to solve slow convergence issues. • We proposed a pre-stylization step called Sketching which improved the image synthesis time by a six-order magnitude. • We experimented a set of optimal weighting factors in feature-guided reconstruction for consistent results over wide variety of styles and contents. [ABSTRACT FROM AUTHOR]

In order to facilitate the robust and precise 3D vessel shape extraction and quantification from in-vivo Magnetic Resonance Imaging (MRI), this paper presents a novel multi-scale Knowledge Transfer Vision Transformer (i.e., KT-ViT) for 3D vessel shape segmentation. First, it uniquely integrates convolutional embeddings with transformer in a U-net architecture, which simultaneously responds to local receptive fields with convolution layers and global contexts with transformer encoders in a multi-scale fashion. Therefore, it intrinsically enriches local vessel feature and simultaneously promotes global connectivity and continuity for a more accurate and reliable vessel shape segmentation. Furthermore, to enable using relatively low-resolution (LR) images to segment fine scale vessel shapes, a novel knowledge transfer network is designed to explore the inter-dependencies of data and automatically transfer the knowledge gained from high-resolution (HR) data to the low-resolution handling network at multiple levels, including the multi-scale feature levels and the decision level, through an integration of multi-level loss functions. The modeling capability of fine-scale vessel shape data distribution, possessed by the HR image transformer network, can be transferred to the LR image transformer to enhance its knowledge for fine vessel shape segmentation. Extensive experimental results on public image datasets have demonstrated that our method outperforms all other state-of-the-art deep learning methods. • A novel knowledge transfer transformer for segmenting fine-scale 3D vasculature. • A unique vision transformer integrating both local convolutions and global encodings. • A U-shape multi-scale processing through the hierarchical convolutional transformer. • Knowledge transfer through transformers to explore the resolution inter-dependencies. [ABSTRACT FROM AUTHOR]

Images acquired during underwater activities suffer from environmental properties of the water, such as turbidity and light attenuation. These phenomena cause color distortion, blurring, and contrast reduction. In addition, irregular ambient light distribution causes color channel unbalance and regions with high-intensity pixels. Recent works related to underwater image enhancement, and based on deep learning approaches, tackle the lack of paired datasets generating synthetic ground-truth. In this paper, we present a self-supervised learning methodology for underwater image enhancement based on deep learning that requires no paired datasets. The proposed method estimates the degradation present in underwater images. Besides, an autoencoder reconstructs this image, and its output image is degraded using the estimated degradation information. Therefore, the strategy replaces the output image with the degraded version in the loss function during the training phase. This procedure misleads the neural network that learns to compensate the additional degradation. As a result, the reconstructed image is an enhanced version of the input image. Also, the algorithm presents an attention module to reduce high-intensity areas generated in enhanced images by color channel unbalances and outlier regions. Furthermore, the proposed methodology requires no ground-truth. Besides, only real underwater images were used to train the neural network, and the results indicate the effectiveness of the method in terms of color preservation, color cast reduction, and contrast improvement. [Display omitted] • Underwater image enhancement and self-supervised learning. • Image restoration and enhancement based on autoencoder and degradation. • Attention mechanism for high-intensity or saturated regions in the image. [ABSTRACT FROM AUTHOR]

This paper describes the methods submitted for evaluation to the SHREC 2022 track on pothole and crack detection in the road pavement. A total of 7 different runs for the semantic segmentation of the road surface are compared, 6 from the participants plus a baseline method. All methods exploit Deep Learning techniques and their performance is tested using the same environment (i.e., a single Jupyter notebook). A training set, composed of 3836 semantic segmentation image/mask pairs and 797 RGB-D video clips collected with the latest depth cameras was made available to the participants. The methods are then evaluated on the 496 image/masks pairs in the validation set, on the 504 pairs in the test set and finally on 8 video clips. The analysis of the results is based on quantitative metrics for image segmentation and qualitative analysis of the video clips. The participation and the results show that the scenario is of great interest and that the use of RGB-D data is still challenging in this context. [Display omitted] • A dataset of image/mask pairs and RGB-D video clips of road damage. • A fair evaluation of the proposed methods implementing them on a single Jupyter notebook. • A qualitative evaluation of videos of road damages from different points of view. [ABSTRACT FROM AUTHOR]

Outdoor point-cloud object localization is an essential processing step for urban scene analysis and modeling in numerous applications, especially in land surveying and site analysis. Given the increasingly use of Terrestrial Laser Scanning (TLS) and LiDAR 3D data acquisition, numerous annotated point-cloud datasets are available and can be used to evaluate computer vision and machine learning based algorithms. Nevertheless, current point-cloud datasets mainly focus on object detection and classification in autonomous driving or urban planning type of applications, and share redundant object classes, e.g., trees, vehicles and pedestrians, which limit their usefulness for land surveying and site analysis. This paper introduces a novel 3D benchmark dataset LiSurveying, which is a large-scale point-cloud dataset with over a billion points and uncommon urban object categories in complex outdoor environments. Our dataset incorporates more urban object classes than existing datasets. Its instances have diverse point densities, shapes and dimensions, which also impose a challenge for point-cloud detection and classification algorithms. We conducted baselines experiments for point-cloud classification using machine learning classifiers and deep learning methods on different subsets of the LiSurveying dataset, and we are able to demonstrate that the various types of object classes, number of instances per class, distribution of object points, and variety of complex scenes, make this LiSurveying benchmark dataset suitable for evaluating 3D point-cloud classification, semantic segmentation, and object detection algorithms. [Display omitted] • Large-scale (point-wise and 3D bounding boxes) TLS-LiDAR dataset for Land Surveying. • 3D Urban benchmark for object classification y detection. • Learning-based methods applied on complex urban environments. • Comparative performance of existing algorithms for LiDAR classification. [ABSTRACT FROM AUTHOR]

This paper presents the methods that have participated in the SHREC 2022 track on the fitting and recognition of simple geometric primitives on point clouds. As simple primitives we mean the classical surface primitives derived from constructive solid geometry, i.e., planes, spheres, cylinders, cones and tori. The aim of the track is to evaluate the quality of automatic algorithms for fitting and recognizing geometric primitives on point clouds. Specifically, the goal is to identify, for each point cloud, its primitive type and some geometric descriptors. For this purpose, we created a synthetic dataset, divided into a training set and a test set, containing segments perturbed with different kinds of point cloud artifacts. Among the six participants to this track, two are based on direct methods, while four are either fully based on deep learning or combine direct and neural approaches. The performance of the methods is evaluated using various classification and approximation measures. [Display omitted] • Evaluation of methods for fitting and recognizing simple primitives in point clouds. • 46,925 3D point clouds, divided into a training set and a test set. • The goal is to recognize a primitive type and provide its analytical representation. • Performance is discussed by classification, recognition, and approximation measures. [ABSTRACT FROM AUTHOR]

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology. [Display omitted] • The problem of multimodal 3D registration of CT volumes and 3D point clouds is formally defined. • A framework for multimodal 3D registration of CT volumes and 3D point clouds is proposed. • A DLN that combines regular CNNs suited for data with standard grid structure and GDL suited for unstructured data. • The proposed network employs siamese architecture for effective multimodality fusion. • A multimodal dataset for evaluating algorithms for aligning CT volumes and point clouds is created. [ABSTRACT FROM AUTHOR]

Point clouds are gaining prominence as a method for representing 3D shapes, but their irregular structure poses a challenge for deep learning methods. In this paper we propose CloudWalker, a novel method for learning 3D shapes using random walks. Previous works attempt to adapt Convolutional Neural Networks (CNNs) or impose a grid or mesh structure to 3D point clouds. This work presents a different approach for representing and learning the shape from a given point set. The key idea is to impose structure on the point set by multiple random walks through the cloud for exploring different regions of the 3D object. Then we learn a per-point and per-walk representation and aggregate multiple walk predictions at inference. Our approach achieves state-of-the-art results for two 3D shape analysis tasks: classification and retrieval. • We propose CloudWalker a novel representation for point clouds using random walks. • This work presents a different approach for representing and learning 3D shape. • Our walks impose order and overcome the lack of connectivity and structure. • We learn per-point and per-walk representation and aggregate multiple walks per shape. • Our approach achieves SOTA results for 3D shape analysis tasks. [ABSTRACT FROM AUTHOR]

Deep neural networks exhibit extreme superiority in natural language processing and computer vision. With the fast development of 3D sensing technology, point cloud has been wildly used in robotics and autonomous driving. In recent years, several attempts have been made to utilize deep neural networks in the field of point cloud processing. However, most top performing network cannot provide the confidence of each prediction. In this paper, we implement Bayesian deep learning method in point cloud processing and propose ProbConv, a three-dimensional convolutional kernel with stochastic weights. Based on ProbConv, a Bayesian deep neural network named ProbNet is further designed to effectively accomplish classification and segmentation tasks on point cloud data. Since Bayesian neural network can naturally calculate the confidence of its prediction, ProbNet is able to provide the confidence of the segmentation result. The experimental results on ModelNet40, ShapeNet and S3DIS demonstrate that ProbNet improves accuracy in both object classification and semantic segmentation tasks. The confidence provided by the ProbNet is reliable to reflect the accuracy of its prediction. [Display omitted] • Bayesian deep learning and pyramid-divided respective field is implemented on ProbConv. • ProbNet is based on the proposed ProbConv operator to provide confidence of its prediction. • Experiment result shows that ProbNet improves accuracy while providing reliable confidence. [ABSTRACT FROM AUTHOR]

Pixel-level annotation for supervised learning tends to be tedious and inaccurate when facing complex scenes, leading models to produce incomplete predictions, especially on self shadows and shadow boundaries. This paper presents a weakly supervised graph convolutional network (namely WSGCN) for generating an accurate pseudo shadow mask from only several annotation scribbles. Given these limited annotations and a priori superpixel segmentation mask, we seek a robust graph construction strategy and a label propagation method. Specifically, our network operates on superpixel-graphs, allowing us to reduce the data dimensions by several magnitudes. Then, under the guidance of scribbles, we formulate the generation of a shadow mask as a weakly supervised learning task and learn a 2-layer graph convolutional network (GCN) separately for each training image. Experimental results on the benchmark datasets SBU and ISTD, show that our network can achieve impressive performance by using a few thousand parameters, and training our re-annotated data can further improve the performance of the state-of-the-art detectors. [Display omitted] • We can generate shadow mask from only a few scribbles. • Designing loss functions to ensure the consistency between pseudo shadow mask and image. • Training on re-annotated data improve the performance of state-of-the-art detectors. [ABSTRACT FROM AUTHOR]

The primary purpose of this paper is to realize robust place recognition algorithms towards simultaneous viewpoint and condition changes, and provide satisfactory computational efficiency. In this paper, we significantly improve the viewpoint invariance of the SeqSLAM algorithm by using state-of-the-art deep learning techniques to generate robust feature representations of images and develop the SeqCNNSLAM . Experimental results show that SeqCNNSLAM outperforms state-of-the-art place recognition systems in most cases, such as, when precision is maintained at 100%, the maximum recall obtained by SeqCNNSLAM is 50% higher than SeqSLAM on the Norland dataset with simultaneous condition change and 12.5% viewpoint change. Besides, we develop an acceleration method called A-SeqCNNSLAM , which exploits the location relationship between the matching images of adjacent images to reduce the matching range of the current image. Experimental results demonstrate that an acceleration of  ∼  5 times is achieved with minimal accuracy degradation of  ∼  5%. Finally, to enable A-SeqCNNSLAM adaptability in new environments, O-SeqCNNSLAM is devised for the online parameter adjustment in A-SeqCNNSLAM. [ABSTRACT FROM AUTHOR]

To display images on panels and screens of different dimensions, there is a need to create algorithms that help adjust them to desired sizes through image retargeting (IR). In this sense, choosing the right algorithms seems to be a challenge. From this perspective, this paper was to propose a method for the assessment of retargeted images using deep learning (DL) models. To this end, the Conditional Random Fields as Recurrent Neural Networks (CRF-RNN) approach was applied to find image pixels in terms of importance and, the You-Only-Look-Once (YOLO) model was employed to identify semantic objects of images. As well, image patch similarity was computed based on Siamese Neural Network (SNN), and output image distortion was recognized by the CNN model. With reference to three well-known databases, viz., MIT RetargetMe, NRID and CUHK, the assessment results of the proposed algorithm demonstrated its superior usage as compared to the existing ones. [Display omitted] • Operating an R-CNN model to extract image importance maps. • Using a pre-trained YOLO to extract regions containing important image objects. • Training SNN to compute corresponding patch-similarity of original and retargeted images. • Training regression-based CNN to measure distortion of retargeted images based on pixel displacement patterns. • Merging measures using Gaussian Process Regression (GPR) and calculating the final measure. [ABSTRACT FROM AUTHOR]

3D reconstruction from a single image is one of the core computer vision problems. Thanks to the development of deep learning, 3D reconstruction of a single image has demonstrated impressive progress in recent years. Existing researches use Chamfer distance as a loss function to guard the training of the neural network. However, the Chamfer loss will give equal weights to all points inside the 3D point clouds. It tends to sacrifice fine-grained and thin structures to avoid incurring a high loss, which will lead to visually unsatisfactory results. This paper proposes a framework that can recover a detailed three-dimensional point cloud from a single image by focusing more on boundaries (edge and corner points). Experimental results demonstrate that the proposed method outperforms existing techniques significantly, both qualitatively and quantitatively, and has fewer training parameters. [Display omitted] • Reconstructing the full 3D shape of an object from a single image. • Using a differentiable point cloud projection module to get the projected images. • Increasing visual reconstruction quality. [ABSTRACT FROM AUTHOR]

Surface reconstruction is a fundamental problem in 3D graphics. In this paper, we propose a learning-based approach for implicit surface reconstruction from raw point clouds without normals. Our method is inspired by Gauss Lemma in potential energy theory, which gives an explicit integral formula for the indicator functions. We design a novel deep neural network to perform surface integral and learn the modified indicator functions from un-oriented and noisy point clouds. We concatenate features with different scales for accurate point-wise contributions to the integral. Moreover, we propose a novel Surface Element Feature Extractor to learn local shape properties. Experiments show that our method generates smooth surfaces with high normal consistency from point clouds with different noise scales and achieves state-of-the-art reconstruction performance compared with current data-driven and non-data-driven approaches. [Display omitted] • A learning-based approach for surface reconstruction from raw point clouds. • A novel Surface Element Feature Extractor to learn local shape properties. • Aggregating the point-wise contributions of Gauss Lemma in the network. • Generating smooth surfaces with high normal consistency. [ABSTRACT FROM AUTHOR]

Learning new representations of 3D point clouds is an active research area in 3D vision, as the order-invariant point cloud structure still presents challenges for the design of neural network architectures. Recent work explored learning global, local, or multi-scale features for point clouds. However, none of the earlier methods focused on capturing contextual shape information by analyzing local orientation distributions of points. In this paper, we use point orientation distributions around a point in order to obtain an expressive local neighborhood representation for point clouds. We achieve this by dividing the spherical neighborhood of a given point into predefined cone volumes, and statistics inside each volume are used as point features. In this way, a local patch can be represented not only by the selected point's nearest neighbors, but also by considering a point density distribution defined along multiple orientations around the point. We are then able to construct an orientation distribution function (ODF) neural network that makes use of an ODFBlock which relies on MLP (multi-layer perceptron) layers. The new ODFNet model achieves state-of-the-art accuracy for object classification on ModelNet40 and ScanObjectNN datasets, and segmentation on ShapeNet and S3DIS datasets. [Display omitted] • ODFs which incorporate the directional information inside a neighborhood is defined. • ODFNet, for classification and segmentation of point clouds is presented. • ODFs provide fully rotation invariance or invariance in x–y plane. • ODFNet obtained SoTA accuracy on ModelNet40, Shapenet, S3DIS and ScanObjectNN. [ABSTRACT FROM AUTHOR]

Deep learning (DL) models have achieved impressive performance in various domains such as medicine, finance, and autonomous vehicle systems with advances in computing power and technologies. However, due to the black-box structure of DL models, the decisions of these learning models often need to be explained to end-users. Explainable Artificial Intelligence (XAI) provides explanations of black-box models to reveal the behavior and underlying decision-making mechanisms of the models through tools, techniques, and algorithms. Visualization techniques help to present model and prediction explanations in a more understandable, explainable, and interpretable way. This survey paper aims to review current trends and challenges of visual analytics in interpreting DL models by adopting XAI methods and present future research directions in this area. We reviewed literature based on two different aspects, model usage and visual approaches. We addressed several research questions based on our findings and then discussed missing points, research gaps, and potential future research directions. This survey provides guidelines to develop a better interpretation of neural networks through XAI methods in the field of visual analytics. [Display omitted] • A comprehensive survey for visual analytics, particularly those that adopted explainable artificial intelligence (XAI) methods, in interpreting Neural Networks is conducted. • We reviewed the literature based on model usage and visual approaches. • We concluded some visual approaches commonly used to support the illustration of XAI methods for various types of data and machine learning models; however, a generic approach is needed for the field. • We listed several future work including data manipulations, scalability, and bias in data representation, and generalizable real-time visualizations integrating XAI. [ABSTRACT FROM AUTHOR]

The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene's materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85 × 10−4 when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes. [Display omitted] • Automatic end-to-end method to estimate the environment lighting in XR applications. • A CNN architecture that learns a latent-space of the environment lighting. • A methodology to generate egocentric mixed-reality-views from HDR panoramas. • Real-time lighting estimation that does not make assumptions about the XR scene. [ABSTRACT FROM AUTHOR]

3D CAD models play an important role in large-scale engineering projects. The increasing demand for highly-detailed datasets presents a challenge for efficient storage, transmission, and rendering. To reduce dataset size, 3D shape matching techniques have been proposed to find repeated triangle meshes, but are strongly dependent on surface triangulation. Meanwhile, existing shape registration techniques are not well suited for the 3D CAD domain. In this paper, we present a deep learning-based framework that uses point clouds to identify repeated instances of triangle meshes (a single 3D CAD model component) overcoming the limitations of previous work and guaranteeing an upper bound on any geometric errors. The framework combines PointNet++ for shape classification with a registration procedure based on Principal Component Analysis and the Adam optimizer. The resulting affine transformation can be used to efficiently instantiate repeated CAD geometries. Using the proposed framework, we were able to reduce a real-world 3D CAD model to 2.61% of its original size, while preserving its geometric accuracy and improving rendering performance. [Display omitted] • Guarantee of an upper bound on any geometric errors. • Independent of vertices ordering and mesh topology. • Generalization for any kind of geometry. • Well suited for the 3D CAD domain. • Accurate and consistent 3D shape registration. [ABSTRACT FROM AUTHOR]

• A comprehensive survey of LIDAR-based 3D object detection methods with Deep learning for autonomous driving • Presentation of a common operational pipeline that sets the base for a structured categorisation which facilitates comparison and emerges the similarities and dissimilarities among the presented methods. • Presentation of an exhaustive up-to-date list of 3D object detectors. The method's description focuses on the particularities of each method with respect to the operational pipeline identifying pros and cons towards an effective and efficient detection outcome. • Fruitful discussion that aims to identify key features which should be either adopted or avoided in the design of new 3D object detectors. [Display omitted] LiDAR-based 3D object detection for autonomous driving has recently drawn the attention of both academia and industry since it relies upon a sensor that incorporates appealing features like insensitivity to light and capacity to capture the 3D spatial structure of an object along with the continuous reduction of its purchase cost. Furthermore, the emergence of Deep Learning as the means to boost performance in 3D data analysis stimulated the production of a multitude of solutions for LIDAR-based 3D object detection which followed different approaches in an effort to respond effectively to several challenges. In view of this, this paper presents a comprehensive survey of LIDAR-based 3D object detection methods wherein an analysis of existing methods is addressed by taking into account a new categorisation that relies upon a common operational pipeline which describes the end-to-end functionality of each method. We next, discuss the existing benchmarking frameworks and present the performance achieved by each method in each of them. Finally, a discussion is presented that provides key insights aiming to capture the essence of current trends in LIDAR-based 3D object detection. [ABSTRACT FROM AUTHOR]

• A network generation model of small data assisting reconstruction is proposed. Small data learning as prior knowledge can guide the generation of large-scale features, and the reconstruction network further reconstructs the details of the original image. • A new illumination normalization model is proposed. The assisting network is based on small data learning, which reduces the dependence of data size. The reconstruction network explores the different network parameters for different images, which avoids the poor generalization ability. [Display omitted] Image transfer based on deep learning methods can achieve good results in face illumination processing. However, data constraints and generalization ability restrict the further development of these methods in this field. In this paper, we propose small data assisting face image illumination normalization. For data constraints, we train the network model on a small number of image pairs. In terms of generalization ability, the proposed normalization network parameters are different for processing different face images. Small data learning can provide prior knowledge, and the reconstruction process can guide detail generation. Therefore, the small data learning network and the reconstruction network are complementary to each other in image generating mode when only a small quantity of data is available. We use this network mode to normalize the illumination and reconstruct the super-resolution of face images. After illumination normalization, super-resolution reconstruction can obtain more precise face information and further improve the face recognition rate. Experiments show that the proposed method has good normalization performance when only 500 face image pairs are used for training. [ABSTRACT FROM AUTHOR]

• Combine TP-Stream and SP-Stream to recognize skeleton-based dynamic hand gestures. • Use compact joints encoding method to generate hand skeleton image. • Propose a global enhancement module to enhance important feature maps. • Enhance motion features on different coordinate axes using motion perception module. [Display omitted] With the development of 3D hand pose estimation technologies, skeleton-based dynamic hand gesture recognition has attracted widespread attention. In this paper, we propose a novel framework for skeleton-based dynamic hand gesture recognition. In the spatial perception stream (SP-Stream), we design a compact joints encoding method. It can adaptively select compact joints based on the convex hull of the hand skeleton and encode them into a skeleton image for fully extracting spatial features. Besides, we present a global enhancement module (GEM) to enhance key feature maps. In the temporal perception stream (TP-Stream), we propose a motion perception module (MPM) to enhance the notable movement of hand gestures on X / Y / Z coordinate axes. Experimental results show that the proposed framework performs better than the state-of-the-art methods on two benchmark datasets. [ABSTRACT FROM AUTHOR]

• The irregularity and complexity of mesh data make shape segmentation challenging. • Mesh data are similar to point cloud data in spatial property so shape segmentation can be treated as a pointwise labeling problem. • Spatial information of mesh data is able to be learned by a point based network architecture by transforming shapes into point sets. • A simple and effective shape segmentation method which treats mesh segmentation as point labeling by converting each face into its barycenter and normal vector per shape with the spatial information of given shape preserving. • A intuitive and efficient operation, called Barycentric Dual Graph Edge Convolution (BDGEC), extracts edge features by operating over the nodes of Barycentric Dual Graph (BDG) intrinsic for the given shape. • A novel point based DNN, named LGCP Net, is proposed for mesh segmentation which can capture local and global features of shape efficient to obtain satisfying segmentation performance. [Display omitted] Segmenting 3D shapes represented by meshes remains a challenging problem, due to the irregularity and complexity of meshes. Point cloud, on the other hand, can be considered as the simplest no-frills approximation for meshes. Therefore, in this paper, we regard the shape segmentation problem as a point labeling task: Given a shape, we first transform it into points encoding barycenters and normal vectors of faces. Then we construct a Barycentric Dual Graph (BDG) on the transformed points, and propose a Barycentric Dual Graph Edge Convolution (BDGEC) to extract features from the graph. Based on the BDGEC, we further propose a novel point-based deep neural network (DNN) named local-global combined point-based network (LGCPNet). Our LGCPNet consists of three modules, of which the Local Module and Global Module capture local and global features respectively, while the Fusion Module uses a gate mechanism to aggregate local features and global features, and obtain the point labeling result. Comprehensive experimental results on various datasets demonstrate that the proposed network inherits the merits of point-based DNNs and achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

• We present CMA, a novel cross-modal data fusion approach that incorporates the attention mechanism. • CMA extracts discriminative cross-modal features which are more robust to 6D object pose estimation. • We evaluate our method on two widely used datasets: LINEMOD dataset and YCB-Video dataset. • Experimental results demonstrate that our method achieves superior performance on both datasets over the state-of-the-art methods as well as high-efficiency. [Display omitted] Deep learning methods for 6D object pose estimation based on RGB and depth (RGB-D) images have been successfully applied to robotic manipulation and grasping. Among these approaches, the fusion of RGB and depth modalities is one of the most critical issues. Most existing works performed fusion via either simple concatenation, or element-wise multiplication of the features generated by these two modalities. Despite achieving impressive progress, such fusion strategies do not explicitly consider the different contributions of RGB and depth modalities, leaving a gap for performance enhancement. In this paper, we present a Cross-Modal Attention (CMA) component for the problem of 6D object pose estimation. With the attention mechanism, features of two different modalities are aggregated adaptively through the attention weights, such that powerful representations from the RGB-D images can be efficiently extracted. Comprehensive experiments on both LINEMOD and YCB-Video datasets demonstrate that the proposed approach achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

• The first deep learning framework for texture cartoonization. • A deep regularization module to predict and enhance the regularity of texture. • An unsupervised module for texture cartoonization. • Tailored self-supervised and adversarial objectives for network training. [Display omitted] Texture plays an important role in cartoon images to represent materials of objects and enrich visual attractiveness. However, manually crafting a cartoon texture is not easy, so amateurs usually directly use cartoon textures downloaded from the Internet. Unfortunately, Internet resources are quite limited and often patented, which restrict the users from generating visually pleasant and personalized cartoon textures. In this paper, we propose a deep learning based method to generate cartoon textures from natural textures. Different from the existing photo cartoonization methods that only aim to generate cartoonic images, the key to our method is to generate cartoon textures that are both cartoonic and regular. To achieve this goal, we propose a regularization module to generate a regular natural texture with similar appearance as the input, and a cartoonization module to cartoffonize the regularized natural texture into a regular cartoon texture. Our method successfully produces cartoonic and regular textures from various natural textures. [ABSTRACT FROM AUTHOR]

Point cloud registration aligns point clouds through transformation or deformation. The existing advanced registration methods perform poorly in registering under various perturbations, especially for large rotations and small overlaps. Moreover, these methods cannot achieve rigid and non-rigid registrations simultaneously. In this paper, we propose a two-step generative optimization (GO) method to achieve registration, which alleviates the constraints of transformation representation and completely circumvents the matching of correspondences, making it suitable for rigid registration with large rotations and small overlaps, as well as for non-rigid registration. GO achieves registration by generating a pseudo point cloud approximating the target point cloud. The pseudo point cloud is iteratively deformed to approach the target point cloud through an estimation step (ES) and an updating step (US). The ES step generates initial pseudo point clouds in three different views and estimates their deformation via the extracted features decoding the similarity between the pseudo point clouds and the target point cloud. The US step calculates regressors to update the current estimations, generating new pseudo point clouds closer to the target point cloud. The final pseudo point clouds generated are averaged to attain the registration result. We evaluate the performance of GO for rigid and non-rigid registrations through the comparison with seven deep learning-based methods on eight datasets, consisting of both 3D and 2D data and ranging from synthetic to real scenes. Our experimental results demonstrate the high robustness and stability of GO in rigid registrations and illustrate the feasibility and high accuracy for non-rigid registrations. [Display omitted] Given a source point cloud S and a target point cloud M, GO aims to register S to M by iteratively generating different pseudo point clouds P t S, P t F, P t O, and making each generated pseudo point cloud approach the target model M to mimic the deformation of the source model S. The registration result S T is the average of the generated pseudo point clouds at the final iteration. Specifically, the GO iteration alternates between performing an estimation (ES) step and an updating (US) step. The former aims to estimate the coordinates of the newly generated pseudo point clouds in different views via the extracted feature of the existing pseudo point clouds. The latter intends to update the existing pseudo point clouds for generating new pseudo point clouds by making the current estimation approach the target point cloud. The generated pseudo point clouds are then used to estimate the coordinates of new pseudo point clouds in the next ES step. GO terminates the iteration once the training error is below a threshold. And the generated pseudo point clouds are averaged to attain the final registration result simultaneously. • A novel two-step generative optimization (GO) is proposed for rigid and non-rigid registration. • GO outperforms both traditional and deep learning methods in robustness and accuracy. • The generation of pseudo models expedites the convergence of GO. [ABSTRACT FROM AUTHOR]

Point cloud registration precision degrades when the scene contains symmetric structures or repeated patches. Also, the common operation of continuous downsampling in existing registration methods causes the loss of detailed information and leads to further registration precision degradation. In this paper, we propose a high precision point cloud registration network (HPNet) to learn discriminative features by considering both multi-scale information and position information. We first propose a multi-scale feature extraction module similar to the feature pyramid that allows the extracted features to contain multi-scale information. Then, an information interaction is performed on the features to learn global contextual information by using Transformer with a hybrid position encoding, which takes into account both absolute and relative positions of points. Finally, the features obtained from the information interaction module are directly used to predict the point correspondences. Comprehensive experiments on 3DMatch, 3DLoMatch, ModelNet, and ModelLoNet datasets demonstrate the state-of-the-art performance of the proposed method and the implementation of HPNet in SLAM shows its effectiveness in real application. [Display omitted] • Point cloud registration loses detail information during continuous downsampling. • Feature fusion of low-level detail information and high-level semantic information. • Hybrid position encoding consider both absolute and relative positions. positions. • Point cloud registration algorithms applied to SLAM. [ABSTRACT FROM AUTHOR]

• Point inception based deep neural network for segmentation of the 3D point clouds. • Extract the local features using inception module to improve the performance. • Use GAP over other pooling methods to avoid overfitting. • Exhaustive evaluation and comparative analysis with the existing methods on two state-of-the-art datasets, namely, ShapeNet and PartNet. • Additional architecture and the ablation study demonstrate the effectiveness of proposed model. [Display omitted] Point clouds, being the simple and compact representation of surface geometry of 3D objects, have gained increasing popularity with the evolution of deep learning networks for classification and segmentation tasks. Unlike human, teaching the machine to analyze the segments of an object is a challenging task and quite essential in various machine vision applications. In this paper, we address the problem of segmentation and labelling of the 3D point clouds by proposing a inception based deep network architecture called PIG-Net, that effectively characterizes the local and global geometric details of the point clouds. In PIG-Net, the local features are extracted from the transformed input points using the proposed inception layers and then aligned by feature transform. These local features are aggregated using the global average pooling layer to obtain the global features. Finally, feed the concatenated local and global features to the convolution layers for segmenting the 3D point clouds. We perform an exhaustive experimental analysis of the PIG-Net architecture on two state-of-the-art datasets, namely, ShapeNet [1] and PartNet [2]. We evaluate the effectiveness of our network by performing ablation study. [ABSTRACT FROM AUTHOR]

• Computer graphics. • Computer vision. • Deep learning. • 3D reconstruction. • Virtual human digitization. • Generative adversarial network. [Display omitted] With the development of AR/VR technologies, a reliable and straightforward way to digitize a three-dimensional human body is in high demand. Most existing methods use complex equipment and sophisticated algorithms, but this is impractical for everyday users. In this paper, we propose a pipeline that reconstructs a 3D human shape avatar from a single image. Our approach simultaneously reconstructs the three-dimensional human geometry and whole body texture map with only a single RGB image as input. We first segment the human body parts from the image and then obtain an initial body geometry by fitting the segment to a parametric model. Next, we warp the initial geometry to the final shape by utilizing a silhouette-based dense correspondence. Finally, to infer invisible back texture from a frontal image, we propose a network called InferGAN. Based on human semantic information, we also propose a method to handle partial occlusion by reconstructing the occluded body parts separately. Comprehensive experiments demonstrate that our solution is robust and effective on both public and our own datasets. Our human avatars can be easily rigged and animated using MoCap data. We have developed a mobile application that demonstrates this capability for AR applications. [ABSTRACT FROM AUTHOR]

• Attention-based local relation learning module can dynamically explore semantic relation in an anisotropic way. • Context-guided aggregation module further enhances the distinguishing ability of points in feature space. • Gated propagation strategy flexibly filters out irrelevant and redundant information between encoder and decoder. • Multi-scale supervision boosts the training process. Semantic segmentation of point cloud is a fundamental problem in scene-level understanding. Despite advancement in recent years by leveraging capabilities of Neural Networks and massive labeling datasets available, providing fine-grained semantic segmentation for point cloud is still challenging, given the fact that point cloud is usually unstructured, unordered and sparse. In this paper, we achieve semantic point cloud labeling by adaptively exploring semantic relation and aggregating contextual information between points. Specifically, we first introduce an attention-based local relation learning module for collecting local features, which can capture semantic relation in a manner of anisotropy. And we then design a novel context aggregation module guided by multi-scale supervision to obtain long-range dependencies between semantically-correlated points and enhance the distinctive ability of points in feature space. In addition, a gated propagation strategy is adopted instead of skip links to conditionally concatenate local point features in different layers. We empirically evaluate our method on public benchmarks (S3DIS and ShapeNetPart), and demonstrate our performance is on par or better than state-of-the-art methods. [ABSTRACT FROM AUTHOR]