Search

Generating an interpretable and compact representation of 3D shapes from point clouds is an important and challenging problem. This paper presents CSG-Stump Net, an unsupervised end-to-end network for learning shapes from point clouds and discovering the underlying constituent modeling primitives and operations as well. At the core is a three-level structure called {\em CSG-Stump}, consisting of a complement layer at the bottom, an intersection layer in the middle, and a union layer at the top. CSG-Stump is proven to be equivalent to CSG in terms of representation, therefore inheriting the interpretable, compact and editable nature of CSG while freeing from CSG's complex tree structures. Particularly, the CSG-Stump has a simple and regular structure, allowing neural networks to give outputs of a constant dimensionality, which makes itself deep-learning friendly. Due to these characteristics of CSG-Stump, CSG-Stump Net achieves superior results compared to previous CSG-based methods and generates much more appealing shapes, as confirmed by extensive experiments. Project page: https://kimren227.github.io/projects/CSGStump/, Accepted to ICCV 2021

As the applications of intelligent agents (IAs) are gradually increasing in daily life, they are expected to have reasonable social intelligence to interact with people by appropriately interpreting human behavior and intention. This paper presents a method to estimate whether people have willingness to join in a conversation, which helps to endow IAs with the capability of detecting potential participants. The method is built on the CNN-LSTM network, which takes image features and social signals as input, making use of general information conveyed in images, semantic social cues proven by social psychology studies, and temporal information in the sequence of inputs. The network is designed to have a multi-branch structure with the flexibility of accommodating different types of inputs. We also discuss the signal transition in multiparty human-robot interaction scenarios. The method is evaluated on three datasets with social signals and/or images as inputs. The results show that the proposed method can infer human engagement intention well.

In this paper, a model free adaptive predictive tracking control method for the robotic manipulators with uncertain parameters is proposed. The compact form dynamic linearization method is used to transform the nonlinear robotic manipulator into a dynamic linearized data model. Then, the model free adaptive predictive tracking control algorithm is designed based on the robot manipulator data model. The proposed control method does not need accurate system model information, merely uses the input and output data of the robot manipulator which is a data-driven control method. Numerical simulations are carried out for PUMA560 robotic manipulator, and the effectiveness of the designed algorithm is verified by the simulation results.

Corbel is a common category of decorative architectural geometry that has clear structure and aesthetic design. This paper presents a method for automatically generating a group of new corbel models from one selected by the user in the dataset. The method consists of offline learning and online generation. The offline learning trains two VAE models (2D CurveVAE and 3D VoxelVAE) for learning the feature representation of corbel parts. The online generation includes a generation algorithm by evolution that evolves to product new generation of models by crossing over and mutating features, and a feature-driven deformation that synthesizes 3D mesh representation of corbel models. By integrating these technical components, we develop a creative corbel modeling tool capable of generating new corbel models that are both “more of the same” and “surprising”, which is demonstrated by experiments. Ministry of Education (MOE) Accepted version This work is supported by the Ministry of Education, Singapore, under its MoE Tier-2 Grant (2017-T2-1-076).

3D instance segmentation plays a predominant role in environment perception of robotics and augmented reality. Many deep learning based methods have been presented recently for this task. These methods rely on either a detection branch to propose objects or a grouping step to assemble same-instance points. However, detection based methods do not ensure a consistent instance label for each point, while the grouping step requires parameter-tuning and is computationally expensive. In this paper, we introduce a novel framework to enable end-to-end instance segmentation without detection and a separate step of grouping. The core idea is to convert instance segmentation to a candidate assignment problem. At first, a set of instance candidates is sampled. Then we propose an assignment module for candidate assignment and a suppression module to eliminate redundant candidates. A mapping between instance labels and instance candidates is further sought to construct an instance grouping loss for the network training. Experimental results demonstrate that our method is more effective and efficient than previous approaches.

This work addresses the problem of 3D human shape reconstruction from point clouds. Considering that human shapes are of high dimensions and with large articulations, we adopt the state-of-the-art parametric human body model, SMPL, to reduce the dimension of learning space and generate smooth and valid reconstruction. However, SMPL parameters, especially pose parameters, are not easy to learn because of ambiguity and locality of the pose representation. Thus, we propose to incorporate skeleton awareness into the deep learning based regression of SMPL parameters for 3D human shape reconstruction. Our basic idea is to use the state-of-the-art technique PointNet++ to extract point features, and then map point features to skeleton joint features and finally to SMPL parameters for the reconstruction from point clouds. Particularly, we develop an end-to-end framework, where we propose a graph aggregation module to augment PointNet++ by extracting better point features, an attention module to better map unordered point features into ordered skeleton joint features, and a skeleton graph module to extract better joint features for SMPL parameter regression. The entire framework network is first trained in an end-to-end manner on synthesized dataset, and then online fine-tuned on unseen dataset with unsupervised loss to bridges gaps between training and testing. The experiments on multiple datasets show that our method is on par with the state-of-the-art solution.

This paper presents a spatial-varying illumination model for shading-based depth refinement that based on a smooth Spherical Harmonics (SH) lighting field. The proposed lighting model is able to recover shading under challenging unknown lighting conditions, thus improving the quality of recovered surface detail. To avoid over-parameterization, local lighting coefficients are treated as a vector-valued function which is represented by subdivided surfaces using Catmull-Clark subdivision. We solve our lighting model utilizing a highly parallelized scheme that recovers lighting in a few milliseconds. A real-time shading-based depth recovery system is implemented with the integration of our proposed lighting model. We conduct quantitative and qualitative evaluations on both synthetic and real world datasets under challenging illumination. The experimental results show our method outperforms the state-of-the-art real-time shading-based depth refinement system. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

Sketch-based modeling uses freeform strokes as basic modeling metaphor and provides an intuitive way for shape modeling, for instance, for cyberworlds. This paper presents a new method to enhance sketch-based modeling. The core idea of the method is to enhance the sketching process by allowing the user to iteratively sketch to progressively create initial shapes that interpolate the sketched strokes. This process considers all the sketches and the up-to-date constructed 3D shape, which enables the user to be aware of the shape of the sketched model. The key underlying technique that supports this process is a novel surface construction algorithm, which generates 3D triangular mesh models with gradual shape changes during iterative sketching. Experiments demonstrate that the presented method can allow users to intuitively and flexibly create and edit 3D models even with complex topology, which is usually difficult in existing sketch-based modeling systems.

Leak detection is an important issue to be resolved of pipeline networks integrity and early leak detection and monitoring is highly demanded in the water network systems and energy transport field, since it can minimize environmental impact and economic loss. Nowadays, most leak detection methods require equipment to be used and/or installed which can be costly and often involve a lot of modelling and simulation. The goal of this work is to offer a solution for accurate in-pipe leak position detection by design of swimming robot (passively in-pipe swimming ball) that is equipped with an Inertial measurement Unit (IMU) board and a microphone required for leak localization. The data (Gyros, Accels) measured by IMU are used in calculating the position of the in-pipe swimbot and the deep learning classifier is applied using data recorded by the microphone to identify different acoustic characteristics of leak sounds.

Permanent Magnet Synchronous Motor (PMSM) as the key component of the Electric Vehicles (EVs), has numerous advantages over other synchronous motors. Fault diagnosis is essential to guarantee the safety and continuity of the operation for EVs. An undetected fault in the PMS M system may lead to high repair cost, or even catastrophic failure. Among lots of faults in the PMSM, the most serious are short circuit faults, such as inter-turn fault, phase-to-ground fault. If it is not detected in time, the inter-turn fault may lead to phase-to-ground fault. For the PMSM, the early detection of motor failure has a very important significance. In this paper, the mathematic model and the phase-to-ground fault model of PMSM in abc coordinates is proposed, and the corresponding model is established based on MATLAB /Simulink. The zero-sequence voltage component (ZSVC) is analyzed in the inter-turn fault, which provides the basis for the fault diagnosis of PMSM.

Fault diagnosis is essential to guarantee the continuity and safety of the operation for Electric Vehicles (EVs). Permanent Magnet Synchronous Motor (PMSM) drive system as the key component of the EVs requires a robust and reliable system to detect and diagnosis various faults that may occur during operation. An undetected fault in the PMSM system may lead to high repair cost, or even catastrophic failure. Therefore, effective and timely fault diagnosis methods are greatly needed, which may realize the preventive maintenance for the PMSM drive system. In this paper, the types and fault diagnosis methods are summarized and compared. The PMSM fault diagnosis methods are divided into traditional and modern groups. The advantages, disadvantages of each method are reported. And the new directions are discussed based on previous work in the end.

The state-of-the-art photometric stereo (PS) methods typically apply shading cues on each vertex and represent a vertex normal as a non-linear function of its neighboring vertices. Such vertex-based representation leads to huge computational cost and limits it from processing dense meshes. In this paper, we propose a PS based surface reconstruction using mesh face based representation. In particular, we propose to apply the shading cue on each mesh face instead of each vertex and optimize face normals instead of vertex normals. We develop a two-step approach to solve the surface recovery problem, where we first optimize face normals using shading cues and then update vertices by the optimized face normals. Experimental results show that, compared with the state-of-the-art, such a two-step approach is able to reduce the runtime significantly as well as handling much denser meshes.

Home healthcare becomes more and more important with the increased aging population. The advent of various low-cost sensing devices makes it tempting to develop low-cost, non-intrusive systems to monitor the variations of human being. In this paper, we describe a Kinect-based gait analysis and visualization system as a case study in this direction. The system uses depth images and skeleton captured by the Kinect to generate a BVH file recording the motion information, extracts features of gait for detecting abnormal gait, and customizes the 3D body model for personalized motion visualization. Compared to previous work in this area, the proposed system has advantages since it integrates gait classification and visualization which may bring new possibilities in healthcare. The experiments show that our proposed system achieves accurate gait classification as well as flexible personalized 3D visualization.

Localizing, identifying and extracting human groups with consistent appearance jointly from a personal photo stream is an important problem and has wide applications. Inspired by recent advances in object detection, scene understanding and image cosegmentation, in this paper we explore explicit constraints to label and segment human objects rather than other non-human objects and “stuff”. We propose a novel soft human shape cue, which is initialized by color line poselet-based human part detection, further processed through a generalized geodesic distance transform, and refined finally with a joint bilateral filter. Such a high-level object cue is then integrated with other low-level unary and pairwise terms into a principled conditional random field framework, which can be efficiently solved by fast graph cut algorithms. We evaluate our algorithm over the FlickrMFC human dataset, and show that it achieves state-of-the-art performance for this challenging task.

Warping a tetrahedral mesh driven by boundary surface deformation is useful in many applications. Although some methods have been developed to transform the mesh to conform to the deformed boundary surface, it is still a challenging problem to construct an inversion free warped mesh maintaining a compatible topology. In this paper, these two problems are solved by a novel method that combines radial basis function (RBF)-based warping and adaptive mesh refinement. We iteratively transform the mesh using RBF-based warping with a safe step size to ensure that no element is inverted. The use of the RBF-based warping ensures a smooth warping and thus generates a high-quality warped volumetric mesh. To avoid too small step sizes, we refine the elements that are potentially inverted. The refinement is performed on the original and the warped meshes in the same way so as to maintain compatible topology between them. The results of our method can be used in many areas such as finite element simulation and shape interpolation. We demonstrate the effectiveness of our method with a set of examples.

The placement of knot vector and the determination of control points are two fundamental issues in B-spline surface reconstruction. This paper presents a variational approach to construct B-spline surfaces from a set of data points. The approach finds the optimal placement of knots and control points simultaneously while most previous methods determine the knots heuristically or in a separate step. Moreover, different from most previous methods using least squares metric, our approach adapts L_1-norm with total variation (TV) as regularization in the fitting procedure, which enables the approach to handle both Gaussian noise and outliers in the same manner and is able to automatically optimize the placement of knot vector to faithfully reconstruct the sharp features. A numerical solver based on the augmented Lagrangian method is also proposed in the paper to efficiently solve the TV-L_1 optimization. Experimental results demonstrate the effectiveness and efficiency of the proposed variational B-spline surface reconstruction.

Considering the existing depth recovery approaches that have different limitations when applying to Kinect depth data, in this paper, we propose to integrate their effective features including adaptive support region selection, reliable depth selection and color guidance together under a unified framework for Kinect depth recovery. In particular, we formulate our depth recovery as an energy minimization problem, which solves the depth hole-filling and denoising simultaneously. The energy function consists of a fidelity term and a regularization term. The fidelity term takes into account the characteristics of Kinect data. The regularization term is designed to incorporate the joint bilateral filtering (JBF) kernel and the joint trilateral filtering (JTF) kernel so as to facilitate both depth hole-filling and denoising. Moreover, the JBF kernel is modified to incorporate the structure information. Both simulations on the benchmark Middlebury dataset and experiments on real Kinect data show that our proposed method achieves state-of-the-art performance in terms of recovery accuracy and visual quality.

Though quite a few image segmentation benchmark datasets have been constructed, there is no suitable benchmark for semantic image segmentation. In this paper, we construct a benchmark for such a purpose, where the ground-truths are generated by leveraging the existing fine granular ground-truths in Berkeley Segmentation Dataset (BSD) as well as using an interactive segmentation tool for new images. We also propose a percept-tree-based region merging strategy for dynamically adapting the ground-truth for evaluating test segmentation. Moreover, we propose a new evaluation metric that is easy to understand and compute, and does not require boundary matching. Experimental results show that, compared with BSD, the generated ground-truth dataset is more suitable for evaluating semantic image segmentation, and the conducted user study demonstrates that the proposed evaluation metric matches user ranking very well.

Given any image input by users, how to automatically cutout the object-of-interest is a challenging problem due to lack of information of the object-of-interest and the background. Saliency detection techniques are able to provide some rough information about object-of-interest since they highlight high-contrast or high attention regions or pixels. However, the generated saliency map is often noisy and directly applying it for segmentation often leads to erroneous results. Motivated by the recent progress on image co-segmentation and internet image retrieval techniques, in this paper, we propose to use the user input image for segmentation as a query image to Google Images and then employ the top returned Google images to build up the knowledge about the object-of-interest in the user input image. Particularly, we develop a lightweight algorithm to learn the knowledge of the object-of-interest in the retrieved images to enhance the saliency map of the input image. Then, the enhanced saliency map is used to initialize the graph-cut to extract the object-of-interest. Experiments with the Mcgill dataset and multiple challenge cases demonstrate the effectiveness of our method in terms of producing a clean cutout.

A powerful image editing system called OVIE is described, which provides fast and accurate creation, composition, rendering and other manipulation of image contents. Flexibility and convenience of the system are achieved by including two modules: image decomposition and image vectorization to understand and represent an image respectively. To understand an image comprehensively, we propose to integrate image segmentation, shape completion and image completion techniques to ensure a seamless image editing. An array of pixels is replaced by vector data with geometric edit ability for image representation since the geometrically-based editing has physical meanings and thus it is more natural or intuitive for users to edit. Compared to the existing works, our system is more convenient and can generate effects with higher quality.

An embedded SCADA system has been studied and fulfilled in order to operate and control industry objects directly at the terminal of production line. A solution is suggested for an embedded SCADA system based on the development of embedded computer and network communication. The design and implementation methods are introduced for technics structure, hardware and software of this system. The first 10 units have been brought into use at power plants and power monitor centers since 2010. The result of site acceptance indicates that measuring precision and reliability follow correlative standards.

The existing high-accuracy environment matting extraction methods usually require the capturing of thousands of sample images and spend several hours in data acquisition. In this paper, a fast environment matting algorithm is proposed to extract the environment matte data effectively and efficiently. In particular, we incorporate the recently developed compressive sensing theory to simplify the data acquisition process. Moreover, taking into account special properties of light refraction and reflection effects of transparent object, we further propose to use hierarchical sampling and group clustering based recovery to accelerate the matte extraction process. Compared with the state-of-the-art approaches, our proposed algorithm significantly accelerates the environment matting extraction process while still achieving high-accuracy results.

This paper presents a sketch-based modeling system with auxiliary planes as references for 3D freeform shape design. The user first creates a rough 3D model of arbitrary topology by sketching some contours of the model. Then the user can use sketching to perform deformation, extrusion, etc, to edit the model. To regularize and interpret the user's inputs properly, we introduce some rules for the strokes into the system, which are based on both the semantic meaning of the sketched strokes and human psychology. Unlike other sketching systems, all the creation and editing operations in the presented system are performed with reference to some auxiliary planes that are automatically constructed based on the user’s sketches or default settings. The use of reference planes provides a heuristic solution to the problem of ambiguity of 2D interface for modeling in 3D space. Examples demonstrate that the presented system can allow the user to intuitively and intelligently create and edit 3D models even with complex topology, which is usually difficult in other similar sketch-based modeling systems.

Patch-based image completion proceeds by iteratively filling the target (unknown) region by the best matching patches in the source image. Inmost existing such algorithms, the size of the patches is either fixed and specified by a default number or simply chosen to be inversely proportional to the spatial frequency. However, it is noted that the patch size affects how well the filled patch captures the local characteristics of the source image and thus the final completion accuracy. Thus in this paper we propose a new method to compute appropriate patch sizes for image completion to improve its performance. In particular, we formulate the patch size determination as an optimization problem that minimizes an objective function involving image gradients and distinct and homogenous features. Experimental results show that our method can provide a significant enhancement to patch-based image completion algorithms.

Most of the previous multi-view reconstruction algorithms focus on minimizing reconstruction distortion, i.e. reconstructing a 3D model as close to the real object as possible, and rely on subsequent simplification process to control the reconstruction rate for practical usage. In this paper, in addition to reconstruction distortion, we directly consider the reconstruction rate in wide-baseline multi-view reconstruction. In particular, we develop a novel rate-distortion efficient multi-view reconstruction system, which consists of two major components: the PDE-based mesh optimization and the local subdivision based mesh refinement. Although both components are prior arts, to achieve rate-distortion optimization, new ingredients are proposed including non-uniform sample distribution for efficient and accurate surface distortion approximation in mesh optimization and non-uniform vertex distribution in mesh refinement. From a set of calibrated and segmented wide baseline images, our developed system is able to reconstruct the optimal 3D mesh that achieves minimum reconstruction distortion at any rate within a certain range. Experimental results show that our proposed system significantly outperforms the original PDE-based reconstruction method in both rate-distortion performance and visual quality.

Seeded image segmentation is a popular type of supervised image segmentation in computer vision and image processing. Previous methods of seeded image segmentation treat the image as a weighted graph and minimize an energy function on the graph to produce a segmentation. In this paper, we propose to conduct the seeded image segmentation according to the result of a heat diffusion process in which the seeded pixels are considered to be the heat sources and the heat diffuses on the image starting from the sources. After the diffusion reaches a stable state, the image is segmented based on the pixel temperatures. It is also shown that our proposed framework includes the RandomWalk algorithm for image segmentation as a special case which diffuses only along the two coordinate axes. To better control diffusion, we propose to incorporate the attributes (such as the geometric structure) of the image into the diffusion process, yielding an anisotropic diffusion method for image segmentation. The experiments show that the proposed anisotropic diffusion method usually produces better segmentation results. In particular, when the method is tested using the groundtruth dataset of Microsoft Research Cambridge (MSRC), an error rate of 4.42% can be achieved, which is lower than the reported error rates of other state-of-the-art algorithms.

Existing digital image composition algorithms neglect the out-of-gamut problem, i.e. some pixel values in a composited image exceed the displayable or printable range. The commonly used solutions, including hard clipping or linear scaling, result in either detail loss or global contrast reduction. Directly applying the existing high dynamic range (HDR) compression algorithms cannot achieve pleasant visual quality either. In our previous work, we proposed a gamut fitting algorithm by formulating gamut fitting as an energy-minimization problem and used the cubic polynomial in the Bernstein-Bezier form to compute the optimal mapping curve. Despite the good performance achieved, the problem of the previous algorithm lies in the necessity of fine tuning the weighting parameter in the proposed energy function. In this paper, we further improve our previous method by using piecewise mapping curves with multiple Bernstein polynomials to find the optimal mapping curve. The proposed approach can be regarded as a post-process to enhance the visual quality of the resulting images from image composition applications. The performance of the proposed method is compared with our previous method and the state-of-the-art HDR compression algorithms.

Cardiovascular disease is the number one cause of death today in developed countries and is projected to remain so. Cardiac intervention is a minimally invasive solution to diagnose and treat the disease. Training of young cardiologists is playing an important role towards quality cardiac healthcare. The advance of interactive and graphics technologies has offered a new avenue to train young cardiologists using virtual patients which can tremendously reduce the cost and risk. This paper presents a novel interactive simulation for cardiac intervention. Two-body collision between the catheter and the heart wall is detected based on a boundary representation. GPU based algorithms are developed to enhance the performance of simulator for real-time and realistic cardiac intervention.

The major drawback of real-time environment matting method is that the extracted environment matte data often contains significant amount of noise and errors. Although some filtering methods have been employed to remove the noise and obtain acceptable composition results, they are incapable of removing potential errors. In this paper, we first establish a light motion field to better describe the environmental matting effect of transparent objects and propose a new vector field fitting algorithm to simultaneously remove both noise and errors in the extracted matte data by using energy minimization approach. Experimental results show that our method is less sensitive to noise and error and can generate perceptually better composition results than the existing real-time environment matting approaches.

Existing digital image composition algorithms neglect the out-of-gamut problem, i.e. some pixel values in a composited image exceed the displayable or printable range. In this paper, we show that the commonly used solution, i.e. hard clipping or linear scaling, results in either detail loss or global contrast reduction. Directly applying the existing high dynamic range (HDR) compression algorithms cannot achieve pleasant visual quality either. Thus, we propose a gamut fitting method to solve this out-of-gamut problem in image composition. In particular, we formulate gamut fitting as a multi-criteria optimization problem. Polynomials in the Bernstein-Bezier form are used to compute the optimal gamut mapping curve. The proposed approach can be regarded as a post-processing procedure to enhance the visual quality of the resulting images from image composition applications.

The latest wavelet based 3D mesh coding schemes convert an irregular mesh into a semi-regular mesh and directly apply the zerotree-like image coders to compress the wavelet vectors generated in the remeshing process. The major problem of such type of approaches is that the particular properties of semi-regular meshes are not being considered in the zerotree-like image coders. In this paper, we propose an improved wavelet based 3D mesh coder. The basic idea is to introduce a preprocessing step to scale up the vector wavelets generated in remeshing so that the inherent dependency of wavelets can be truly understood by the zerotree-like image compression algorithms. The weights used in the scaling process are carefully designed through thoroughly analyzing the distortions of wavelets at different refinement levels. Experimental results show that our proposed mesh coder significantly outperforms the state-of-the-art wavelet based 3D mesh compression scheme.

For network-based graphics applications, a key challenge is how to efficiently transmit complex three-dimensional (3D) models over bandwidth-limited communication channels such as wireless links. Most existing 3D mesh coding algorithms do not consider the view-dependent rendering issue, and therefore result in transmitting unnecessary portions of 3D mesh models which leads to the waste in precious network bandwidth. In this paper, we propose a novel view-dependent 3D model transmission scheme, where a 3D model is partitioned into a number of segments, each segment is then independently coded using the MPEG-4 3DMC coding algorithm, and finally only the visible segments are selected and delivered to the client. Such a view-based 3D model transmission is able to substantially save the transmission bandwidth and therefore has a significant impact on wireless network based graphics applications.

Digital 3D models represented by triangular meshes are now broadly available and they are becoming a new multimedia data type after sound, images and video. For many applications there is a need to convert such mesh models into spline surfaces. This paper describes an adaptive algorithm that automatically converts a triangular mesh into a T-spline surface based on optimization techniques. The key issues of the algorithm include the parameterization of input data points, the determination of topology of the T-spline surface, and the calculation of the least squares T-spline surface control points. Some examples are provided to demonstrate the efficiency of the algorithm.