Search

We present a simple yet effective method for constructing 3D self-supporting surfaces with planar quadrilateral (PQ) elements. Starting with a triangular discretization of a self-supporting surface, we first compute the principal curvatures and directions of each triangular face using a new discrete differential geometry approach, yielding more accurate results than existing methods. Then, we smooth the principal direction field to reduce the number of singularities. Next, we partition all faces into two groups in terms of principal curvature difference. For each face with small curvature difference, we compute a stretch matrix that turns the principal directions into a pair of conjugate directions. For the remaining triangular faces, we simply keep their smoothed principal directions. Finally, applying a mixed-integer programming solver to the mixed principal and conjugate direction field, we obtain a planar quadrilateral mesh. Experimental results show that our method is computationally efficient and can yield high-quality PQ meshes that well approximate the geometry of the input surfaces and maintain their self-supporting properties. Ministry of Education (MOE) Published version This work was partially supported by National Natural Science Foundation of China (62172257, 61802228), Singapore Ministry of Education (T2EP20220-0014), and the RIE2020 Industry Alignment Fund–Industry Collaboration Projects (IAF–ICP) Funding Initiative, as well as cash and in-kind contribution from the industrial partner, Rolls-Royce.

With the powerfulness of convolution neural networks (CNN), CNN based face reconstruction has recently shown promising performance in reconstructing detailed face shape from 2D face images. The success of CNN-based methods relies on a large number of labeled data. The state-of-the-art synthesizes such data using a coarse morphable face model, which however has difficulty to generate detailed photo-realistic images of faces (with wrinkles). This paper presents a novel face data generation method. Specifically, we render a large number of photo-realistic face images with different attributes based on inverse rendering. Furthermore, we construct a fine-detailed face image dataset by transferring different scales of details from one image to another. We also construct a large number of video-type adjacent frame pairs by simulating the distribution of real video data. With these nicely constructed datasets, we propose a coarse-to-fine learning framework consisting of three convolutional networks. The networks are trained for real-time detailed 3D face reconstruction from monocular video as well as from a single image. Extensive experimental results demonstrate that our framework can produce high-quality reconstruction but with much less computation time compared to the state-of-the-art. Moreover, our method is robust to pose, expression and lighting due to the diversity of data., Comment: Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

Most current anthropomorphic robotic hands can realize part of the human hand functions, particularly for object grasping. However, due to the complexity of the human hand, few current designs target at daily object manipulations, even for simple actions like rotating a pen. To tackle this problem, we introduce a gesture based framework, which adopts the widely-used 33 grasping gestures of Feix as the bases for hand design and implementation of manipulation. In the proposed framework, we first measure the motion ranges of human fingers for each gesture, and based on the results, we propose a simple yet dexterous robotic hand design with 13 degrees of actuation. Furthermore, we adopt a frame interpolation based method, in which we consider the base gestures as the key frames to represent a manipulation task, and use the simple linear interpolation strategy to accomplish the manipulation. To demonstrate the effectiveness of our framework, we define a three-level benchmark, which includes not only 62 test gestures from previous research, but also multiple complex and continuous actions. Experimental results on this benchmark validate the dexterity of the proposed design and our video is available in \url{https://drive.google.com/file/d/1wPtkd2P0zolYSBW7_3tVMUHrZEeXLXgD/view?usp=sharing}., Comment: 8 pages, 9 figures, 3 tables, accepted by IEEE Robotics and Automation Letters

We present a neural-based model for relighting a half-body portrait image by simply referring to another portrait image with the desired lighting condition. Rather than following classical inverse rendering methodology that involves estimating normals, albedo and environment maps, we implicitly encode the subject and lighting in a latent space, and use these latent codes to generate relighted images by neural rendering. A key technical innovation is the use of a novel overcomplete lighting representation, which facilitates lighting interpolation in the latent space, as well as helping regularize the self-organization of the lighting latent space during training. In addition, we propose a novel multiplicative neural render that more effectively combines the subject and lighting latent codes for rendering. We also created a large-scale photorealistic rendered relighting dataset for training, which allows our model to generalize well to real images. Extensive experiments demonstrate that our system not only outperforms existing methods for referral-based portrait relighting, but also has the capability generate sequences of relighted images via lighting rotations.

Motivated by how humans perceive geometry and color to recognize objects, we propose a Boundary Embedded Attentional Convolution (BEACon) network for point cloud instance segmentation. At the core of BEACon, we introduce the attentional weight in the convolution layer to adjust the neighboring features, with the weight being adapted to the relationship between geometry and color changes. As a result, BEACon makes use of both geometry and color information, takes instance boundary as an important feature, and thus learns a more discriminative feature representation in the neighborhood. Experimental results show that BEACon outperforms the state-of-the-art by a large margin. Ablation studies are also provided to prove the large benefit of incorporating both geometry and color into attention weight for instance segmentation. National Research Foundation (NRF) Accepted version This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

Corbel is a common category of decorative architectural geometry that has clear structure and aesthetic design. This paper presents a method for automatically generating a group of new corbel models from one selected by the user in the dataset. The method consists of offline learning and online generation. The offline learning trains two VAE models (2D CurveVAE and 3D VoxelVAE) for learning the feature representation of corbel parts. The online generation includes a generation algorithm by evolution that evolves to product new generation of models by crossing over and mutating features, and a feature-driven deformation that synthesizes 3D mesh representation of corbel models. By integrating these technical components, we develop a creative corbel modeling tool capable of generating new corbel models that are both “more of the same” and “surprising”, which is demonstrated by experiments. Ministry of Education (MOE) Accepted version This work is supported by the Ministry of Education, Singapore, under its MoE Tier-2 Grant (2017-T2-1-076).

3D instance segmentation plays a predominant role in environment perception of robotics and augmented reality. Many deep learning based methods have been presented recently for this task. These methods rely on either a detection branch to propose objects or a grouping step to assemble same-instance points. However, detection based methods do not ensure a consistent instance label for each point, while the grouping step requires parameter-tuning and is computationally expensive. In this paper, we introduce a novel framework to enable end-to-end instance segmentation without detection and a separate step of grouping. The core idea is to convert instance segmentation to a candidate assignment problem. At first, a set of instance candidates is sampled. Then we propose an assignment module for candidate assignment and a suppression module to eliminate redundant candidates. A mapping between instance labels and instance candidates is further sought to construct an instance grouping loss for the network training. Experimental results demonstrate that our method is more effective and efficient than previous approaches.

Human bodies exhibit various shapes for different identities or poses, but the body shape has certain similarities in structure and thus can be embedded in a low-dimensional space. This paper presents an autoencoder-like network architecture to learn disentangled shape and pose embedding specifically for the 3D human body. This is inspired by recent progress of deformation-based latent representation learning. To improve the reconstruction accuracy, we propose a hierarchical reconstruction pipeline for the disentangling process and construct a large dataset of human body models with consistent connectivity for the learning of the neural network. Our learned embedding can not only achieve superior reconstruction accuracy but also provide great flexibility in 3D human body generation via interpolation, bilinear interpolation, and latent space sampling. The results from extensive experiments demonstrate the powerfulness of our learned 3D human body embedding in various applications., Comment: This manuscript is accepted for publication in the IEEE Transactions on Visualization and Computer Graphics Journal (IEEE TVCG). The Code is available at https://github.com/Juyong/DHNN_BodyRepresentation

The problem of deforming an artist-drawn caricature according to a given normal face expression is of interest in applications such as social media, animation and entertainment. This paper presents a solution to the problem, with an emphasis on enhancing the ability to create desired expressions and meanwhile preserve the identity exaggeration style of the caricature, which imposes challenges due to the complicated nature of caricatures. The key of our solution is a novel method to model caricature expression, which extends traditional 3DMM representation to caricature domain. The method consists of shape modelling and texture generation for caricatures. Geometric optimization is developed to create identity-preserving blendshapes for reconstructing accurate and stable geometric shape, and a conditional generative adversarial network (cGAN) is designed for generating dynamic textures under target expressions. The combination of both shape and texture components makes the non-trivial expressions of a caricature be effectively defined by the extension of the popular 3DMM representation and a caricature can thus be flexibly deformed into arbitrary expressions with good results visually in both shape and color spaces. The experiments demonstrate the effectiveness of the proposed method., Comment: Accepted by the 28th ACM International Conference on Multimedia (ACM MM 2020)

Facial expression retargeting from humans to virtual characters is a useful technique in computer graphics and animation. Traditional methods use markers or blendshapes to construct a mapping between the human and avatar faces. However, these approaches require a tedious 3D modeling process, and the performance relies on the modelers' experience. In this paper, we propose a brand-new solution to this cross-domain expression transfer problem via nonlinear expression embedding and expression domain translation. We first build low-dimensional latent spaces for the human and avatar facial expressions with variational autoencoder. Then we construct correspondences between the two latent spaces guided by geometric and perceptual constraints. Specifically, we design geometric correspondences to reflect geometric matching and utilize a triplet data structure to express users' perceptual preference of avatar expressions. A user-friendly method is proposed to automatically generate triplets for a system allowing users to easily and efficiently annotate the correspondences. Using both geometric and perceptual correspondences, we trained a network for expression domain translation from human to avatar. Extensive experimental results and user studies demonstrate that even nonprofessional users can apply our method to generate high-quality facial expression retargeting results with less time and effort., IEEE Transactions on Visualization and Computer Graphics (TVCG), to appear

Reverse engineering of 3D industrial objects such as automobiles and electric appliances is typically performed by fitting B-spline surfaces to scanned point cloud data with a fairing term to ensure smoothness, which often smooths out sharp features. This paper proposes a radically different approach to constructing fair B-spline surfaces, which consists of fitting a surface without a fairing term to capture sharp edges, smoothing the normal field of the constructed surface with feature preservation, and reconstructing the B-spline surface from the smoothed normal field. The core of our method is an image processing based feature-preserving normal field fairing technique. This is inspired by the success of many recent research works on the use of normal field for reconstructing mesh models, and makes use of the impressive simplicity and effectiveness of bilateral-like filtering for image denoising. In particular, our approach adaptively partitions the B-spline surface into a set of segments such that each segment has approximately uniform parameterization, generates an image from each segment in the parameter space whose pixel values are the normal vectors of the surface, and then applies a bilateral filter in the parameter domain to fair the normal field. As a result, our approach inherits the advantages of image bilateral filtering techniques and is able to effectively smooth B-spline surfaces with feature preservation as demonstrated by various examples. MOE (Min. of Education, S’pore)

Reconstructing the shape of a 3D object from multi-view images under unknown, general illumination is a fundamental problem in computer vision. High quality reconstruction is usually challenging especially when fine detail is needed and the albedo of the object is non-uniform. This paper introduces vertex overall illumination vectors to model the illumination effect and presents a total variation (TV) based approach for recovering surface details using shading and multi-view stereo (MVS). Behind the approach are the two important observations: (1) the illumination over the surface of an object often appears to be piecewise smooth and (2) the recovery of surface orientation is not sufficient for reconstructing the surface, which was often overlooked previously. Thus we propose to use TV to regularize the overall illumination vectors and use visual hull to constrain partial vertices. The reconstruction is formulated as a constrained TV-minimization problem that simultaneously treats the shape and illumination vectors as unknowns. An augmented Lagrangian method is proposed to quickly solve the TV-minimization problem. As a result, our approach is robust, stable and is able to efficiently recover high-quality surface details even when starting with a coarse model obtained using MVS. These advantages are demonstrated by extensive experiments on the state-of-the-art MVS database, which includes challenging objects with varying albedo. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

A fundamental step to employ RGB-D cameras is to register color and depth images, whose misalignment may be caused by differences of camera poses and parameters, depth noises, etc. Previous methods mainly devote to more accurate camera calibration, which can only deal with misalignment that are parameterized with camera projection model. Other misalignment, which we call deviations, are more difficult to be measured and modeled. In this paper, we propose a method to model and remove RGB-D camera deviations. First, a specially-designed checkerboard with hollow squares is utilized to measure deviations and camera parameters, it takes advantage of the regularity of corner arrangements and can achieve high accuracy even with noisy depth inputs. Second, we propose a general deviation model to deal with irregular deviations that can not be handled by RGB-D camera projection model. Third, we introduce a registration method that incorporates the estimated deviation model to well register color and depth information. As demonstrated in the experiments, comparing with manufacturer’s calibration and some state-of-the-art algorithms, our approach produces significant better accuracy.

This work addresses the problem of 3D human shape reconstruction from point clouds. Considering that human shapes are of high dimensions and with large articulations, we adopt the state-of-the-art parametric human body model, SMPL, to reduce the dimension of learning space and generate smooth and valid reconstruction. However, SMPL parameters, especially pose parameters, are not easy to learn because of ambiguity and locality of the pose representation. Thus, we propose to incorporate skeleton awareness into the deep learning based regression of SMPL parameters for 3D human shape reconstruction. Our basic idea is to use the state-of-the-art technique PointNet++ to extract point features, and then map point features to skeleton joint features and finally to SMPL parameters for the reconstruction from point clouds. Particularly, we develop an end-to-end framework, where we propose a graph aggregation module to augment PointNet++ by extracting better point features, an attention module to better map unordered point features into ordered skeleton joint features, and a skeleton graph module to extract better joint features for SMPL parameter regression. The entire framework network is first trained in an end-to-end manner on synthesized dataset, and then online fine-tuned on unseen dataset with unsupervised loss to bridges gaps between training and testing. The experiments on multiple datasets show that our method is on par with the state-of-the-art solution.

While learning based depth estimation from images/videos has achieved substantial progress, there still exist intrinsic limitations. Supervised methods are limited by a small amount of ground truth or labeled data and unsupervised methods for monocular videos are mostly based on the static scene assumption, not performing well on real world scenarios with the presence of dynamic objects. In this paper, we propose a new learning based method consisting of DepthNet, PoseNet and Region Deformer Networks (RDN) to estimate depth from unconstrained monocular videos without ground truth supervision. The core contribution lies in RDN for proper handling of rigid and non-rigid motions of various objects such as rigidly moving cars and deformable humans. In particular, a deformation based motion representation is proposed to model individual object motion on 2D images. This representation enables our method to be applicable to diverse unconstrained monocular videos. Our method can not only achieve the state-of-the-art results on standard benchmarks KITTI and Cityscapes, but also show promising results on a crowded pedestrian tracking dataset, which demonstrates the effectiveness of the deformation based motion representation. Code and trained models are available at https://github.com/haofeixu/rdn4depth., IJCAI 2019

This paper presents subdivision‐based representations for both lighting and geometry in shape‐from‐shading. A very recent shading‐based method introduced a per‐vertex overall illumination model for surface reconstruction, which has advantage of conveniently handling complicated lighting condition and avoiding explicit estimation of visibility and varied albedo. However, due to its discrete nature, the per‐vertex overall illumination requires a large amount of memory and lacks intrinsic coherence. To overcome these problems, in this paper we propose to use classic subdivision to define the basic smooth lighting function and surface, and introduce additional independent variables into the subdivision to adaptively model sharp changes of illumination and geometry. Compared to previous works, the new model not only preserves the merits of the per‐vertex illumination model, but also greatly reduces the number of variables required in surface recovery and intrinsically regularizes the illumination vectors and the surface. These features make the new model very suitable for multi‐view stereo surface reconstruction under general, unknown illumination condition. Particularly, a variational surface reconstruction method built upon the subdivision representations for lighting and geometry is developed. The experiments on both synthetic and real‐world data sets have demonstrated that the proposed method can achieve memory efficiency and improve surface detail recovery. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

Human hand is the best sample for humanoid robotic hand and a nature grasping is the final target that most robotic hands are pursuing. Many prior researches had been done in virtual and real for simulation the human grasping. Unfortunately, there is no perfect solution to duplicate the nature grasping of human. The main difficulty comes from three points. 1. How to 3D modelling and fabricate the real hand. 2. How actuated the robotic hand as real hand. 3. How to grasp objects in different shapes like human hand. To deal with these three problems and further to provide a partial solution for duplicate human grasping, this paper introduces our method to solve these problems from robotic hand design, fabrication, actuation and grasping plan. Our modelling progress takes only around 12 minutes that include 10 minutes of 3D scanning of a real human hand and two minutes for changing the scanned model to an articulated model by running our algorithm. Our grasping plan is based on the sampled trajectory and easy to implement for grasping different objects. Followed these steps, a seven DOF robotic hand is created and tested in the experiments. Published version

It is often desirable for humanoid robots to have dexterous hands and perform human-like grasps. This requires a deliberate design of hands, in addition to a good actuation system and an efficient 3D vision system. In this paper, we present a simple method to produce a customized articulated robotic hand for a humanoid robot (eg., Nadine social robot). Our method acquires the 3D shape by 3D scanning, which can cost-effectively generate customized hands. Guided by the human hand anatomy and inspired by the progress of 3D printing technology, we model the structure of the robotic hand using standard shapes such as cuboids, spheres and cylinders, which are adapted to match the scanned shape and fabricated using 3D printing. Moreover, we choose flexible resin materials11https://formlabs.com/3d-printers/form-2/, which are supported by 3D printers, to make the fabricated hand flexible enough for natural deformation. As a result, our designed robotic hand has six degrees of freedom and together with a cable-driven actuation system can achieve the range of motion as a human hand. Experimental results demonstrated that our robotic hand is capable of grasping a variety of objects with different shapes. NRF (Natl Research Foundation, S’pore) Accepted version

This paper presents a system that aims to achieve autonomous grasping for micro-controller based humanoid robots such as the Inmoov robot [1]. The system consists of a visual sensor, a central controller and a manipulator. We modify the open sourced objection detection software YOLO (You Only Look Once) v2 [2] and associate it with the visual sensor to make the sensor be able to detect not only the category of the target object but also the location with the help of a depth camera. We also estimate the dimensions (i.e., the height and width) of the target based on the bounding box technique (Fig. 1). After that, we send the information to the central controller (a humanoid robot), which controls the manipulator (customised robotic hand) to grasp the object with the help of inverse kinematics theory. We conduct experiments to test our method with the Inmoov robot. The experiments show that our method is capable of detecting the object and driving the robotic hands to grasp the target object. Open image in new window Fig. 1. Autonomous grasping with real-time object detection

This paper presents a spatial-varying illumination model for shading-based depth refinement that based on a smooth Spherical Harmonics (SH) lighting field. The proposed lighting model is able to recover shading under challenging unknown lighting conditions, thus improving the quality of recovered surface detail. To avoid over-parameterization, local lighting coefficients are treated as a vector-valued function which is represented by subdivided surfaces using Catmull-Clark subdivision. We solve our lighting model utilizing a highly parallelized scheme that recovers lighting in a few milliseconds. A real-time shading-based depth recovery system is implemented with the integration of our proposed lighting model. We conduct quantitative and qualitative evaluations on both synthetic and real world datasets under challenging illumination. The experimental results show our method outperforms the state-of-the-art real-time shading-based depth refinement system. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

Leak detection is an important issue to be resolved of pipeline networks integrity and early leak detection and monitoring is highly demanded in the water network systems and energy transport field, since it can minimize environmental impact and economic loss. Nowadays, most leak detection methods require equipment to be used and/or installed which can be costly and often involve a lot of modelling and simulation. The goal of this work is to offer a solution for accurate in-pipe leak position detection by design of swimming robot (passively in-pipe swimming ball) that is equipped with an Inertial measurement Unit (IMU) board and a microphone required for leak localization. The data (Gyros, Accels) measured by IMU are used in calculating the position of the in-pipe swimbot and the deep learning classifier is applied using data recorded by the microphone to identify different acoustic characteristics of leak sounds.

Caricature is an art form that expresses subjects in abstract, simple and exaggerated view. While many caricatures are 2D images, this paper presents an algorithm for creating expressive 3D caricatures from 2D caricature images with a minimum of user interaction. The key idea of our approach is to introduce an intrinsic deformation representation that has a capacity of extrapolation enabling us to create a deformation space from standard face dataset, which maintains face constraints and meanwhile is sufficiently large for producing exaggerated face models. Built upon the proposed deformation representation, an optimization model is formulated to find the 3D caricature that captures the style of the 2D caricature image automatically. The experiments show that our approach has better capability in expressing caricatures than those fitting approaches directly using classical parametric face models such as 3DMM and FaceWareHouse. Moreover, our approach is based on standard face datasets and avoids constructing complicated 3D caricature training set, which provides great flexibility in real applications., Comment: Accepted to CVPR 2018

This paper considers episodic memory for companion–human interaction, aiming at improving user experience of interactions by endowing social companions with awareness of past experience. Due to noise and incomplete cues from natural language and speech in real-world interaction, accurate memory retrieval is very challenging and the noise resistance is important in practice. To improve the robustness of companion–human interaction, we propose a mixed-correlation analysis-based episodic memory (MCAEM) model, in which the correlations between memory elements are analyzed and then utilized for memory retrieval. In particular, the correlations are analyzed in three aspects: the relations between elements, importance of attributes and order of events. Based on the mixed-correlation analysis, a new similarity measure is constructed, which has substantially enhanced the noise resistance of memory retrieval. Experiments on a dataset collected from interaction in movies quantitatively evaluate the MCAEM model and compare it with prior work. Also, a user study is conducted to investigate the benefits of integrating the MCAEM model into social companions. The results demonstrate that the companions equipped with the MCAEM model not only have better retrieval performance, but also improve user experience in many aspects. NRF (Natl Research Foundation, S’pore) Accepted version

This paper presents a Quadtree Convolutional Neural Network (QCNN) for efficiently learning from image datasets representing sparse data such as handwriting, pen strokes, freehand sketches, etc. Instead of storing the sparse sketches in regular dense tensors, our method decomposes and represents the image as a linear quadtree that is only refined in the non-empty portions of the image. The actual image data corresponding to non-zero pixels is stored in the finest nodes of the quadtree. Convolution and pooling operations are restricted to the sparse pixels, leading to better efficiency in computation time as well as memory usage. Specifically, the computational and memory costs in QCNN grow linearly in the number of non-zero pixels, as opposed to traditional CNNs where the costs are quadratic in the number of pixels. This enables QCNN to learn from sparse images much faster and process high resolution images without the memory constraints faced by traditional CNNs. We study QCNN on four sparse image datasets for sketch classification and simplification tasks. The results show that QCNN can obtain comparable accuracy with large reduction in computational and memory costs.

When building robotic hands, researchers are always face with two main issues of how to make robotic hands look human-like and how to make robotic hands function like real hands. Most existing solutions solve these issues by manually modelling the robotic hand [10-18]. However, the design processes are long, and it is difficult to duplicate the geometry shape of a human hand. To solve these two issues, this paper presents a simple and effective method that combines 3D printing and digitization techniques to create a 3D printable cable-driven robotic hand from scanning a physical hand. The method involves segmenting the 3D scanned hand model, adding joints, and converting it into a 3D printable model. Comparing to other robotic solutions, our solution retains more than 90% geometry information of a human hand1, which is attained from 3D scanning. Our modelling progress takes around 15 minutes that include 10 minutes of 3D scanning and five minutes for changing the scanned model to an articulated model by running our algorithm. Compared to other articulated modelling solutions [19, 20], our solution is compatible with an actuation system which provides our robotic hand with the ability to mimic different gestures. We have also developed a way of representing hand skeletons based on the hand anthropometric. As a proof of concept, we demonstrate our robotic hand's performance in the grasping experiments. NRF (Natl Research Foundation, S’pore) Accepted version

How to endow the virtual characters with personalized behavior patterns remains a challenging problem. Instead of heuristically designing behaviors for certain personalities, this paper bridges the gap between personalities and behaviors using a medium concept, mood, to make the behaviors of the characters consistent enough to convey their personalities, while flexible enough to make appropriate response in various situations. We propose a personality-characterized mood dynamics model, in which the emotion weights are computed as a solution of a convex optimization problem that is constructed to make the overall mood approaches the personality after sufficient interactions. The convergence of the algorithm is demonstrated by numerical simulations. The implementation of the personality-characterized mood dynamics model enables an emotion-oriented virtual human, Sophie, to show personalized behaviors in the emotional interactions with users. Copyright © 2015 John Wiley & Sons, Ltd.

Considering that the existing depth recovery approaches have different limitations when applied to Kinect depth data, in this article, we propose to integrate their effective features including adaptive support region selection, reliable depth selection, and color guidance together under an optimization framework for Kinect depth recovery. In particular, we formulate our depth recovery as an energy minimization problem, which solves the depth hole filling and denoising simultaneously. The energy function consists of a fidelity term and a regularization term, which are designed according to the Kinect characteristics. Our framework inherits and improves the idea of guided filtering by incorporating structure information and prior knowledge of the Kinect noise model. Through analyzing the solution to the optimization framework, we also derive a local filtering version that provides an efficient and effective way of improving the existing filtering techniques. Quantitative evaluations on our developed synthesized dataset and experiments on real Kinect data show that the proposed method achieves superior performance in terms of recovery accuracy and visual quality.

The registration of multiple consumer-grade depth sensors is a challenging task due to noisy and systematic distortions in depth measurements. Most of the existing works heavily rely on large number of checkerboard observations for calibration and registration of multiple depth cameras, which is tedious and not flexible. In this paper, we propose a more practical method for conducting and maintaining registration of multi-depth sensors, via replacing checkerboards with everyday objects found in the scene, such as regular furniture. Particularly, high quality pre-scanned 3D shapes of standard furniture are used as calibration targets. We propose a unified framework that jointly computes the optimal extrinsic calibration and depth correction parameters. Experimental results show that our proposed method significantly outperforms state-of-the-art depth camera registration methods. NRF (Natl Research Foundation, S’pore) MOE (Min. of Education, S’pore)

Cooperative dual-crane lifting is an important but challenging process involved in heavy and critical lifting tasks. This paper considers the path planning for the cooperative dual-crane lifting. It aims to automatically generate optimal dual-crane lifting paths under multiple constraints, i.e., collision avoidance, coordination between the two cranes, and balance of the lifting target. Previous works often used oversimplified models for the dual-crane lifting system, the lifting environment, and the motion of the lifting target. They were thus limited to simple lifting cases and might even lead to unsafe paths in some cases. We develop a novel path planner for dual-crane lifting that can quickly produce optimized paths in complex 3-D environments. The planner has fully considered the kinematic structure of the lifting system. Therefore, it is able to robustly handle the nonlinear movement of the suspended target during lifting. The effectiveness and efficiency of the planner are enabled by three novel aspects: 1) a comprehensive and computationally efficient mathematical modeling of the lifting system; 2) a new multiobjective parallel genetic algorithm designed to solve the path planning problem; and 3) a new efficient approach to perform continuous collision detection for the dual-crane lifting target. The planner has been tested in complex industrial environments. The results show that the planner can generate dual-crane lifting paths that are easy for conductions and optimized in terms of costs for complex environments. Comparisons with two previous methods demonstrate the advantages of the planner, including safer paths, higher success rates, and the ability to handle general lifting cases. NRF (Natl Research Foundation, S’pore)

We present a variational method for subdivision surface reconstruction from a noisy dense mesh. A new set of subdivision rules with continuous sharpness control is introduced into Loop subdivision for better modeling subdivision surface features such as semi-sharp creases, creases, and corners. The key idea is to assign a sharpness value to each edge of the control mesh to continuously control the surface features. Based on the new subdivision rules, a variational model with L1 norm is formulated to find the control mesh and the corresponding sharpness values of the subdivision surface that best fits the input mesh. An iterative solver based on the augmented Lagrangian method and particle swarm optimization is used to solve the resulting non-linear, non-differentiable optimization problem. Our experimental results show that our method can handle meshes well with sharp/semi-sharp features and noise. MOE (Min. of Education, S’pore) Published version

This paper presents an example-guided, anthro-pometry-based modeling method for creating 3D human body models from users' input of partial anthropometric measurements with a given example dataset. Rather than directly forming a mapping between the partial measurements and the body model, we first estimate a set of chosen 30 measurements from the input based on the example-oriented measurement analysis. We then create an initial 3D model using the example-oriented radial basis function model that maps the set of 30 measurements to the body shape space and is established based on the given examples. We finally refine the 3D model by constrained optimization to create the target body model. Our method has several advantages: (1) the created model is guaranteed to match the input measurements and reflects the shape characteristics of examples; (2) the input requirement is modest, which makes it useful in practice; and (3) the information of both the measurements and examples is fully utilized. We demonstrate the effectiveness, accuracy, flexibility and extensibility of the method by various experimental evaluations and a Kinect-based body customization application.

The existing high-quality environment matting methods usually require the capturing of a few thousand sample images and spend a few hours in data acquisition. In this paper, a novel environment matting algorithm is proposed to capture and extract the environment matte data effectively and efficiently. First, the recently developed compressive sensing theory is incorporated to reformulate the environment matting problem and simplify the data acquisition process. Next, taking into account the special properties of light refraction and reflection effects of transparent objects, two advanced priors, group clustering and Gaussian priors, as well as other basic constraints are introduced during the matte data recovery process to combat with the limited image samples, suppress the effects of the measurement noise resulted from data acquisition, and faithfully recover the sparse environment matte data. Compared with most of the existing environment matting methods, our algorithm significantly simplifies and accelerates the environment matting extraction process while still achieving high-accurate composition results.

The problem of segmenting a foreground object out from its complex background is of great interest in image processing and computer vision. Many interactive segmentation algorithms such as graph cut have been successfully developed. In this paper, we present four technical components to improve graph cut based algorithms, which are combining both color and texture information for graph cut, including structure tensors in the graph cut model, incorporating active contours into the segmentation process, and using a ''softbrush'' tool to impose soft constraints to refine problematic boundaries. The integration of these components provides an interactive segmentation method that overcomes the difficulties of previous segmentation algorithms in handling images containing textures or low contrast boundaries and producing a smooth and accurate segmentation boundary. Experiments on various images from the Brodatz, Berkeley and MSRC data sets are conducted and the experimental results demonstrate the high effectiveness of the proposed method to a wide range of images.

Despite the great progress on interactive image segmentation, image co-segmentation, 2D and 3D segmentation, there is still no workable solution to the problem: given a set of calibrated or un-calibrated multi-view images (say, more than 40 images), by interactively cutting 3~4 images, can the foreground object of the rest images be quickly cutout automatically and accurately? In this paper, we propose a non-trivial engineering solution to this problem. Our basic idea is to integrate 3D segmentation with 2D segmentation so as to combine their advantages. Our proposed system iteratively performs 2D and 3D segmentation, where the 3D segmentation results are used to initialize 2D segmentation and ensure the silhouette consistency among different views and the 2D segmentation results are used to provide more accurate cues for the 3D segmentation. The experimental results show that the proposed system is able to generate highly accurate segmentation results, even for some challenging real-world multi-view image sequences, with a small amount of user input.

The state-of-the-art photometric stereo (PS) methods typically apply shading cues on each vertex and represent a vertex normal as a non-linear function of its neighboring vertices. Such vertex-based representation leads to huge computational cost and limits it from processing dense meshes. In this paper, we propose a PS based surface reconstruction using mesh face based representation. In particular, we propose to apply the shading cue on each mesh face instead of each vertex and optimize face normals instead of vertex normals. We develop a two-step approach to solve the surface recovery problem, where we first optimize face normals using shading cues and then update vertices by the optimized face normals. Experimental results show that, compared with the state-of-the-art, such a two-step approach is able to reduce the runtime significantly as well as handling much denser meshes.

In digital image editing, environment matting and compositing are fundamental and interesting operations that can capture and simulate the refraction and reflection effects of light from an environment. The state-of-the-art real-time environment matting and compositing method is short of flexibility, in the sense that it has to repeat the entire complex matte acquisition process if the distance between the object and the background is different from that in the acquisition stage, and also lacks accuracy, in the sense that it can only remove noises but not errors. In this paper, we introduce the concept of refractive vector and propose to use a refractive vector field as a new representation for environment matte. Such refractive vector field provides great flexibility for transparent-object environment matting and compositing. Particularly, with only one process of the matte acquisition and the refractive vector field extraction, we are able to composite the transparent object into an arbitrary background at any distance. Furthermore, we introduce a piecewise vector field fitting algorithm to simultaneously remove both noises and errors contained in the extracted matte data. Experimental results show that our method is less sensitive to artefacts and can generate perceptually good composition results for more general scenarios.

One weakness in the existing interactive image segmentation algorithms is the lack of more intelligent ways to understand the intention of user inputs. In this paper, we advocate the use of multiple intuitive user inputs to better reflect a user's intention. In particular, we propose a constrained random walks algorithm that facilitates the use of three types of user inputs: 1) foreground and background seed input, 2) soft constraint input, and 3) hard constraint input, as well as their combinations. The foreground and background seed input allows a user to draw strokes to specify foreground and background seeds. The soft constraint input allows a user to draw strokes to indicate the region that the boundary should pass through. The hard constraint input allows a user to specify the pixels that the boundary must align with. Our proposed method supports all three types of user inputs in one coherent computational framework consisting of a constrained random walks and a local editing algorithm, which allows more precise contour refinement. Experimental results on two benchmark data sets show that the proposed framework is highly effective and can quickly and accurately segment a wide variety of natural images with ease.

This paper considers the problem of interactively finding the cutting contour to extract components from a given mesh. Some existing methods support cuts of arbitrary shape but require careful and tedious input from the user. Others need little user input however they are sensitive to user input and need a postprocessing step to smooth the generated jaggy cutting contours. The popular geometric snake can be used to optimize the cutting contour, but it cannot deal with the topology change. In this paper, we propose a geodesic curvature flow based framework to overcome all these problems. Since in many cases the meaningful cutting contour on a 3D mesh is locally shortest in the sense of some weighted curve length, the geodesic curvature flow is an ideal tool for our problem. It evolves the cutting contour to the nearby local minimum. We should mention that the previous numerical scheme, discretized geodesic curvature flow (dGCF) is too slow and has not been applied to mesh segmentation. With a careful observation to dGCF, we devise here a fast computation scheme called fast geodesic curvature flow (FGCF), which only needs to solve a smaller and easier problem. The initial cutting contour is generated by a variant of random walks algorithm, which is very fast and gives reasonable cutting result with little user input. Experiment results on the benchmark mesh segmentation data set show that our proposed framework is robust to user input and capable of producing good results reflecting geometric features and human shape perception.

Texture mapping is an efficient and effective tool in computer graphics and animation. While computationally very cost-effective, texture mapping may produce non-realistic appearances of shapes in 3D environment, especially when viewing closely. To improve the realism of 3D modeling, bump mapping technique is developed to add details with the 3D models on top of texture mapping. Bump mapping, however, offers only simple and visual enhancement. Displacement mapping technique can further improve the localized detail of geometry. In this paper, Monge mapping technique is developed for detail and local shape modification of NURBS represented geometry in a 3D environment. Based on multiresolution and refinement schemes, Hierarchical NURBS (H-NURBS) is first investigated to design a mechanism for the purpose of carrying localized geometric information. Monge mapping on H-NURBS patch can be easily performed via simple cut-and-paste operation. Parametric control of the local shapes is developed to facilitate easier and better 3D local modeling.

For transmitting complex 3-D models over bandwidth-limited networks, efficient mesh coding and transmission are indispensable. The state-of-the-art 3-D mesh transmission system employs a wavelet-based progressive mesh coder, which converts an irregular mesh into a semi-regular mesh and directly applies the zerotree-like image coders to compress the wavelet vectors, and view-dependent transmission, which saves the transmission bandwidth through only delivering the visible portions of a mesh model. We propose methods to improve both progressive mesh coding and transmission based on thorough rate-distortion analysis. In particular, by noticing that the dependency among the wavelet coefficients generated in remeshing is not being considered in the existing approaches, we propose to introduce a preprocessing step to scale up the wavelets so that the inherent dependency of wavelets can be truly understood by the zerotree-like image compression algorithms. The weights used in the scaling process are carefully designed through thoroughly analyzing the distortions of wavelets at different refinement levels. For the transmission part, we propose to incorporate the illumination effects into the existing view-depend progressive mesh transmission system to further improve the performance. We develop a novel distortion model that considers both illumination distortion and geometry distortion. Based on our proposed distortion model, given the viewing and lighting parameters, we are able to optimally allocate bits among different segments in real time. Simulation results show significant improvements in both progressive compression and transmission.

Home healthcare becomes more and more important with the increased aging population. The advent of various low-cost sensing devices makes it tempting to develop low-cost, non-intrusive systems to monitor the variations of human being. In this paper, we describe a Kinect-based gait analysis and visualization system as a case study in this direction. The system uses depth images and skeleton captured by the Kinect to generate a BVH file recording the motion information, extracts features of gait for detecting abnormal gait, and customizes the 3D body model for personalized motion visualization. Compared to previous work in this area, the proposed system has advantages since it integrates gait classification and visualization which may bring new possibilities in healthcare. The experiments show that our proposed system achieves accurate gait classification as well as flexible personalized 3D visualization.

Bio structural and functional research and education is playing an increasingly important role in today's post-genome era. Protein geometry and shape modeling is thus a fundamental issue for protein visualization. The number of protein structures determined by X-ray crystallography or Nuclear Magnetic Resonance (NMR) is expanding in an exponential rate. Recent technology advancement has also made it possible for the determination of larger and more complicated proteins structure. A generic and automatic shape modeling for protein structures is therefore highly desired for effective and efficient protein visualization. We propose a bio-native geometric modeling technique in this paper for constructing protein secondary structure. Our emphasis is placed on the shape compatibility with the protein conformation property. Efforts are also made to handle smooth sweeping for complex protein structures. We describe as well a Virtual Reality (VR) application for protein structure education based on our Bio-native shape modeling and visualization techniques developed in this work.

Localizing, identifying and extracting human groups with consistent appearance jointly from a personal photo stream is an important problem and has wide applications. Inspired by recent advances in object detection, scene understanding and image cosegmentation, in this paper we explore explicit constraints to label and segment human objects rather than other non-human objects and “stuff”. We propose a novel soft human shape cue, which is initialized by color line poselet-based human part detection, further processed through a generalized geodesic distance transform, and refined finally with a joint bilateral filter. Such a high-level object cue is then integrated with other low-level unary and pairwise terms into a principled conditional random field framework, which can be efficiently solved by fast graph cut algorithms. We evaluate our algorithm over the FlickrMFC human dataset, and show that it achieves state-of-the-art performance for this challenging task.

This paper presents a subdivision-based vector graphics for image representation and creation. The graphics representation is a subdivision surface defined by a triangular mesh augmented with color attribute at vertices and feature attribute at edges. Special cubic B-splines are proposed to describe curvilinear features of an image. New subdivision rules are then designed accordingly, which are applied to the mesh and the color attribute to define the spatial distribution and piecewise-smoothly varying colors of the image. A sharpness factor is introduced to control the color transition across the curvilinear edges. In addition, an automatic algorithm is developed to convert a raster image into such a vector graphics representation. The algorithm first detects the curvilinear features of the image, then constructs a triangulation based on the curvilinear edges and feature attributes, and finally iteratively optimizes the vertex color attributes and updates the triangulation. Compared with existing vector-based image representations, the proposed representation and algorithm have the following advantages in addition to the common merits (such as editability and scalability): 1) they allow flexible mesh topology and handle images or objects with complicated boundaries or features effectively; 2) they are able to faithfully reconstruct curvilinear features, especially in modeling subtle shading effects around feature curves; and 3) they offer a simple way for the user to create images in a freehand style. The effectiveness of the proposed method has been demonstrated in experiments.

Collision detection plays a critical role in real-time applications such as game, simulation, and virtual reality. Collision avoidance is important in robotics path planning. Industrial safety, especially in construction and building, has a close linkage with the concept of contact avoidance. This chapter is interested in the investigation of collision detection problem using hardware graphics acceleration. Axis aligned bounding boxes (AABB) technique will be applied also for fast collision detection.

Considering the existing depth recovery approaches that have different limitations when applying to Kinect depth data, in this paper, we propose to integrate their effective features including adaptive support region selection, reliable depth selection and color guidance together under a unified framework for Kinect depth recovery. In particular, we formulate our depth recovery as an energy minimization problem, which solves the depth hole-filling and denoising simultaneously. The energy function consists of a fidelity term and a regularization term. The fidelity term takes into account the characteristics of Kinect data. The regularization term is designed to incorporate the joint bilateral filtering (JBF) kernel and the joint trilateral filtering (JTF) kernel so as to facilitate both depth hole-filling and denoising. Moreover, the JBF kernel is modified to incorporate the structure information. Both simulations on the benchmark Middlebury dataset and experiments on real Kinect data show that our proposed method achieves state-of-the-art performance in terms of recovery accuracy and visual quality.

Though quite a few image segmentation benchmark datasets have been constructed, there is no suitable benchmark for semantic image segmentation. In this paper, we construct a benchmark for such a purpose, where the ground-truths are generated by leveraging the existing fine granular ground-truths in Berkeley Segmentation Dataset (BSD) as well as using an interactive segmentation tool for new images. We also propose a percept-tree-based region merging strategy for dynamically adapting the ground-truth for evaluating test segmentation. Moreover, we propose a new evaluation metric that is easy to understand and compute, and does not require boundary matching. Experimental results show that, compared with BSD, the generated ground-truth dataset is more suitable for evaluating semantic image segmentation, and the conducted user study demonstrates that the proposed evaluation metric matches user ranking very well.

This paper considers the problem of automatically segmenting an image into a small number of regions that correspond to objects conveying semantics or high-level structure. Although such object-level segmentation usually requires additional high-level knowledge or learning process, we explore what low level cues can produce for this purpose. Our idea is to construct a feature vector for each pixel, which elaborately integrates spectral attributes, color Gaussian mixture models, and geodesic distance, such that it encodes global color and spatial cues as well as global structure information. Then, we formulate the Potts variational model in terms of the feature vectors to provide a variational image segmentation algorithm that is performed in the feature space. We also propose a heuristic approach to automatically select the number of segments. The use of feature attributes enables the Potts model to produce regions that are coherent in color and position, comply with global structures corresponding to objects or parts of objects and meanwhile maintain a smooth and accurate boundary. We demonstrate the effectiveness of our algorithm against the state-of-the-art with the data set from the famous Berkeley benchmark.

Given any image input by users, how to automatically cutout the object-of-interest is a challenging problem due to lack of information of the object-of-interest and the background. Saliency detection techniques are able to provide some rough information about object-of-interest since they highlight high-contrast or high attention regions or pixels. However, the generated saliency map is often noisy and directly applying it for segmentation often leads to erroneous results. Motivated by the recent progress on image co-segmentation and internet image retrieval techniques, in this paper, we propose to use the user input image for segmentation as a query image to Google Images and then employ the top returned Google images to build up the knowledge about the object-of-interest in the user input image. Particularly, we develop a lightweight algorithm to learn the knowledge of the object-of-interest in the retrieved images to enhance the saliency map of the input image. Then, the enhanced saliency map is used to initialize the graph-cut to extract the object-of-interest. Experiments with the Mcgill dataset and multiple challenge cases demonstrate the effectiveness of our method in terms of producing a clean cutout.