Search

Purpose Surgical gesture recognition has been an essential task for providing intraoperative context-aware assistance and scheduling clinical resources. However, previous methods present limitations in catching long-range temporal information, and many of them require additional sensors. To address these challenges, we propose a symmetric dilated network, namely SD-Net, to jointly recognize surgical gestures and assess surgical skill levels only using RGB surgical video sequences. Methods We utilize symmetric 1D temporal dilated convolution layers to hierarchically capture gesture clues under different receptive fields such that features in different time span can be aggregated. In addition, a self-attention network is bridged in the middle to calculate the global frame-to-frame relativity. Results We evaluate our method on a robotic suturing task from the JIGSAWS dataset. The gesture recognition task largely outperforms the state of the arts on the frame-wise accuracy up to $$\sim $$ ∼ 6 points and the F1@50 score $$\sim $$ ∼ 8 points. We also keep the 100% predicted accuracy for the skill assessment task using LOSO validation scheme. Conclusion The results indicate that our architecture is able to obtain representative surgical video features by extensively considering the spatial, temporal and relational context from raw video input. Furthermore, the better performance in multi-task learning implies that surgical skill assessment has a complementary effects to gesture recognition task.

Detail-and-geometry richness is essential to bas-relief modelling. However, existing image-based and model-based bas-relief modelling techniques commonly suffer from detail monotony or geometry loss. In this paper, we introduce a new bas-relief modelling framework for detail abundance with visual attention based mask generation and geometry preservation, which benefits from our two key contributions. For detail richness, we propose a novel semantic neural network of normal transfer to enrich the texture styles on bas-reliefs. For geometry preservation, we introduce a normal decomposition scheme based on Domain Transfer Recursive Filter (DTRF). Experimental results demonstrate that our approach is advantageous on producing bas-relief modellings with both fine details and geometry preservation.

We present a vision-based system for real-time pose tracking of the rigid object, it can not only estimate a single pose in six degrees of freedom (6DoF), but also suitable for recovering compound movements. The system is comprised of a monocular camera, and a series of 3D printed platonic solids with squared fiducial markers attached on each single face, which is easy to setup and extend, extra cameras are allowed to incorporate into the pipeline for meeting different requirements. The system realizes object tracking by estimating the pose of the fiducial platonic solid (FPS) which can be fixed onto the surface of the target object. Different sizes and shapes of the platonic solids are allowed to combine with each other to adapt to different application scenarios, this strategy provides enormous flexibility and applicability to our system. In order to track the motion of the fiducial platonic solid accurately, a robust algorithm that combines the fiducial constraint and the statistical constraint is introduced, which is able to handle illumination changes, motion blur and partial occlusion. We evaluate the performance of the proposed approach with qualitative and quantitative experiments, in addition, a couple of mixed reality (MR) applications are developed for demonstrating the effectiveness of the system.

Various investigations have shown that driver fatigue is the main cause of traffic accidents. Research on the use of computer vision techniques to detect signs of fatigue from facial actions, such as yawning, has demonstrated good potential. However, accurate and robust detection of yawning is difficult because of the complicated facial actions and expressions of drivers in the real driving environment. Several facial actions and expressions have the same mouth deformation as yawning. Thus, a novel approach to detecting yawning based on subtle facial action recognition is proposed in this study to alleviate the abovementioned problems. A 3D deep learning network with a low time sampling characteristic is proposed for subtle facial action recognition. This network uses 3D convolutional and bidirectional long short-term memory networks for spatiotemporal feature extraction and adopts SoftMax for classification. A keyframe selection algorithm is designed to select the most representative frame sequence from subtle facial actions. This algorithm rapidly eliminates redundant frames using image histograms with low computation cost and detects outliers by median absolute deviation. A series of experiments are also conducted on YawDD benchmark and self-collected datasets. Compared with several state-of-the-art methods, the proposed method has high yawning detection rates and can effectively distinguish yawning from similar facial actions.

In this paper, we make two contributions. The first is to propose a new keyframe extraction algorithm, which reduces the keyframe redundancy and reduces the motion sequence reconstruction error. Secondly, a new motion sequence reconstruction method is proposed, which further reduces the error of motion sequence reconstruction. Specifically, we treated the input motion sequence as curves, then the binomial fitting was extended to obtain the points where the slope changes dramatically in the vicinity. Then we took these points as inputs to obtain keyframes by density clustering. Finally, the motion curves were segmented by keyframes and the segmented curves were fitted by binomial formula again to obtain the binomial parameters for motion reconstruction. Experiments show that our methods outperform existing techniques, in terms of reconstruction error.

© 2020 The Authors. Computer Animation and Virtual Worlds published by John Wiley & Sons, Ltd. Human motion prediction is a fundamental problem in understanding human natural movements. This task is very challenging due to the complex human body constraints and diversity of action types. Due to the human body being a natural graph, graph convolutional network (GCN)-based models perform better than the traditional recurrent neural network (RNN)-based models on modeling the natural spatial and temporal dependencies lying in the motion data. In this paper, we develop the GCN-based models further by adding densely connected links to increase their feature utilizations and address oversmoothing problem. More specifically, the GCN block is used to learn the spatial relationships between the nodes and each feature map of the GCN block propagates directly to every following block as input rather than residual linked. In this way, the spatial dependency of human motion data is exploited more sufficiently and the features of different level of scale are fused more efficiently. Extensive experiments demonstrate our model achieving the state-of-the-art results on CMU dataset.

© 2020 The Authors. Computer Animation and Virtual Worlds published by John Wiley & Sons, Ltd. Sketch-based modeling aims to recover three-dimensional (3D) shape from two-dimensional line drawings. However, due to the sparsity and ambiguity of the sketch, it is extremely challenging for computers to interpret line drawings of physical objects. Most conventional systems are restricted to specific scenarios such as recovering for specific shapes, which are not conducive to generalize. Recent progress of deep learning methods have sparked new ideas for solving computer vision and pattern recognition issues. In this work, we present an end-to-end learning framework to predict 3D shape from line drawings. Our approach is based on a two-steps strategy, it converts the sketch image to its normal image, then recover the 3D shape subsequently. A differentiable renderer is proposed and incorporated into this framework, it allows the integration of the rendering pipeline with neural networks. Experimental results show our method outperforms the state-of-art, which demonstrates that our framework is able to cope with the challenges in single sketch-based 3D shape modeling.

© 2020 John Wiley & Sons, Ltd. In recent years, the topic of skeleton-based human action recognition has attracted significant attention from researchers and practitioners in graphics, vision, animation, and virtual environments. The most fundamental issue is how to learn an effective and accurate representation from spatiotemporal action sequences towards improved performance, and this article aims to address the aforementioned challenge. In particular, we design a novel method of hybrid features' extraction based on the construction of multistream networks and their organic fusion. First, we train a convolution neural networks (CNN) model to learn CNN-based features with the raw skeleton coordinates and their temporal differences serving as input signals. The attention mechanism is injected into the CNN model to weigh more effective and important information. Then, we employ long short-term memory (LSTM) to obtain long-term temporal features from action sequences. Finally, we generate the hybrid features by fusing the CNN and LSTM networks, and we classify action types with the hybrid features. The extensive experiments are performed on several large-scale publically available databases, and promising results demonstrate the efficacy and effectiveness of our proposed framework.

Automatic surgical gesture recognition is a prerequisite of intra-operative computer assistance and objective surgical skill assessment. Prior works either require additional sensors to collect kinematics data or have limitations on capturing temporal information from long and untrimmed surgical videos. To tackle these challenges, we propose a novel temporal convolutional architecture to automatically detect and segment surgical gestures with corresponding boundaries only using RGB videos. We devise our method with a symmetric dilation structure bridged by a self-attention module to encode and decode the long-term temporal patterns and establish the frame-to-frame relationship accordingly. We validate the effectiveness of our approach on a fundamental robotic suturing task from the JIGSAWS dataset. The experiment results demonstrate the ability of our method on capturing long-term frame dependencies, which largely outperform the state-of-the-art methods on the frame-wise accuracy up to ~6 points and the F1@50 score ~6 points., Comment: Accepted to MICCAI 2020

Sketch-based image retrieval is a fundamental computer vision problem. Instead of using hand-designed features to represent sketches and images, recent researches apply deep learning approaches combined with fine-grained matching to retrieve images with fine-grained details. Although these researches allow user to use hand-free sketches drawn for retrieving similar objects, the color matching is ignored which induces a low retrieval precision. To address this problem, we propose a single color sketch-based image retrieval (SCSBIR) approach using HSV color feature considering both shape matching and color matching in this paper. The SCSBIR problem is investigated using deep learning networks, in which deep features are used to represent color sketches and images. A novel ranking method considering both shape matching and color matching is also proposed. In addition, we build a SCSBIR dataset with color sketches and images, and train and test our method by using this dataset. The test results show that our method has a better retrieval performance. The research in this paper can not only promote its application in the commercial field, but also provide reference for the future research in this field.

We present a novel approach for automatically detecting and tracking facial landmarks across poses and expressions from in-the-wild monocular video data, e.g., YouTube videos and smartphone recordings. Our method does not require any calibration or manual adjustment for new individual input videos or actors. Firstly, we propose a method of robust 2D facial landmark detection across poses, by combining shape-face canonical-correlation analysis with a global supervised descent method. Since 2D regression-based methods are sensitive to unstable initialization, and the temporal and spatial coherence of videos is ignored, we utilize a coarse-todense 3D facial expression reconstruction method to refine the 2D landmarks. On one side, we employ an in-the-wild method to extract the coarse reconstruction result and its corresponding texture using the detected sparse facial landmarks, followed by robust pose, expression, and identity estimation. On the other side, to obtain dense reconstruction results, we give a face tracking flow method that corrects coarse reconstruction results and tracks weakly textured areas; this is used to iteratively update the coarse face model. Finally, a dense reconstruction result is estimated after it converges. Extensive experiments on a variety of video sequences recorded by ourselves or downloaded from YouTube show the results of facial landmark detection and tracking under various lighting conditions, for various head poses and facial expressions. The overall performance and a comparison with state-of-art methods demonstrate the robustness and effectiveness of our method.

We present a unified learning framework for recovering both 3D mesh and camera pose of the object from a single image. Our approach learns to recover outer shape and surface geometric details of the mesh without relying on 3D supervision. We adopt multi-view normal maps as the 2D supervision so that the silhouette and geometric details information can be transferred to neural network. A normal mismatch based objective function is introduced to train the network, and the camera pose is parameterized into the objective, it integrates pose estimation with the mesh reconstruction in a same optimization procedure. We demonstrate the abilities of the proposed approach in generating 3D mesh and estimating camera pose with qualitative and quantitative experiments.

© 2019, The Author(s). Human motion prediction is a challenging problem due to the complicated human body constraints and high-dimensional dynamics. Recent deep learning approaches adopt RNN, CNN or fully connected networks to learn the motion features which do not fully exploit the hierarchical structure of human anatomy. To address this problem, we propose a convolutional hierarchical autoencoder model for motion prediction with a novel encoder which incorporates 1D convolutional layers and hierarchical topology. The new network is more efficient compared to the existing deep learning models with respect to size and speed. We train the generic model on Human3.6M and CMU benchmark and conduct extensive experiments. The qualitative and quantitative results show that our model outperforms the state-of-the-art methods in both short-term prediction and long-term prediction.

We propose a novel fine-grained color sketch-based image retrieval (CSBIR) approach. The CSBIR problem is investigated for the first time using deep learning networks, in which deep features are used to represent color sketches and images. A novel ranking method considering both shape matching and color matching is also proposed. In addition, we build a CSBIR dataset with color sketches and images to train and test our method. The results show that our method has better retrieval performance.

Facial expression transfer has been actively researched in the past few years. Existing methods either suffer from depth ambiguity or require special hardware. We present a novel marker-less, real-time facial transfer method that requires only a single video camera. We develop a robust model, which is adaptive to user-specific facial data. It computes expression variances in real time and rapidly transfers them onto a target character either from images or videos. Our method can be applied to videos without prior camera calibration and focal adjustment. It enables realistic online facial expression editing and performance transferring in many scenarios such as video conference, news broadcasting, lip-syncing for song performances and so on. With low computational cost and hardware requirement, our method tracks a single user at an average of 38fps and runs smoothly even in web browsers. Copyright © 2016 John Wiley & Sons, Ltd.

Multiple visual features have been proposed and used in 3-dimensional (3D) model retrieval in recent years. Since each visual feature reflects a unique characteristic about the model, they have unequal discriminative power with respect to a specific category of 3D model, and they are complementary to each other in model representation. Thus, it would be beneficial to combine multiple visual features together in 3D model retrieval. In light of this, we propose an efficient Semi-supervised Multiple Feature Fusion (SMFF) method for view-based 3D model retrieval in this paper. Specifically, We first extract multiple visual features to describe both the local and global appearance characteristics of multiple 2D projected images that are generated from 3D models. Then, SMFF is adopted to learn a more compact and discriminative low-dimensional feature representation via multiple feature fusion using both the labeled and unlabeled 3D models. Once the low-dimensional features have been learned, many existing methods such as SVM and KNN can be used in the subsequent retrieval phase. Moreover, an out-of-sample extension of SMFF is provided to calculate the low-dimensional features for the newly added 3D models in linear time. Experiments on two public 3D model datasets demonstrate that using such a learned feature representation can significantly improve the performance of 3D model retrieval and the proposed method outperforms the other competitors.

© 2018 ACM. The techniques for photographic style transfer have been researched for a long time, which explores effective ways to transfer the style features of a reference photo onto another content photograph. Recent works based on convolutional neural networks present an effective solution for style transfer, especially for paintings. The artistic style transformation results are visually appealing, however, the photorealism is lost because of content-mismatching and distortions even when both input images are photographic. To tackle this challenge, this paper introduces a similarity loss function and a refinement method into the style transfer network. The similarity loss function can solve the content-mismatching problem, however, the distortion and noise artefacts may still exist in the stylized results due to the content-style trade-off. Hence, we add a post-processing refinement step to reduce the artefacts. The robustness and effectiveness of our approach has been evaluated through extensive experiments which show that our method can obtain finer content details and less artefacts than state-of-the-art methods, and transfer style faithfully. In addition, our approach is capable of processing photographic style transfer in almost real-time, which makes it a potential solution for video style transfer.

With the increasing amount of videos recorded using 2D mobile cameras, the technique for recovering the 3D dynamic facial models from these monocular videos has become a necessity for many image and video editing applications. While methods based parametric 3D facial models can reconstruct the 3D shape in dynamic environment, large structural changes are ignored. Structure-from-motion methods can reconstruct these changes but assume the object to be static. To address this problem we present a novel method for realtime dynamic 3D facial tracking and reconstruction from videos captured in uncontrolled environments. Our method can track the deforming facial geometry and reconstruct external objects that protrude from the face such as glasses and hair. It also allows users to move around, perform facial expressions freely without degrading the reconstruction quality.

Puppetry has been a popular art form for many centuries in different cultures, which becomes a valuable and fascinating heritage assert. Traditional Chinese marionette art with over 2000years history is one of the most representative forms offering a mixture of stage performance of singing, dancing, music, poetry, opera, story narrative and action. Apart from a set of string rules, which controls the dynamics, head carving skill is another important pillar in this art form.

© 2017 The Author(s)Non-isometric surface registration, aiming to align two surfaces with different sizes and details, has been widely used in computer animation industry. Various existing surface registration approaches have been proposed for accurate template fitting; nevertheless, two challenges remain. One is how to avoid the mesh distortion and fold over of surfaces during transformation. The other is how to reduce the amount of landmarks that have to be specified manually. To tackle these challenges simultaneously, we propose a consistent as-similar-as-possible (CASAP) surface registration approach. With a novel defined energy, it not only achieves the consistent discretization for the surfaces to produce accurate result, but also requires a small number of landmarks with little user effort only. Besides, CASAP is constrained as-similar-as-possible so that angles of triangle meshes are preserved and local scales are allowed to change. Extensive experimental results have demonstrated the effectiveness of CASAP in comparison with the state-of-the-art approaches.

Face alignment and tracking play important roles in facial performance capture. Existing data-driven methods for monocular videos suffer from large variations of\ud pose and expression. In this paper, we propose an efficient and robust method for this task by introducing a novel supervised coordinate descent method with 3D bilinear representation. Instead of learning the mapping between the whole parameters and image features directly with a cascaded regression framework in current methods,\ud we learn individual sets of parameters mappings separately step by step by a coordinate descent mean. Because different parameters make different contributions to the displacement of facial landmarks, our method is more discriminative to current whole-parameter cascaded regression methods. Benefiting from a 3D bilinear model learned from public databases, the proposed method can handle the head pose changes and extreme expressions out of plane better than other 2D-based methods. We present the reliable result of face tracking under various head poses and facial expressions on challenging video sequences collected online. The experimental results show that our method outperforms state-of-art data-driven methods.

Face alignment is an essential task for facial performance capture and expression analysis. As a complex nonlinear problem in computer vision, face alignment across poses is still not studied well. Although the state-of-the-art Supervised Descent Method (SDM) has shown good performance, it learns conflict descent direction in the whole complex space due to various poses and expressions. Global SDM has been presented to deal with this case by domain partition in feature and shape PCA spaces for face tracking and pose estimation. However, it is not suitable for the face alignment problem due to unknown ground truth shapes. In this paper we propose a sign-correlation subspace method for the domain partition of global SDM. In our method only one reduced low dimensional subspace is enough for domain partition, thus adjusting the global SDM efficiently for face alignment. Unlike previous methods, we analyze the sign correlation between features and shapes, and project both of them into a mutual sign-correlation subspace. Each pair of projected shape and feature keep sign consistent in each dimension of the subspace, so that each hyperoctant holds the condition that one general descent exists. Then a set of general descent directions are learned from the samples in different hyperoctants. Our sign-correlation partition method is validated in the public face datasets, which includes a range of poses. It indicates that our methods can reveal their latent relationships to poses. The comparison with state-of-the-art methods for face alignment demonstrates that our method outperforms them especially in uncontrolled conditions with various poses, while keeping comparable speed.

Using motion capture to create naturally looking motion sequences for virtual character animation has become a standard procedure in the games and visual effects industry. With the fast growth of motion data, the task of automatically annotating new motions is gaining an importance. In this paper, we present a novel statistic feature to represent each motion according to the pre-labeled categories of key-poses. A probabilistic model is trained with semi-supervised learning of the Gaussian mixture model (GMM). Each pose in a given motion could then be described by a feature vector of a series of probabilities by GMM. A motion feature descriptor is proposed based on the statistics of all pose features. The experimental results and comparison with existing work show that our method performs more accurately and efficiently in motion retrieval and annotation.

With the explosive growth of motion capture data, it becomes very imperative in animation production to have an efficient search engine to retrieve motions from large motion repository. However, because of the high dimension of data space and complexity of matching methods, most of the existing approaches cannot return the result in real time. This paper proposes a high level semantic feature in a low dimensional space to represent the essential characteristic of different motion classes. On the basis of the statistic training of Gauss Mixture Model, this feature can effectively achieve motion matching on both global clip level and local frame level. Experiment results show that our approach can retrieve similar motions with rankings from large motion database in real-time and also can make motion annotation automatically on the fly. Copyright © 2013 John Wiley & Sons, Ltd.

Texture mapping has been widely used in image processing and graphics to enhance the realism of CG scenes. However to perfectly match the feature points of a 3D model with the corresponding pixels in texture images, the parameterisation which maps a 3D mesh to the texture space must satisfy the positional constraints. Despite numerous research efforts, the construction of a mathematically robust foldover-free parameterisation subject to internal constraints is still a remaining issue. In this paper, we address this challenge by developing a two-step parameterisation method. First, we produce an initial parameterisation with a method traditionally used to solve structural engineering problems, called the bar-network. We then derive a mathematical foldover-free condition, which is incorporated into a Radial Basis Function based scheme. This method is therefore able to guarantee that the resulting parameterization meets the hard constraints without foldovers. Keywords-Foldover; constrained texture mapping; parameterization

Human motion capture techniques (MOCAP) are widely applied in many areas such as computer vision, computer animation, digital effect and virtual reality. Even with professional MOCAP system, the acquired motion data still always contains noise and outliers, which highlights the need for the essential motion refinement methods. In recent years, many approaches for motion refinement have been developed, including signal processing based methods, sparse coding based methods and low-rank matrix completion based methods. However, motion refinement is still a challenging task due to the complexity and diversity of human motion. In this paper, we propose a data-driven-based human motion refinement approach by exploiting the structural sparsity and spatio-temporal information embedded in motion data. First of all, a human partial model is applied to replace the entire pose model for a better feature representation to exploit the abundant local body posture. Then, a dictionary learning which is for special task of motion refinement is designed and applied in parallel. Meanwhile, the objective function is derived by taking the statistical and locality property of motion data into account. Compared with several state-of-art motion refine methods, the experimental result demonstrates that our approach outperforms the competitors.

Estimation of 3D body shapes from dressed-human photos is an important but challenging problem in virtual fitting. We propose a novel automatic framework to efficiently estimate 3D body shapes under clothes. We construct a database of 3D naked and dressed body pairs, based on which we learn how to predict 3D positions of body landmarks (which further constrain a parametric human body model) automatically according to dressed-human silhouettes. Critical vertices are selected on 3D registered human bodies as landmarks to represent body shapes, so as to avoid the time-consuming vertices correspondences finding process for parametric body reconstruction. Our method can estimate 3D body shapes from dressed-human silhouettes within 4 seconds, while the fastest method reported previously need 1 minute. In addition, our estimation error is within the size tolerance for clothing industry. We dress 6042 naked bodies with 3 sets of common clothes by physically based cloth simulation technique. To the best of our knowledge, We are the first to construct such a database containing 3D naked and dressed body pairs and our database may contribute to the areas of human body shapes estimation and cloth simulation.

Motion capture (MOCAP) is an important technique that is widely used in many areas such as computer animation, film industry, physical training and so on. Even with professional MOCAP system, the missing marker problems always occur. Motion refinement is an essential preprocessing step for MOCAP data based applications. Although many existing approaches for motion refinement have been developed, it is still a challenging task due to the complexity and diversity of human motion. A data driven based motion refinement method is proposed in this paper, which modifies the traditional sparse coding process for special task of motion recovery from missing parts. Meanwhile, the objective function is derived by taking both statistical and kinematical property of motion data into account. Poselet model and moving window grouping are applied in the proposed method to achieve a fine-grained feature representation, which preserves the embedded spatial-temporal kinematic information. 5 motion dictionaries are learnt for each kind of poselet from training data in parallel. The motion refine problem is finally solved as an e1-minimization problem. Compared with several state-of-art motion refine methods, the experimental result shows that our approach outperforms the competitors.

Human motion retrieval plays an important role in many motion data based applications. In the past, many researchers tended to use a single type of visual feature as data representation. Because different visual feature describes different aspects about motion data, and they have dissimilar discriminative power with respect to one particular class of human motion, it led to poor retrieval performance. Thus, it would be beneficial to combine multiple visual features together for motion data representation. In this article, we present an Adaptive Multi-view Feature Selection (AMFS) method for human motion retrieval. Specifically, we first use a local linear regression model to automatically learn multiple view-based Laplacian graphs for preserving the local geometric structure of motion data. Then, these graphs are combined together with a non-negative view-weight vector to exploit the complementary information between different features. Finally, in order to discard the redundant and irrelevant feature components from the original high-dimensional feature representation, we formulate the objective function of AMFS as a general trace ratio optimization problem, and design an effective algorithm to solve the corresponding optimization problem. Extensive experiments on two public human motion database, i.e., HDM05 and MSR Action3D, demonstrate the effectiveness of the proposed AMFS over the state-of-art methods for motion data retrieval. The scalability with large motion dataset, and insensitivity with the algorithm parameters, make our method can be widely used in real-world applications. Display Omitted An Adaptive Multi-view Feature Selection (AMFS) algorithm is proposed to fuse multiple features formotion data retrieval.The local regression model isused to learn a datum-adaptive graph for each feature to preserve local structure information.The selection matrix is learnt from all local graphs by exploiting the complementary information between different features.An efficient iterative optimization approach is designed to solve objective function represented in trace ratio form.

In this paper, a novel feature representation to multimodal face recognition is proposed, which possesses three properties: completeness, robustness and compactness. This feature descriptor allows all information of an object to be reproduced and its representation is invariant to rigid motion. In order to effectively take advantage of the proposed feature descriptor, we amend our previous ND-PCA scheme with multidirectional decomposition technique, and provide the estimation of the upper bound error of the amended classifier. It is proved to be linear optimal compared to other linear classifiers. To investigate the numerical performance of the presented feature descriptor, we apply it to both multiple modal and single modal samples, and the revised ND-PCA classifier is performed on the resulting feature representations. The experiments of verification and identification are carried out on two different gallery-probe face databases in order for the results to be evaluated by ROC and CMC curves independently.

Defects by the linear blend skinning (LBS) approach, such as "collapsing joints" and "candy-wrapper artefacts", are visually unpleasant when skinning animated characters and they are tedious to rectify. The cause of the defects is that the inherent non-linear skinning problem is handled linearly. Despite its shortcomings, this linear technique has been incorporated into the industrial production pipeline as a standard approach in character rigging. In this paper, we present a new skinning technique aiming at remedying the defects of the LBS approach without much disturbance of the current production pipeline. Our technique can be implemented by reusing the LBS as its sub-module offering an important advantage over other existing non-linear skinning techniques in terms of its simplicity and robustness. In our approach, we represent the skinning process as a continuous temporal deformation problem, where the skin deforms step-by-step to its final configuration. As a consequence, we reach a smooth and continuous solution without noticeable defects. Furthermore, our approach can be accelerated by caching the previous frames to boost the efficiency.

Human motion denoising is an indispensable step of data preprocessing for many motion data based applications. In this paper, we propose a data-driven based human motion denoising method that sparsely selects the most correlated subset of motion bases for clean motion reconstruction. Meanwhile, it takes the statistic property of two common noises, i.e., Gaussian noise and outliers, into account in deriving the objective functions. In particular, our method firstly divides each human pose into five partitions termed as poselets to gain a much fine-grained pose representation. Then, these poselets are reorganized into multiple overlapped poselet groups using a lagged window moving across the entire motion sequence to preserve the embedded spatial-temporal motion patterns. Afterward, five compacted and representative motion dictionaries are constructed in parallel by means of fast K-SVD in the training phase; they are used to remove the noise and outliers from noisy motion sequences in the testing phase by solving ?1-minimization problems. Extensive experiments show that our method outperforms its competitors. More importantly, compared with other data-driven based method, our method does not need to specifically choose the training data, it can be more easily applied to real-world applications. HighlightsA fine-grained pose representation model is proposed to boost the performance.We present a data-driven based motion denoising method by solving ?1-minimization problems.The proposed model selects the most correlated motion bases for motion reconstruction.Our method does not need to choose the training data and can be implemented in parallel mode.

Motion capture is an important technique with a wide range of applications in areas such as computer vision, computer animation, film production, and medical rehabilitation. Even with the professional motion capture systems, the acquired raw data mostly contain inevitable noises and outliers. To denoise the data, numerous methods have been developed, while this problem still remains a challenge due to the high complexity of human motion and the diversity of real-life situations. In this paper, we propose a data-driven-based robust human motion denoising approach by mining the spatial-temporal patterns and the structural sparsity embedded in motion data. We first replace the regularly used entire pose model with a much fine-grained partlet model as feature representation to exploit the abundant local body part posture and movement similarities. Then, a robust dictionary learning algorithm is proposed to learn multiple compact and representative motion dictionaries from the training data in parallel. Finally, we reformulate the human motion denoising problem as a robust structured sparse coding problem in which both the noise distribution information and the temporal smoothness property of human motion have been jointly taken into account. Compared with several state-of-the-art motion denoising methods on both the synthetic and real noisy motion data, our method consistently yields better performance than its counterparts. The outputs of our approach are much more stable than that of the others. In addition, it is much easier to setup the training dataset of our method than that of the other data-driven-based methods.

**************** **************** **************** **************** **************** ************* The skeleton driven skinning technique is still the most popular method for animating deformable human and creature characters. Albeit an industry de facto due to its computational performance and intuitiveness, it suffers from problems like collapsing elbow and candy wrapper joint. To remedy these problems, one needs to formulate the non-linear relationship between the skeleton and the skin shape of a character properly, which however proves mathematically very challenging. Placing additional joints where the skin bends increases the sampling rate and is an ad hoc way of approximating this non-linear relationship. In this paper, we propose a method that is able to accommodate the inherent non-linear relationships between the movement of the skeleton and the skin shape. We use the so-called curve skeletons along with the joint-based skeletons to animate the skin shape. Since the deformation follows the tangent of the curve skeleton and also due to higher sampling rates received from the curve points, collapsing skin and other undesirable skin deformation problems are avoided. The curve skeleton retains the advantages of the current skeleton driven skinning. It is easy to use and allows full control over the animation process. As a further enhancement, it is also fairly simple to build realistic muscle and fat bulge effect. A practical implementation in the form of a Maya plug-in is created to demonstrate the viability of the technique. Copyright # 2006 John Wiley & Sons, Ltd. Received: 10 April 2006; Revised: 2 May 2006; Accepted: 10 May 2006

Even though there is an explosive growth of motion capture data, there is still a lack of efficient and reliable methods to automatically annotate all the motions in a database. Moreover, because of the popularity of mocap devices in home entertainment systems, real-time human motion annotation or recognition becomes more and more imperative. This paper presents a new motion annotation method that achieves both the aforementioned two targets at the same time. It uses a probabilistic pose feature based on the Gaussian Mixture Model to represent each pose. After training a clustered pose feature model, a motion clip could be represented as an action string. Then, a dynamic programming-based string matching method is introduced to compare the differences between action strings. Finally, in order to achieve the real-time target, we construct a hierarchical action string structure to quickly label each given action string. The experimental results demonstrate the efficacy and efficiency of our method. Copyright © 2014 John Wiley & Sons, Ltd.

In this paper, we present an integrated framework of human motion retrieval based on freehand sketch. With some simple rules, the user can acquire a desired motion by sketching several key postures. To retrieve efficiently and accurately by sketch, the 3D postures are projected onto several 2D planes. The limb direction feature is proposed to represent the input sketch and the projected-postures. Furthermore, a novel index structure based on k-d tree is constructed to index the motions in the database, which speeds up the retrieval process. With our posture-by-posture retrieval algorithm, a continuous motion can be got directly or generated by using a pre-computed graph structure. What's more, our system provides an intuitive user interface. The experimental results demonstrate the effectiveness of our method. Copyright © 2014 John Wiley & Sons, Ltd.

Animation of models often introduces distortions to their parameterisation, as these are typically optimised for a single frame. The net effect is that under deformation, the mapped features, i.e. UV texture maps, bump maps or displacement maps, may appear to stretch or scale in an undesirable way. Ideally, what we would like is for the appearance of such features to remain feasible given any underlying deformation. In this paper we introduce a real-time technique that reduces such distortions based on a distortion control (rigidity) map. In two versions of our proposed technique, the parameter space is warped in either an axis or a non-axis aligned manner based on the minimisation of a non-linear distortion metric. This in turn is solved using a highly optimised hybrid CPU-GPU strategy. The result is real-time dynamic content-aware texturing that reduces distortions in a controlled way. The technique can be applied to reduce distortions in a variety of scenarios, including reusing a low geometric complexity animated sequence with a multitude of detail maps, dynamic procedurally defined features mapped on deformable geometry and animation authoring previews on texture-mapped models. © 2013 ACM.

The animation of realistic characters necessitates the construction of complicated anatomical structures such as muscles, which allow subtle shape variation of the character's outer surface to be displayed believably. Unfortunately, despite numerous efforts, the modelling of muscle structures is still left for an animator who has to painstakingly build up piece by piece, making it a very tedious process. What is even more frustrating is the animator has to build the same muscle structure for every new character. We propose a muscle retargeting technique to help an animator to automatically construct a muscle structure by reusing an already built and tested model (the template model). Our method defines a spatial transfer between the template model and a new model based on the skin surface and the rigging structure. To ensure that the retargeted muscle is tightly packed inside a new character, we define a novel spatial optimization based on spherical parameterization. Our method requires no manual input, meaning that an animator does not require anatomical knowledge to create realistic accurate musculature models.

Driver fatigue and inattention have long been recognized as one of the main contributing factors in traffic accidents. Therefore, the development of intelligent driver assistance systems, which provides automatic monitoring of driver's vigilance, is an urgent and challenging task. This paper presents a novel system for video-based driving behavior recognition. The fundamental idea is to monitor driver's hand movements and to use these as predictors for safe/unsafe driving behavior. In comparison to previous work, the proposed method utilizes hierarchical classification and treats driving behavior in terms of a spatio-temporal reference framework as opposed to a static image. The approach was verified using the Southeast University Driving-Posture Dataset, a dataset comprised of video clips covering aspects of driving such as: normal driving, responding to a cell phone call, eating and smoking. After pre-processing for illumination variations and motion sequence segmentation, eight classes of behavior were identified. The overall prediction accuracy obtained using the proposed approach was [Formula: see text] when using a hierarchical classification approach. The proposed approach was able to clearly identify two dangerous driving behaviors, Responding to a cellphone call and Eating, with recognition rates of 92.39% and 92.29% respectively.

Bipedal walking is not fully understood. Motion generated from methods employed in robotics literature is stiff and is not nearly as energy efficient as what we observe in nature. In this paper, we propose validity conditions for motion adaptation from biological principles in terms of the topology of the dynamic system. This allows us to provide a closed-form solution to the problem of motion adaptation to environmental perturbations. We define both global and local controllers that improve structural and state stability, respectively. Global control is achieved by coupling the dynamic system with a neural oscillator, which preserves the periodic structure of the motion primitive and ensures stability by entrainment. A group action derived from Lie group symmetry is introduced as a local control that transforms the underlying state space while preserving certain motor invariants. We verify our method by evaluating the stability and energy consumption of a synthetic passive dynamic walker and compare this with motion data of a real walker. We also demonstrate that our method can be applied to a variety of systems.

Texture mapping has long been used in computer graphics to \ud enhance the realism of virtual scenes. However, to match the 3D model feature points with the corresponding pixels in a texture image, surface parameterization must satisfy specific positional constraints. However, despite numerous \ud research efforts, the construction of a mathematically robust, foldover‐free parameterization that is subject to \ud positional constraints continues to be a challenge. In the \ud present paper, this foldover problem is addressed by developing radial basis function (RBF) based reparameterization. Given initial 2D embedding of a 3D \ud surface, the proposed method can reparameterize 2D embedding into a foldover ‐free 2D mesh, satisfying a set \ud of user‐specified constraint points. In addition, this approach is mesh‐free. Therefore, generating smooth texture \ud mapping results is possible without extra smoothing optimization.

Skin movement modeling is a crucial element for realistic animation of both human and creature characters. In this paper, we are primarily concerned with a special skin movement effect, skin sliding. Although physical properties, such as skin elasticity, are usually regarded important in physical simulation, it often leads to an increasing computational load. A complete physical solution will also limit the animator's ability to fine-tune the visual effects. In this paper we present a physically based skin sliding modeling method borrowing an idea from structural mechanics called bar networks. The advantage of using a bar network system in simulating the physical properties of skin surfaces is that only a set of sparse linear equations need to be solved, making the method much faster than a simulation based technique. As only one additional step is added to the animation pipeline to restore the skin elastic property, this algorithm is compatible with all existing skin deformation method such as smooth skinning, muscle base skin deformation, cluster and Free Form Deformations, and it therefore does not disturb the existing production pipeline.

Optimization techniques have proven to be a powerful approach for generating new motions. In this paper, we present a physically based optimization method to synthesize motions by using motion capture data as input. We assume that the captured motion data is physically plausible. We start by defining and estimating the physical properties of human characters. The procedure of motion synthesis is from coarse to fine according to the objective function and physical constraints. Our motion synthesis is like a motion editing method, which is appropriate for motion correction and extrapolation. By this means, we can correct and eliminate unrealistic motion data.

Skeleton driven animation is a popular method for the animation of deformable human and creature characters. The main advantage is its computational performance. However it suffers from a number of problems, such as collapsing elbow and candy wrapper joint. In this paper, we present a new method which is able to solve these defects; reduce the animator's manual work still allowing his/her full control over the process; and realistically simulate the fat bulge effect around a joint.

In this paper, we demonstrate complex dynamics in a classical Hopfield-type neural network with three neurons. There are no interconnections between the first one and the third one, so it may be a part with ignorable input from a complex neural network. However, the stable points, limit circles, single-scroll chaotic attractors and double-scrolls chaotic attractor have been observed as we adjust the weight from the second neuron to itself.

Wrinkles are important visual features on the skin of an animated character. However, realistic animation of wrinkles is not easy, especially for the hand, as it gathers many types of different wrinkles. In this paper, we present a computationally efficient method for the generation of hand wrinkles, both for the palm and the fingers. This method is developed by exploiting the inherent properties relating to wrinkles, which are volume preservation of the underlying flesh and incompressibility of the skin. Although this method is for the hand model, it is also applicable to other parts of the body where the skin is deformed by underlying skeleton joints.

Inspired by the development of the matrix completion theories and algorithms, a low-rank based motion capture (mocap) data refinement method has been developed, which has achieved encouraging results. However, it does not guarantee a stable outcome if we only consider the low-rank property of the motion data. To solve this problem, we propose to exploit the temporal stability of human motion and convert the mocap data refinement problem into a robust matrix completion problem, where both the low-rank structure and temporal stability properties of the mocap data as well as the noise effect are considered. An efficient optimization method derived from the augmented Lagrange multiplier algorithm is presented to solve the proposed model. Besides, a trust data detection method is also introduced to improve the degree of automation for processing the entire set of the data and boost the performance. Extensive experiments and comparisons with other methods demonstrate the effectiveness of our approaches on both predicting missing data and de-noising. © 2014 Elsevier Inc. All rights reserved.