Search

Real time facial animation for virtual 3D characters has important applications such as AR/VR, interactive 3D entertainment, pre-visualization and video conferencing. Yet despite important research breakthroughs in facial tracking and performance capture, there are very few commercial examples of real-time facial animation applications in the consumer market. Mass adoption requires realtime performance on commodity hardware and visually pleasing animation that is robust to real world conditions, without requiring manual calibration. We present an end-to-end deep learning framework for regressing facial animation weights from video that addresses most of these challenges. Our formulation is fast (3.2 ms), utilizes images of real human faces along with millions of synthetic rendered frames to train the network on real-world scenarios, and produces jitter-free visually pleasing animations.

Avatar use on video-conference platforms has found dual purpose in recent times as a potential method for ensuring privacy and improving subjective engagement with remote meeting, provided one can also ensure a minimal loss in the quality of social interaction and sense of personal presence. This work focuses on interactions of this sort through real-time motion captured 3D personalized virtual avatars in a 2D video-conferencing context. Our experiments were designed with the intention of exploring previously defined perceptual illusions that occur with avatar-use in Virtual and Augmented Reality settings, outside of the immersive technological domains where they are normally measured. The research described here was aimed at empirically evaluating three separate dimensions of human-avatar interaction. The first was humans-as-avatars, with experimental conditions that were designed to measure changes to subjective perceptions of self-face ownership and self-concept. The second focus was other-perception, with the unique design of the studies outlined below among the first to measure social presence in a video-call between two human-driven avatars. The third emphasis was on the experiential content involved in avatar use, as there were measurements for emotion induction, fatigue and behavior change included in the data collection. The results describe some evidence for face and body ownership, while participants also reported high levels of social presence with the other avatar, indicating that avatar cameras could be a favorable alternative to non-camera feeds in video conferencing. There were also some useful insights gained regarding emotion elicitation in non-video vs. avatar conditions, as well as avatar-induced behavior change.

Facial alignment is an essential task for many higher level facial analysis applications, such as animation, human activity recognition and human - computer interaction. Although the recent availability of big datasets and powerful deep-learning approaches have enabled major improvements on the state of the art accuracy, the performance of current approaches can severely deteriorate when dealing with images in highly unconstrained conditions, which limits the real-life applicability of such models. In this paper, we propose a composite recurrent tracker with internal denoising that jointly address both single image facial alignment and deformable facial tracking in the wild. Specifically, we incorporate multilayer LSTMs to model temporal dependencies with variable length and introduce an internal denoiser which selectively enhances the input images to improve the robustness of our overall model. We achieve this by combining 4 different sub-networks that specialize in each of the key tasks that are required, namely face detection, bounding-box tracking, facial region validation and facial alignment with internal denoising. These blocks are endowed with novel algorithms resulting in a facial tracker that is both accurate, robust to in-the-wild settings and resilient against drifting. We demonstrate this by testing our model on 300-W and Menpo datasets for single image facial alignment, and 300-VW dataset for deformable facial tracking. Comparison against 20 other state of the art methods demonstrates the excellent performance of the proposed approach. This work is partly supported by the Spanish Ministry of Economy and Competitiveness under project grant TIN2017-90124-P, the Ramon y Cajal Programme, the Maria de Maeztu Units of Excellence Programme (MDM-2015-0502) and the donation bahi2018-19 to the CMTech at the UPF.

This paper presents a real time facial feature tracking system employing an inexpensive imaging device and two servo motors, to control the pan and tilt of the camera, for face tracking to capture vital facial features. Kanade Lucas Tomasi (KLT) method was adopted to track the facial features. The system was tested on various indoor and outdoor environments and the method shows good robustness and accuracy with a success rate of 95.7%. The proposed method has the ability to track from a selection of facial parts in comparison to being confined to track only the face.

In this paper, we present a system for real-time performance-driven facial animation. With the system, the user can control the facial expression of a digital character by acting out the desired facial action in front of an ordinary camera. First, we create a muscle-based 3D face model. The muscle actuation parameters are used to animate the face model. To increase the reality of facial animation, the orbicularis oris in our face model is divided into the inner part and outer part. We also establish the relationship between jaw rotation and facial surface deformation. Second, a real-time facial tracking method is employed to track the facial features of a performer in the video. Finally, the tracked facial feature points are used to estimate muscle actuation parameters to drive the face model. Experimental results show that our system runs in real time and outputs realistic facial animations. Compared with most existing performance-based facial animation systems, ours does not require facial markers, intrusive lighting, or special scanning equipment, thus it is inexpensive and easy to use.

Real-time facial performance capture has recently been gaining popularity in virtual film production, driven by advances in machine learning, which allows for fast inference of facial geometry from video streams. These learning-based approaches are significantly influenced by the quality and amount of labelled training data. Tedious construction of training sets from real imagery can be replaced by rendering a facial animation rig under on-set conditions expected at runtime. We learn a synthetic actor-specific prior by adapting a state-of-the-art facial tracking method. Synthetic training significantly reduces the capture and annotation burden and in theory allows generation of an arbitrary amount of data. But practical realities such as training time and compute resources still limit the size of any training set. We construct better and smaller training sets by investigating which facial image appearances are crucial for tracking accuracy, covering the dimensions of expression, viewpoint and illumination. A reduction of training data in 1-2 orders of magnitude is demonstrated whilst tracking accuracy is retained for challenging on-set footage.

When face and facial part detectors encounter partial or complete occlusion, their output becomes increasingly noisy, making it difficult for such algorithms to tackle complex scenes: parts are only detected intermittently, false alarms become more prevalent, parts are mis-identified, and some parts are consistently missed altogether. In this paper we propose a remedy to this problem: probabilistic fusion of face and facial detectors. The method jointly improves the localization and tracking performance of face and facial parts detectors by modelling the spatial and temporal inter-dependences of these detectors in a probabilistic graphical model. Spatial modelling can be used to rectify mis-identification, filter out false detections and infer the presence of missed detections during partial occlusion. Temporal modelling improves face and facial tracking in scenarios involving completely missed detections or complete occlusion. Experimental results quantify substantial benefits in terms of improved precision, recall and F-measure both for the detection of facial parts and of the face.

A user study was performed to compare two non-touch input methods for mobile gaming: tilt-input and facial tracking. User performance was measured on a mobile game called Star Jelly installed on a Google Nexus 7 HD tablet. The tilt-input method yielded significantly better performance. The mean game-score attained using tilt-input was 665.8. This was 7× higher than the mean of 95.1 for facial tracking. Additionally, participants were more precise with tilt-input with a mean star count of 19.7, compared to a mean of 1.9 using facial tracking. Although tilt-input was superior, participants praised facial tracking as challenging and innovative.

Accurate tracking of facial tissue in thermal infrared imaging is challenging because it is affected not only by positional but also physiological (functional) changes. This paper presents a particle filter tracker driven by a probabilistic template function with both spatial and temporal smoothing components, which is capable of adapting to abrupt positional and physiological changes. The method was tested on tracking facial regions of subjects under varying physiological and environmental conditions in 25 thermal clips. It demonstrated robustness and accuracy, outperforming other strategies. This new method promises improved performance in a number of biomedical applications that involve physiological measurements on the face, such as unobtrusive sleep and stress studies.

This paper presents the implementation of watershed segmentation for facial tracking. It focuses on tracking face of human in a continuous video transfer. The database is obtained from internet resource. Tracking efficiency is 97.2%. Keywords: Facial Tracking, Watershed Segmentation, Motion Estimation.