Search

The paper introduces a novel detection and tracking system that provides both frame-view and world-coordinate human location information, based on video from multiple synchronized and calibrated cameras with overlapping fields of view. The system is developed and evaluated for the specific scenario of a seminar lecturer presenting in front of an audience inside a “smart room”, its aim being to track the lecturer’s head centroid in the three-dimensional (3D) space and also yield two-dimensional (2D) face information in the available camera views. The proposed approach is primarily based on a statistical appearance model of human faces by means of well-known AdaBoost-like face detectors, extended to address the head pose variation observed in the smart room scenario of interest. The appearance module is complemented by two novel components and assisted by a simple tracking drift detection mechanism. The first component of interest is the initialization module, which employs a spatio-temporal dynamic programming approach with appropriate penalty functions to obtain optimal 3D location hypotheses. The second is an adaptive subspace learning based 2D tracking scheme with a novel forgetting mechanism, introduced to reduce tracking drift and increase robustness. System performance is benchmarked on an extensive database of realistic human interaction in the lecture smart room scenario, collected as part of the European integrated project “CHIL”. The system consistently achieves excellent tracking precision, with a 3D mean tracking error of less than 16 cm, and is demonstrated to outperform four alternative tracking schemes. Furthermore, the proposed system performs relatively well in detecting frontal and near-frontal faces in the available frame views.

We propose a flexible technique to easily calibrate a camera. It only requires the camera to observe a planar pattern shown at a few (at least two) different orientations. Either the camera or the planar pattern can be freely moved. The motion need not be known. Radial lens distortion is modeled. The proposed procedure consists of a closed-form solution, followed by a nonlinear refinement based on the maximum likelihood criterion. Both computer simulation and real data have been used to test the proposed technique and very good results have been obtained. Compared with classical techniques which use expensive equipment such as two or three orthogonal planes, the proposed technique is easy to use and flexible. It advances 3D computer vision one more step from laboratory environments to real world use.

Precise 3D pose estimation plays a significant role in developing human-computer interfaces and practical face recognition systems. This task is challenging due to the personality in pose variation for a certain subject. In this work, the pose data space is considered as a union of the submanifolds which characterize different subjects, instead of a single continuous manifold as conventionally regarded. A novel manifold embedding algorithm dually supervised by subjects and poses, called Synchronized Submanifold Embedding(SSE), is proposed for person-independentprecise pose estimation. First, the submanifold of a certain subject is approximated as a set of simplexes constructed using neighboring samples. Then, these simplexized submanifolds from different subjects are embedded by synchronizing the locally propagated poses within the simplexes and at the same time maximizing the intra-submanifold variances. Finally, the pose of a new datum is estimated as the median of the poses for the nearest neighbors in the dimensionality reduced feature space. The experiments on the 3D pose estimation database, CHIL data for CLEAR07 evaluation demonstrate the effectiveness of our proposed algorithm.

Head pose estimation in low resolution is a challenge problem. Traditional pose estimation algorithms, which assume faces have been well aligned before pose estimation, would face much difficulty in this situation, since face alignment itself does not work well in this low resolution scenario. In this paper, we propose to estimate head pose using view-based multi-view face detectors directly. Naive Bayesian classifier is then applied to fuse the information of head pose from multiple camera views. To model the temporal changing of head pose, Hidden Markov Model is used to obtain the optimal sequence of head pose with greatest likelihood.

We present the IBM systems submitted and evaluated within the CLEAR'06 evaluation campaign for the tasks of single person visual 3D tracking (localization) and 2D face tracking on CHIL seminar data. The two systems are significantly inter-connected to justify their presentation within a single paper as a joint vision system for single person 2D-face and 3D-head tracking, suitable for smart room environments with multiple synchronized, calibrated, stationary cameras. Indeed, in the developed system, face detection plays a pivotal role in 3D person tracking, being employed both in system initialization as well as in detecting possible tracking drift. Similarly, 3D person tracking determines the 2D frame regions where a face detector is subsequently applied. The joint system consists of a number of components that employ detection and tracking algorithms, some of which operate on input from all four corner cameras of the CHIL smart rooms, while others select and utilize two out of the four available cameras. Main system highlights constitute the use of AdaBoost-like multi-pose face detectors, a spatio-temporal dynamic programming algorithm to initialize 3D location hypotheses, and an adaptive subspace learning based tracking scheme with a forgetting mechanism as a means to reduce tracking drift. The system is benchmarked on the CLEAR'06 CHIL seminar database, consisting of 26 lecture segments recorded inside the smart rooms of the UKA and ITC CHIL partners. Its resulting 3D single-person tracking performance is 86% accuracy with a precision of 88 mm, whereas the achieved face tracking score is 54% correct with 37% wrong detections and 19% misses. In terms of speed, an inefficient system implementation runs at about 2 fps on a P4 2.8 GHz desktop.

A minimum variance estimation framework for 3D face reconstruction from multiple views and a new 3D surface reconstruction algorithm based on deformable subdivision mesh is proposed in this paper. First, an efficient 2D-to-3D integrated face reconstruction approach is introduced to reconstruct a personalized 3D face model from a single frontal face image with minimum variance estimation. Then, a new deformable mesh based surface reconstruction algorithm is applied to the images from different views to get more observation of the 3D face, especially the depth information, which could not be obtained from a single image directly. Based on the result of the 3D surface reconstruction, we use the minimum variance estimation again to refine the estimation of the 3D face. We combine the texture from different views, and the result looks photorealistic.

Robust face detection presents a difficult problem in real interaction scenarios, that, in order to achieve, most often requires employing additional sources of information. In this paper, we consider two such sources: Temporal information, available in the form of video sequences, and spatial information, available from multiple calibrated cameras with synchronous, overlapping fields of view of the 3D scene of interest. These two sources are exploited jointly, using a novel dynamic programming approach, for a lecture scenario inside appropriately equipped smart rooms, aiming at robust face detection of the lecturer within the available 2D camera views. Experimental results, reported on the CHIL project database, demonstrate that the proposed approach outperforms purely frame-based face detection.

A realtime system for shrug detection is discussed in this paper. The system is automatically initialized by a face detector based on Ada-boost [P. Viola and M. Jones, May 2004]. After frontal face is localized by the face detector, shoulder position is detected by fitting a parabola to the nearby horizontal edges using weighted Hough transform [K. Sugawara, 1997]. Since shrug is an action which is defined not only by the distance between face and shoulder but also the relative temporal-spatial changing between them, we propose a parameterizing scheme using two different parabolas, named as "stable parabola" (SP) and "transient parabola" (TP) to characterize the action shrug. Stable parabola represents the mean shoulder position over a long time duration, while transient parabola represents the mean shoulder position of a very short time duration. By using this scheme (only 6 dimensions), we avoid the high dimensional representation of the temporal process-shrug, and therefore make the realtime implementation possible. The shrug detector is then trained in the parameter space using Fisher discriminant analysis (FDA). The experiments show that the proposed shrug detector is able to not only detect the shrug action correctly and efficiently (in realtime), but also tolerate the large in-class variation caused by different subject, different action speed, illumination, partial occlusion, and background clutter. So the proposed realtime shrug detector is promising in video analysis under an uncontrolled environment.

Visual detection and tracking of humans in complex scenes is a challenging problem with a wide range of applications, for example surveillance and human-computer interaction. In many such applications, time-synchronous views from multiple calibrated cameras are available, and both frame-view and space-level human location information is desired. In such scenarios, efficiently combining the strengths of face detection and person tracking is a viable approach that can provide both levels of information required and improve robustness. In this paper, we propose a novel vision system that detects and tracks human faces automatically, using input from multiple calibrated cameras. The method uses an Adaboost algorithm variant combined with mean shift tracking applied on single camera views for face detection and tracking, and fuses the results on multiple camera views to check for consistency and obtain the three-dimensional head estimate. We apply the proposed system to a lecture scenario in a smart room, on a corpus collected as part of the CHIL European Union integrated project. We report results on both frame-level face detection and three-dimensional head tracking. For the latter, the proposed algorithm achieves similar results with the IBM “PeopleVision” system.

Perhaps the most fundamental application of affective computing will be Human-Computer Interaction (HCI) in which the computer should have the ability to detect and track the user's affective states, and make corresponding feedback. The human multi-sensor affect system defines the expectation of multimodal affect analyzer. In this paper, we present our efforts toward audio-visual HCI-related affect recognition. With HCI applications in mind, we take into account some special affective states which indicate users' cognitive/motivational states. Facing the fact that a facial expression is influenced by both an affective state and speech content, we apply a smoothing method to extract the information of the affective state from facial features. In our fusion stage, a voting method is applied to combine audio and visual modalities so that the final affect recognition accuracy is greatly improved. We test our bimodal affect recognition approach on 38 subjects with 11 HCI-related affect states. The extensive experimental results show that the average person-dependent affect recognition accuracy is almost 90% for our bimodal fusion.

In this paper, a new boosting algorithm, called FloatBoost, is proposed to construct a strong face-nonface classifier. FloatBoost incorporates the idea of Floating Search into AdaBoost, and yields similar or higher classification accuracy than AdaBoost with a smaller number of weak classifiers. We also present a novel framework for fast multi-view face detection. A detector-pyramid architecture is designed to quickly discard a vast number of non-face sub-windows and hence perform multi-view face detection efficiently. This results in the first real-time multi-view face detection system which runs at 5 frames per second for 320x240 image sequence.

We present a detector-pyramid architecture for real-time multi-view face detection. Using a coarse to fine strategy, the full view is partitioned into finer and finer views. Each face detector in the pyramid detects faces of its respective view range. Its training is performed by using a new meta booting learning algorithm. This results in the first real-time multi-view face detection system which runs at 5 frames per second for 320/spl times/240 image sequence.

In this paper, we present a system which learns to detect multi-view faces. The system uses a coarse-to-fine, simple-to-complex architecture called detector-pyramid. A new boosting algorithm, called FloatBoost, is proposed to construct a strong face-nonface classifier from weak classifiers for the component detectors in the pyramid. FloatBoost incorporates the idea of Floating Search into AdaBoost, and yields similar or higher classification accuracy than AdaBoost with a smaller number of weak classifiers. This work leads to the first real-time multi-view face detection system in the world. It runs at 200 ms per image of size 320/spl times/240 pixels on a Pentium-III CPU of 700 MHz.