Search

This study presents a human action recognition system from multi-view image sequences. The authors' approach to human action recognition is based on an estimation of local motion from multiple camera views. The authors propose a new motion descriptor, called histogram of motion intensity and direction, to capture local motion characteristics of human activity. After image normalisation, they estimate motion flow using dense optical flow. Using regular grids, they extract local flow motion and estimate the dominant angle and the intensity of optical flow. The histogram of the dominant angle and its intensity are used as a descriptor for each sequence. After the identification of head direction, they concatenate descriptors in each view as a single feature vector from multiple-view sequences. Classification based on the proposed feature vector using support vector machine shows better performance than three-dimensional optical flow-based approaches, but with lower computational requirements. The authors evaluated action recognition on the publicly available i3DPost and the Institut de Recherche en Informatique et en Automatique (INRIA) Xmas Motion Acquisition Sequences database. Experimental results show promising state-of-the-art results and validate the advanced performance of the authors' approach.

The problem we address in this paper is how to learn joint representation from data lying on multiple manifolds. We are given multiple data sets, and there is an underlying common manifold among the different data sets. Each data set is considered to be an instance of this common manifold. The goal is to achieve an embedding of all the points on all the manifolds in a way that preserves the local structure of each manifold and that, at the same time, collapses all the different manifolds into one manifold in the embedding space while preserving the implicit correspondences between the points across different data sets. We propose a framework to learn embedding of such data, which can preserve the intra-manifolds' local geometric structure and the inter-manifolds' correspondence structure. The proposed solution works as extensions to current state-of-the-art spectral-embedding approaches to handling multiple manifolds. Graphical abstractDisplay Omitted HighlightsA framework to learn joint embedding space from multiple manifold data is presented.Intra-manifolds' geometric structure and inter-manifolds' structure are preserved.Implicit correspondence between the points across different data sets is estimated.Several embedding examples in different data sets are provided.Current spectral-embedding approaches are extended to handle multiple manifolds.

This paper presents a new framework for human action classification using a tensor dynamical model of human action from 3-dimensional (3D) volume sequences and distance measurement on Grassmann manifold. The tensor dynamical model is an extension of linear dynamical models for multi-dimensional sequence analysis. Each subdimensional linear dynamic models are estimated from tensor sequences using an iterative expectation-maximization (EM) algorithm after projection of tensor sequence to each dimensional axis. The combination of distances on Grassmann manifold of linear dynamic systems in each dimension of the tensor dynamic model provides similarity measurement between two tensor dynamical systems. The proposed approach can be applied to 3D depth or convex hull data as well as 2D video image sequences. Experimental results show good performance in human action recognition from INRIA multiview human action database.

This paper presents a new approach for tracking hand rotation and various grasping gestures through an infrared camera. For the complexity and ambiguity of an observed hand shape, it is difficult to simultaneously estimate hand configuration and orientation from a silhouette image of a grasping hand gesture. This paper proposes a dynamic shape model for hand grasping gestures using cylindrical manifold embedding to analyze variations of hand shape in different hand configurations between two key hand poses and in simultaneous circular view change by hand rotation. An arbitrary hand shape between two key hand poses from any view can be generated using a cylindrical manifold embedding point after learning nonlinear generative models from the embedding space to the corresponding hand shape observed. The cylindrical manifold embedding model is extended to various grasping gestures by decomposing multiple cylindrical manifold embeddings through grasping style analysis. Grasping hand gestures with simultaneous hand rotation are tracked using particle filters on the manifold space with grasping style estimation. Experimental results for synthetic and real data indicate that the proposed model can accurately track various grasping gestures with hand rotation. The proposed approach may be applied to advanced user interfaces in dark environments by using images beyond the visible spectrum.

Preferred combinations of illuminance and color temperature of lighting depend on daily living activities. We investigated whether the illumination stimuli of LED lighting can enhance attention and relaxation level by controlling color temperature and illuminance level according to activities. Illuminations and color temperatures of LED flat panels are controlled in accordance with activities such as office work and resting. The attention and relaxation level under the task specific lightings are compared with those under normal lighting condition. Single channel EEG signals from the NeuroSky`s Mindset are used to estimate attention and relaxation level of human subjects under different lighting conditions. Experiment results show that high color temperature with high illuminance of LED lightings (6600K, 800lx) shows improved attention level compared with conventional lighting conditions (4000K, 500lx).

The problem of separation of style and content is an essential element of visual perception, and is a fundamental mystery of perception. This problem appears extensively in different computer vision applications. The problem we address in this paper is the separation of style and content when the content lies on a low-dimensional nonlinear manifold representing a dynamic object. We show that such a setting appears in many human motion analysis problems. We introduce a framework for learning parameterization of style and content in such settings. Given a set of topologically equivalent manifolds, the Homeomorphic Manifold Analysis (HMA) framework models the variation in their geometries in the space of functions that maps between a topologically-equivalent common representation and each of them. The framework is based on decomposing the style parameters in the space of nonlinear functions that map between a unified embedded representation of the content manifold and style-dependent visual observations. We show the application of the framework in synthesis, recognition, and tracking of certain human motions that follow this setting, such as gait and facial expressions.

Modeling and analyzing the dynamic shape of human motion is a challenging task owing to temporal variations in the shape and multiple sources of observed shape variations such as viewpoint, motion speed, clothing, etc. We present a new framework for dynamic shape analysis based on temporal normalization and factorized shape style analysis. Using a nonlinear generative model with motion manifold embedding in a low-dimensional space, we detect cycles of periodic motion like gait in different views and synthesize temporally-aligned shape sequences from the same type of motion at different speeds. The bilinear analysis of temporally-aligned shape sequences decomposes dynamic motion into time-invariant shape style factors and time-dependent motion factors. We extend the bilinear model into a tensor shape model, a multilinear decomposition of dynamic shape sequences for view-invariant shape style representations. The shape style is a view-invariant, time-invariant, and speed-invariant shape signature and is used as a feature vector for human identification. The shape style can be adapted to new environmental conditions by iterative estimation of style and content factors to reflect new observation conditions. We present the experimental results of gait recognition using the CMU Mobo gait database and the USF gait challenging database.

In this paper, we consider modeling data lying on multiple continuous manifolds. In particular, we model the shape manifold of a person performing a motion observed from different viewpoints along a view circle at a fixed camera height. We introduce a model that ties together the body configuration (kinematics) manifold and visual (observations) manifold in a way that facilitates tracking the 3D configuration with continuous relative view variability. The model exploits the low-dimensionality nature of both the body configuration manifold and the view manifold, where each of them are represented separately. The resulting representation is used for tracking complex motions within a Bayesian framework, in which the model provides a low-dimensional state representation as well as a constrained dynamic model for both body configuration and view variations. Experimental results estimating the 3D body posture from a single camera are presented for the HUMANEVA dataset and other complex motion video sequences.

Our objective is to learn representations for the shape and the appearance of moving (dynamic) objects that support tasks such as synthesis, pose recovery, reconstruction, and tracking. In this paper, we introduce a framework that aims to learn landmark-free correspondence-free global representations of dynamic appearance manifolds. We use nonlinear dimensionality reduction to achieve an embedding of the global deformation manifold which preserves the geometric structure of the manifold. Given such embedding, a nonlinear mapping is learned from the embedding space into the visual input space. Therefore, any visual input is represented by a linear combination of nonlinear bases functions centered along the manifold in the embedding space. We also show how approximate solution for the inverse mapping can be obtained in a closed form which facilitate recovery of the intrinsic body configuration. We use the framework to learn the gait manifold as an example of a dynamic shape manifold, as well as to learn the manifolds for some simple gestures and facial expressions as examples of dynamic appearance manifolds.

Hand shapes vary for different views or hand rotations. In addition, the high degree of freedom of hand configurations makes it difficult to track hand shape variations. This paper presents a new manifold embedding method that models hand shape variations in different hand configurations and in different views due to hand rotation. Instead of traditional silhouette images, the hand shapes are modeled using depth map images, which provides rich shape information invariant to illumination changes. These depth map images vary for different viewing directions, similar to shape silhouettes. Sample data along view circles are collected for all the hand configuration variations. A new manifold embedding method using a 4D torus for modeling low dimensional hand configuration and hand rotation is proposed to model the product of three circular manifolds. After learning nonlinear mapping from the proposed embedding space to depth map images, we can achieve the tracking of arbitrary shape variations with hand rotation using particle filter on the embedding manifold. The experiment results from both synthetic and real data show accurate estimations of hand rotation through the estimation of the view parameters and hand configuration from key hand poses and hand configuration phases.

This paper presents performance evaluation of silhouette-based action recognition from virtual human silhouette(VIHASI), which is publicly available. We especially evaluate action recognition performance based on tensor analysis. We applied tensor analysis for the silhouette data itself after sampling the same number of silhouette frames from given sample frames. Geometric manifold embedding based on tangent bundle is applied and shows better performance compared with baseline performance.

This paper presents a new framework that models facial expressions in multiple people with different expressions and synthesize new stylized subtle facial expressions using the generative models. As facial expressions pass through nonlinear shape deformations during facial expressions, we model the facial expression in nonlinear mapping space based on low dimensional embedding and kernel mapping. Characteristics of different type of expressions and variances in different people are decomposed by analyzing the nonlinear mapping between the Euclidean space of facial motion and a low dimensional embedding of these expressions. Using high resolution tracking of densely sampled 3D data, the generative model can control subtle facial expression characteristics of different person in different expression by low dimensional person dependent style factor and expression type dependent expression factor. The temporal characteristics of the motion can also be controlled by the trajectory sampling on the low dimensional embedding manifold which is independent of person style and expression type. Our experimental results are shown for subtle differences in different smile expressions in different people from dense 3D tracking.

The problem we address in the paper is how to learn a joint representation from data lying on multiple manifolds. We are given multiple data sets and there is an underlying common manifold among the different data set. We propose a framework to learn an embedding of all the points on all the manifolds in a way that preserves the local structure on each manifold and, in the same time, collapses all the different manifolds into one manifold in the embedding space, while preserving the implicit correspondences between the points across different data sets. The proposed solution works as extensions to current state of the art spectral-embedding approaches to handle multiple manifolds.

This paper presents a new approach for hand rotation and grasping tracking from a single IR camera. For the complexity and ambiguity of hand pose, it is difficult to track hand pose and view variations simultaneously from a single camera. We propose a cylindrical manifold embedding for one dimensional hand pose variation and cyclic viewpoint variation. A hand pose shape from a specific viewpoint can be generated from an embedding point on the cylindrical manifold after learning nonlinear generative models from the embedding space to the corresponding observed shape. Hand grasping with simultaneous hand rotation is tracked using particle filter on the manifold space. Experimental results for synthetic and real data show accurate tracking of grasping hand with rotation. The proposed approach shows potentials for advanced user interface in dark environments.

We present a framework for monocular 3D kinematic pose tracking and viewpoint estimation of periodic and quasi-periodic human motions from an uncalibrated camera. The approach we introduce here is based on learning both the visual observation manifold and the kinematic manifold of the motion using a joint representation. We show that the visual manifold of the observed shape of a human performing a periodic motion, observed from different viewpoints, is topologically equivalent to a torus manifold. The approach we introduce here is based on the supervised learning of both the visual and kinematic manifolds. Instead of learning an embedding of the manifold, we learn the geometric deformation between an ideal manifold (conceptual equivalent topological structure) and a twisted version of the manifold (the data). Experimental results show accurate estimation of the 3D body posture and the viewpoint from a single uncalibrated camera.

We present a framework for tracking large facial deformations using nonlinear dynamic shape and appearance model based upon local motion estimation. Local facial deformation estimation based on a given single template fails to track large facial deformations due to significant appearance variations. A nonlinear generative model that uses low dimensional manifold representation provides adaptive facial appearance templates depending upon the movement of the facial motion state and the expression type. The proposed model provides a generative model for Bayesian tracking of facial motions using particle filtering with simultaneous estimation of the expression type. We estimate the geometric transformation and the global deformation using the generative model. The appearance templates from the global model then estimate local deformation based on thin-plate spline parameters.

If we consider the appearance of human motion such as gait, facial expression and gesturing, most of such activities result in nonlinear manifolds in the image space. Although the intrinsic body configuration manifolds might be very low in dimensionality, the resulting appearance manifold is challenging to model given various aspects that affects the appearance such as the view point, the person shape and appearance, etc. In this paper we learn decomposable generative models that explicitly decompose the intrinsic body configuration as a function of time from other conceptually orthogonal aspects that affects the appearance such as the view point, the person performing the action, etc. The frameworks is based on learning nonlinear mappings from a conceptual representation of the motion manifold that is homeomorphic to the actual manifold and decompose other sources of variation in the mapping coefficient space.

In this paper we consider modeling data lying on multiple continuous manifolds. In particular, we model the shape manifold of a person performing a motion observed from different view points along a view circle at fixed camera height. We introduce a model that ties together the body configuration (kinematics) manifold and the visual manifold (observations) in a way that facilitates tracking the 3D configuration with continuous relative view variability. The model exploits the low dimensionality nature of both the body configuration manifold and the view manifold where each of them are represented separately.

Inferring 3D body pose as well as viewpoint from a single silhouette image is a challenging problem. We present a new generative model to represent shape deformations according to view and body configuration changes on a two dimensional manifold. We model the two continuous states by a product space (different configurations times different views) embedded on a conceptual two dimensional torus manifold. We learn a nonlinear mapping between torus manifold embedding and visual input (silhouettes) using empirical kernel mapping. Since every view and body pose has a corresponding embedding point on the torus manifold, inferring view and body pose from a given image becomes estimating the embedding point from a given input. As the shape varies in different people even in the same view and body pose, we extend our model to be adaptive to different people by decomposing person dependent style factors. Experimental results with real data as well as synthetic data show simultaneous estimation of view and body configuration from given silhouettes from unknown people

We propose motion manifold learning and motion primitive segmentation framework for human motion synthesis from motion-captured data. High dimensional motion capture date are represented using a low dimensional representation by topology preserving network, which maps similar motion instances to the neighborhood points on the low dimensional motion manifold. Nonlinear manifold learning between a low dimensional manifold representation and high dimensional motion data provides a generative model to synthesize new motion sequence by controlling trajectory on the low dimensional motion manifold. We segment motion primitives by analyzing low dimensional representation of body poses through motion from motion captured data. Clustering techniques like k-means algorithms are used to find motion primitives after dimensionality reduction. Motion dynamics in training sequences can be described by transition characteristics of motion primitives. The transition matrix represents the temporal dynamics of the motion with Markovian assumption. We can generate new motion sequences by perturbing the temporal dynamics

Characteristics of the 2D contour shape deformation in human motion contain rich information and can be useful for human identification, gender classification, 3D pose reconstruction and so on. In this paper we introduce a new approach for contour tracking for human motion using an explicit modeling of the motion manifold and learning a decomposable generative model. We use nonlinear dimensionality reduction to embed the motion manifold in a low dimensional configuration space utilizing the constraints imposed by the human motion. Given such embedding, we learn an explicit representation of the manifold, which reduces the problem to a one-dimensional tracking problem and also facilitates linear dynamics on the manifold. We also utilize a generative model through learning a nonlinear mapping between the embedding space and the visual input space, which facilitates capturing global deformation characteristics. The contour tracking problem is formulated as states estimation in the decomposed generative model parameter within a Bayesian tracking framework. The result is a robust, adaptive gait tracking with shape style estimation.

Bilinear and multi-linear models have been successful in decomposing static image ensembles into perceptually orthogonal sources of variations, e.g., separation of style and content. If we consider the appearance of human motion such as gait, facial expression and gesturing, most of such activities result in nonlinear manifolds in the image space. The question that we address in this paper is how to separate style and content on manifolds representing dynamic objects. In this paper we learn a decomposable generative model that explicitly decomposes the intrinsic body configuration (content) as a function of time from the appearance (style) of the person performing the action as time-invariant parameter. The framework we present in this paper is based on decomposing the style parameters in the space of nonlinear functions which map between a learned unified nonlinear embedding of multiple content manifolds and the visual input space.

We aim to infer 3D body pose directly from human silhouettes. Given a visual input (silhouette), the objective is to recover the intrinsic body configuration, recover the viewpoint, reconstruct the input and detect any spatial or temporal outliers. In order to recover intrinsic body configuration (pose) from the visual input (silhouette), we explicitly learn view-based representations of activity manifolds as well as learn mapping functions between such central representations and both the visual input space and the 3D body pose space. The body pose can be recovered in a closed form in two steps by projecting the visual input to the learned representations of the activity manifold, i.e., finding the point on the learned manifold representation corresponding to the visual input, followed by interpolating 3D pose.

Facial expressions convey personal characteristics and subtle emotional states. This paper presents a new framework for modeling subtle facial motions of different people with different types of expressions from high-resolution facial expression tracking data to synthesize new stylized subtle facial expressions. A conceptual facial motion manifold is used for a unified representation of facial motion dynamics from three-dimensional (3D) high-resolution facial motions as well as from two-dimensional (2D) low-resolution facial motions. Variant subtle facial motions in different people with different expressions are modeled by nonlinear mappings from the embedded conceptual manifold to input facial motions using empirical kernel maps. We represent facial expressions by a factorized nonlinear generative model, which decomposes expression style factors and expression type factors from different people with multiple expressions. We also provide a mechanism to control the high-resolution facial motion model from low-resolution facial video sequence tracking and analysis. Using the decomposable generative model with a common motion manifold embedding, we can estimate parameters to control 3D high resolution facial expressions from 2D tracking results, which allows performance-driven control of high-resolution facial expressions.