Search

We consider scenarios where we have zero instances of real pedestrian data (e.g., a newly installed surveillance system in a novel location in which no labeled real data or unsupervised real data exists yet) and a pedestrian detector must be developed prior to any observations of pedestrians. Given a single image and auxiliary scene information in the form of camera parameters and geometric layout of the scene, our approach infers and generates a large variety of geometrically and photometrically accurate potential images of synthetic pedestrians along with purely accurate ground-truth labels through the use of computer graphics rendering engine. We first present an efficient discriminative learning method that takes these synthetic renders and generates a unique spatially-varying and geometry-preserving pedestrian appearance classifier customized for every possible location in the scene. In order to extend our approach to multi-task learning for further analysis (i.e., estimating pose and segmentation of pedestrians besides detection), we build a more generalized model employing a fully convolutional neural network architecture for multi-task learning leveraging the “free" ground-truth annotations that can be obtained from our pedestrian synthesizer. We demonstrate that when real human annotated data is scarce or non-existent, our data generation strategy can provide an excellent solution for an array of tasks for human activity analysis including detection, pose estimation and segmentation. Experimental results show that our approach (1) outperforms classical models and hybrid synthetic-real models, (2) outperforms various combinations of off-the-shelf state-of-the-art pedestrian detectors and pose estimators that are trained on real data, and (3) surprisingly, our method using purely synthetic data is able to outperform models trained on real scene-specific data when data is limited. [ABSTRACT FROM AUTHOR]

This paper describes the development of the IAS-Society and the trends in the Intelligent Autonomous Systems conferences. The first IAS conference was held in 1986 and was the first conference on this topic. The Society, founded in 1994, laid the basis for the organization of the IAS conferences. The topics presented at the successive IAS conferences showed clearly the development of the field of Intelligent Autonomous Systems. A short overview is given of the specific topics and percentage of papers in the different research areas in the conferences before and after the year 2000. [ABSTRACT FROM AUTHOR]

Abstract: Phase contrast, a noninvasive microscopy imaging technique, is widely used to capture time-lapse images to monitor the behavior of transparent cells without staining or altering them. Due to the optical principle, phase contrast microscopy images contain artifacts such as the halo and shade-off that hinder image segmentation, a critical step in automated microscopy image analysis. Rather than treating phase contrast microscopy images as general natural images and applying generic image processing techniques on them, we propose to study the optical properties of the phase contrast microscope to model its image formation process. The phase contrast imaging system can be approximated by a linear imaging model. Based on this model and input image properties, we formulate a regularized quadratic cost function to restore artifact-free phase contrast images that directly correspond to the specimen’s optical path length. With artifacts removed, high quality segmentation can be achieved by simply thresholding the restored images. The imaging model and restoration method are quantitatively evaluated on microscopy image sequences with thousands of cells captured over several days. We also demonstrate that accurate restoration lays the foundation for high performance in cell detection and tracking. [Copyright &y& Elsevier]

The active appearance models (AAMs) provide the detailed descriptive parameters that are useful for various autonomous face analysis problems. However, they are not suitable for robust face tracking across large pose variation for the following reasons. First, they are suitable for tracking the local movements of facial features within a limited pose variation. Second, they use gradient-based optimization techniques for model fitting and the fitting performance is thus very sensitive to initial model parameters. Third, when their fitting is failed, it is difficult to obtain appropriate model parameters to re-initialize them. To alleviate these problems, we propose to combine the active appearance models and the cylinder head models (CHMs), where the global head motion parameters obtained from the CHMs are used as the cues of the AAM parameters for a good fitting or re-initialization. The good AAM parameters for robust face tracking are computed in the following manner. First, we estimate the global motion parameters by the CHM fitting algorithm. Second, we project the previously fitted 2D shape points onto the 3D cylinder surface inversely. Third, we transform the inversely projected shape points by the estimated global motion parameters. Fourth, we project the transformed 3D points onto the input image and computed the AAM parameters from them. Finally, we treat the computed AAM parameters as the initial parameters for the fitting. Experimental results showed that face tracking combining AAMs and CHMs is more pose robust than that of AAMs in terms of 170% higher tracking rate and the 115% wider pose coverage. [ABSTRACT FROM AUTHOR]

Active Appearance Model (AAM) framework is a very useful method that can fit the shape and appearance model to the input image for various image analysis and synthesis problems. However, since the goal of the AAM fitting algorithm is to minimize the residual error between the model appearance and the input image, it often fails to accurately converge to the landmark points of the input image. To alleviate this weakness, we have combined Active Shape Models (ASM) into AAMs, in which ASMs try to find correct landmark points using the local profile model. Since the original objective function of the ASM search is not appropriate for combining these methods, we derive a gradient based iterative method by modifying the objective function of the ASM search. Then, we propose a new fitting method that combines the objective functions of both ASM and AAM into a single objective function in a gradient based optimization framework. Experimental results show that the proposed fitting method reduces the average fitting error when compared with existing fitting methods such as ASM, AAM, and Texture Constrained-ASM (TC-ASM) and improves the performance of facial expression recognition significantly. [ABSTRACT FROM AUTHOR]

-Geometric reconstruction problems in computer vision are often solved by minimizing a cost function that combines the reprojection errors in the 2D images. In this paper, we show that, for various geometric reconstruction problems, their reprojection error functions share a common and quasiconvex formulation. Based on the quasiconvexity, we present a novel quasiconvex optimization framework in which the geometric reconstruction problems are formulated as a small number of small-scale convex programs that are readily solvable. Our final reconstruction algorithm is simple and has intuitive geometric interpretation. In contrast to existing local minimization approaches, our algorithm is deterministic and guarantees a predefined accuracy of the minimization result. The quasiconvexity also provides an intuitive method to handle directional uncertainties and outliers in measurements. For a large-scale problem or in a situation where computational resources are constrained, we provide an efficient approximation that gives a good upper bound (but not global minimum) on the reconstruction error. We demonstrate the effectiveness of our algorithm by experiments on both synthetic and real data. [ABSTRACT FROM AUTHOR]

Most image registration problems are formulated in an asymmetric fashion. Given a pair of images, one is implicitly or explicitly regarded as a template and warped onto the other to match as well as possible. In this paper, we focus on this seemingly arbitrary choice of the roles and reveal how it may lead to biased warp estimates in the presence of relative scaling. We present a principled way of selecting the template and explain why only the correct asymmetric form, with the potential inclusion of a blurring step, can yield an unbiased estimator. We validate our analysis in the domain of model-based face tracking. We show how the usual Active Appearance Model (AAM) formulation overlooks the asymmetry issue, causing the fitting accuracy to degrade quickly when the observed objects are smaller than their model. We formulate a novel, ‘resolution-aware fitting’ (RAF) algorithm that respects the asymmetry and incorporates an explicit model of the blur caused by the camera's sensing elements into the fitting formulation. We compare the RAF algorithm against a state-of- the-art tracker across a variety of resolutions and AAM complexity levels. Experimental results show that RAF significantly improves the estimation accuracy of both shape and appearance parameters when fitting to low-resolution data. Recognizing and accounting for the asymmetry of image registration leads to tangible accuracy improvements in analyzing low-resolution imagery. [ABSTRACT FROM AUTHOR]

The article presents information on dynamic events and methods for recording these events digitally. It is stated that the recorded events can be changed by capturing them in a four dimensional (4D) mode. The ability to be present at an event taking place at a distance is known as telepresence, which involves the digitization of dynamic events. It is informed that the method of converting real events to virtual ones is called virtualized reality. The first system to capture a large time-varying dynamic event was the virtualized reality system. In a virtual reality system the digitization process begins with the non-intrusive recording of dynamic events.

We propose a system that is capable of detailed analysis of eye region images in terms of the position of the iris, degree of eyelid opening, and the shape, complexity, and texture of the eyelids. The system uses a generative eye region model that parameterizes the fine structure and motion of an eye. The structure parameters represent structural individuality of the eye, including the size and color of the iris, the width, boldness, and complexity of the eyelids, the width of the bulge below the eye, and the width of the illumination reflection on the bulge. The motion parameters represent movement of the eye, including the up-down position of the upper and lower eyelids and the 2D position of the iris. The system first registers the eye model to the input in a particular frame and individualizes it by adjusting the structure parameters. The system then tracks motion of the eye by estimating the motion parameters across the entire image sequence. Combined with image stabilization to compensate for appearance changes due to head motion, the system achieves accurate registration and motion recovery of eyes. [ABSTRACT FROM AUTHOR]

This paper describes a vision system that recognizes moving targets such as vehicles and pedestrians on public streets. This system can: (1) classify targets {vehicle, pedestrian, others} and, for “vehicles,” discriminate vehicle types and (2) estimate the main colors of targets. According to the input images to the system, the categories of targets are set as {mule (golf cart for workers), sedan, van, truck, pedestrian (single or plural), and other (such as noise)}. Their colors are set as six color groups {red, orange, yellow; green; blue, light blue; white, silver, gray; dark blue, dark gray, black; dark red, dark orange}. In this experiment, we collected images of targets from 9: 00 a.m. to 5: 00 p.m. on sunny and cloudy days as system training samples. The recognition ratio was 91.1% under the condition that both the recognition results of type and color agreed with the operator’s judgment. In addition, the system can detect predefined specific targets such as delivery vans, post office vans, and police cars by combining recognition results for type and color. The recognition ratio for specific targets was 92.9%. For the classification and estimation of targets, we employed a statistical linear discrimination method (linear discriminant analysis, LDA) and a nonlinear decision rule (weighted K-nearest neighbor rule, K-NN). [ABSTRACT FROM AUTHOR]

In this paper we describe an algorithm to recover the scene structure, the trajectories of the moving objects and the camera motion simultaneously given a monocular image sequence. The number of the moving objects is automatically detected without prior motion segmentation. Assuming that the objects are moving linearly with constant speeds, we propose a unified geometrical representation of the static scene and the moving objects. This representation enables the embedding of the motion constraints into the scene structure, which leads to a factorization-based algorithm. We also discuss solutions to the degenerate cases which can be automatically detected by the algorithm. Extension of the algorithm to weak perspective projections is presented as well. Experimental results on synthetic and real images show that the algorithm is reliable under noise. [ABSTRACT FROM AUTHOR]

In this paper we describe a trainable object detector and its instantiations for detecting faces and cars at any size, location, and pose. To cope with variation in object orientation, the detector uses multiple classifiers, each spanning a different range of orientation. Each of these classifiers determines whether the object is present at a specified size within a fixed-size image window. To find the object at any location and size, these classifiers scan the image exhaustively. Each classifier is based on the statistics of localized parts. Each part is a transform from a subset of wavelet coefficients to a discrete set of values. Such parts are designed to capture various combinations of locality in space, frequency, and orientation. In building each classifier, we gathered the class-conditional statistics of these part values from representative samples of object and non-object images. We trained each classifier to minimize classification error on the training set by using Adaboost with Confidence-Weighted Predictions (Shapire and Singer, 1999). In detection, each classifier computes the part values within the image window and looks up their associated class-conditional probabilities. The classifier then makes a decision by applying a likelihood ratio test. For efficiency, the classifier evaluates this likelihood ratio in stages. At each stage, the classifier compares the partial likelihood ratio to a threshold and makes a decision about whether to cease evaluation—labeling the input as non-object—or to continue further evaluation. The detector orders these stages of evaluation from a low-resolution to a high-resolution search of the image. Our trainable object detector achieves reliable and efficient detection of human faces and passenger cars with out-of-plane rotation. [ABSTRACT FROM AUTHOR]

Differential interference contrast (DIC) microscopy is a powerful visualization tool used to study live biological cells. Its use, however, has been limited to qualitative observations. The inherent nonlinear relationship between the object properties and the image intensity makes quantitative analysis difficult. Toward quantitatively measuring optical properties of objects from DIC images, we develop a method to reconstruct the specimen's optical properties over a three-dimensional (3-D) volume. The method is a nonlinear optimization which uses hierarchical representations of the specimen and data. As a necessary tool, we have developed and validated a computational model for the DIC image formation process. We test our algorithm by reconstructing the optical properties of known specimens. [ABSTRACT FROM AUTHOR]

In this paper, we describe a reconstruction method for multiple motion scenes, which are scenes containing multiple moving objects, from uncalibrated views. Assuming that the objects are moving with constant velocities, the method recovers the scene structure, the trajectories of the moving objects, the camera motion, and the camera intrinsic parameters (except skews) simultaneously. We focus on the case where the cameras have unknown and varying focal lengths while the other intrinsic parameters are known. The number of the moving objects is automatically detected without prior motion segmentation. The method is based on a unified geometrical representation of the static scene and the moving objects. It first performs a projective reconstruction using a bilinear factorization algorithm and, then, converts the projective solution to a Euclidean one by enforcing metric constraints. Experimental results on synthetic and real images are presented. [ABSTRACT FROM AUTHOR]

Studies the fundamental constraints in producing low resolution input images in image formation process. Modeling of the point spread function; Invertibility analysis for square point spread functions; Bayesian formulation of super-resolution.

Introduces an Automatic Face Analysis (AFA) system developed to analyze facial expressions based on permanent and transient facial features in a nearly frontal-view face image sequence. Limitations addressed by the AFA system; Problems that an automated facial expression analysis system must solve; Facial feature representation and action unit recognition by neural networks; Details on the experiments conducted to evaluate the AFA system's performance.

Presents information on a study which described a cooperative stereo algorithm using global constraints to find a dense depth map. Assumptions for stereo vision algorithm; Explicit detection of occluded areas; Summary of algorithm; Conclusions.

Provides information on a study which recognized Facial Action Coding System action units, which are the smallest visible discriminable changes in facial expression. Experimental results; Discussion.

Presents an intensity-to-time paradigm suitable for very large scale integration computational sensor implementation of global manufacturing operations over sensed images. Use of global cumulative histogram; Development of a sorting computational sensor; Indices from the sorting sensor and inferred input intensity.

Describes a visual odometer which estimates helicopter position by visually locking on to and tracking ground objects. Positioning with a visual odometer; Analysis of the odometer's position sensing algorithm; Position estimation results.

Provides information on a study showing the relationship between the eigenvectors by Karhunen-Loeve (K-L) expansion and basis vectors for the discrete cosine transform (DCT). Information on eigenvectors of uniformly rotated images; Conclusions reached.

Presents a fast pattern matching algorithm with a large set of templates. Description of the algorithm; Explanation of vector subspace and normalized correlation; Objectives of the algorithm; Benefits of using the algorithm; Provision of statistical information.

Presents a sequential factorization method for recovering the three-dimentional shape of an object and the motion of the camera from a sequence of images using tracked features. Description of this method; Benefits of using the method developed; Details of the tests conducted on the performance of an algorithm; Methodology used in the tests; Results from the tests.

Presents a linear algorithm developed to recover 3D affine shape and motion from line correspondences with uncalibrated affine cameras. Description of the algorithm; Objectives of the algorithm; Investigation of the properties of projection of lines; Details on the study conducted on the uncalibrated one-dimensional camera; Information on affine shape recovery; Results from the study conducted.

Presents a factorization method based on the paraperspective projection model. Factorization method developed by Tomasi and Kanade; Application of the paraperspective factorization method to a wide range of motion scenarios; Description of camera and world reference frames; Introduction of the mathematical notation; Analysis of the paraperspective method.

This paper describes three-wheeled vehicles that can move inside a pipe and adjust to the shape and size of the pipe. We propose two types of vehicles: tractive and nontractive. Both types are based on two hinged arms. The tractive vehicle has a driving wheel at a hinge and two sphere bearings at the ends of the arms. The driving wheel rotates about the axis perpendicular to the plane in which the two arms move. The wheel can freely move sideways. The sphere bearings can move in all directions like ball casters. Since the stretch force of the arm to the pipe wall is generated mechanically by pulleys and a spring, the vehicle rests in the pipe by pressing the two arms in opposite directions where the diameter is the biggest and it moves according to the action of the driving wheel. Three wheels of the nontractive vehicle are sphere bearings. We analyze the shape geometry of the pipe to obtain stability conditions under which the vehicle can move, and we consider friction and gravity bringing the vehicle to rest in the pipe. We also analyze the kinematics and dynamics of locomotion, and we simulate locomotion of the vehicle in the plane in which the biggest diameter of the pipe is measured. The results of the simulation prove the self-adjustability of the vehicle to the shape and size of the pipe. Experimental results show that the vehicle can move and self-adjust in an inclined or twisted pipe with a deep angle. [ABSTRACT FROM AUTHOR]

The manipulator trajectory tracking control problem revolves around computing the torques to be applied to achieve accurate tracking. This problem has been extensively studied in simulations, but real-time results have been lacking in the robotics literature. In this paper, we present the experimental results of the real-time performance of model-based control algorithms. We compare the computed-torque control scheme with the feedforward dynamics compensation scheme. The feedforward scheme compensates for the manipulator dynamics in the feedforward path, whereas the computed-torque scheme uses the dynamics in the feedback loop for linearization and decoupling. The parameters in the dynamics model for the computed-torque and feedforward schemes were estimated by using an identification algorithm. Our experiments underscore the importance of including the off-diagonal terms of the manipulator inertia matrix in the torque computation. This observation is further supported by our analysis of the dynamics equations. The manipulator control schemes have been implemented on the CMU DD arm II with a sampling period of 2 ms. [ABSTRACT FROM AUTHOR]

Reports on integration-time-based, time domain computation that provides an area-efficient way to process image information by directly handling photo-created charge during photo-sensing. Charge-coupled device (CCD)-based range-finding image sensor; Test cell experiment; One-dimensional-array range-finding sensor.

The article discusses various reports published within the issue including one by Nico Cornelis, Bastian Leibe, Kurt Cornelis and Luc Van Gool on a complete system that allows a futuristic looking stereo video from a moving car into a model from which a virtual drive-through of a city can be seen and another by Jiang Zheng and Min Shi on depth estimation approach for urban streets.

Detailed and accurate characteristics of preimplantation embryos are fundamental for a deep understanding of their development. Recent studies indicate that various geometric features of cells, such as size, shape, volume, and position play a significant role in embryo growth. However, a quantitative assessment of these characteristics first needs a segmentation of the individual cells. The manual separation and labeling of cells is extremely inefficient, and an automated approach is highly desirable. This paper presents an automatic method for early stage embryo segmentation into its constituent cells and membranes using three-dimensional (3D) data. The input data consist of two Z-stacks of fluorescence microscope images containing nuclei and membranes. The method uses a 3D level set segmentation algorithm. Its evaluation is based on a dataset composed of 20 mouse embryos, each with 4-32 blastomeres. Segmentation accuracy was evaluated by calculating F-scores with ground truth obtained by manually labeling desired regions. We also compared output of our method with the one acquired with a watershed algorithm. The proposed approach was able to achieve more than $$90\%$$ accuracy for embryos with 4 and 8 cells, while for embryos with higher number of cells it was lower, reaching $$75\%$$ for 32-cell embryo. [ABSTRACT FROM AUTHOR]

To enable real-time, person-independent 3D registration from 2D video, we developed a 3D cascade regression approach in which facial landmarks remain invariant across pose over a range of approximately 60°. From a single 2D image of a person's face, a dense 3D shape is registered in real time for each frame. The algorithm utilizes a fast cascade regression framework trained on high-resolution 3D face-scans of posed and spontaneous emotion expression. The algorithm first estimates the location of a dense set of landmarks and their visibility, then reconstructs face shapes by fitting a part-based 3D model. Because no assumptions are required about illumination or surface properties, the method can be applied to a wide range of imaging conditions that include 2D video and uncalibrated multi-view video. The method has been validated in a battery of experiments that evaluate its precision of 3D reconstruction, extension to multi-view reconstruction, temporal integration for videos and 3D head-pose estimation. Experimental findings strongly support the validity of real-time, 3D registration and reconstruction from 2D video. The software is available online at http://zface.org . [ABSTRACT FROM AUTHOR]

This paper presents a convenient self-calibration method for an inertial measurement unit (IMU) using matrix factorization. Using limited information about applied loads (accelerations or angular rates) available from natural references, the proposed method can linearly solve all the parameters of an IMU in any configuration of its inertial components. Our factorization-based calibration method exploits the bilinear form of an IMU measurement, which is the product of intrinsic calibration parameters and exerted loads. For a redundant IMU, we prove that partial knowledge of the loads, such as magnitude, can produce a linear solution space for a proper decomposition of the measurement. Theoretical analysis on this linear space reveals that a 1-D null space should be considered when load magnitudes are all equal (e.g., gravity loads). Degenerate load distributions are also geometrically identified to avoid singular measurement collection. Since a triad IMU has a lower number of sensor components than a 4-D parameter space, we propose an iterative factorization in which only initial bias is required. A wide convergence region of the bias can provide an automatic setting of the initial bias as the mean of the measurements. Performance of the proposed method is evaluated with respect to various noise levels and constraint types. Self-calibration capability is demonstrated using natural references, which are gravity for accelerometers and image stream from an attached camera for gyroscopes. Calibration results are globally optimal and identical to those of nonlinear optimization. [ABSTRACT FROM AUTHOR]

We propose a semi-Markov model trained in a max-margin learning framework for mitosis event segmentation in large-scale time-lapse phase contrast microscopy image sequences of stem cell populations. Our method consists of three steps. First, we apply a constrained optimization based microscopy image segmentation method that exploits phase contrast optics to extract candidate subsequences in the input image sequence that contains mitosis events. Then, we apply a max-margin hidden conditional random field (MM-HCRF) classifier learned from human-annotated mitotic and nonmitotic sequences to classify each candidate subsequence as a mitosis or not. Finally, a max-margin semi-Markov model (MM-SMM) trained on manually-segmented mitotic sequences is utilized to reinforce the mitosis classification results, and to further segment each mitosis into four predefined temporal stages. The proposed method outperforms the event-detection CRF model recently reported by Huh as well as several other competing methods in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells. For mitosis detection, an overall precision of 95.8% and a recall of 88.1% were achieved. For mitosis segmentation, the mean and standard deviation for the localization errors of the start and end points of all mitosis stages were well below 1 and 2 frames, respectively. In particular, an overall temporal location error of 0.73 \pm 1.29 frames was achieved for locating daughter cell birth events. [ABSTRACT FROM AUTHOR]

When a camera rotates rapidly or shakes severely, a conventional KLT (Kanade–Lucas–Tomasi) feature tracker becomes vulnerable to large inter-image appearance changes. Tracking fails in the KLT optimization step, mainly due to an inadequate initial condition equal to final image warping in the previous frame. In this paper, we present a gyro-aided feature tracking method that remains robust under fast camera–ego rotation conditions. The knowledge of the camera’s inter-frame rotation, obtained from gyroscopes, provides an improved initial warping condition, which is more likely within the convergence region of the original KLT. Moreover, the use of an eight-degree-of-freedom affine photometric warping model enables the KLT to cope with camera rolling and illumination change in an outdoor setting. For automatic incorporation of sensor measurements, we also propose a novel camera/gyro auto-calibration method which can be applied in an in-situ or on-the-fly fashion. Only a set of feature tracks of natural landmarks is needed in order to simultaneously recover intrinsic and extrinsic parameters for both sensors. We provide a simulation evaluation for our auto-calibration method and demonstrate enhanced tracking performance for real scenes with aid from low-cost microelectromechanical system gyroscopes. To alleviate the heavy computational burden required for high-order warping, our publicly available GPU implementation is discussed for tracker parallelization. [ABSTRACT FROM AUTHOR]

Abstract: We investigated how engineered gradients of exogenous growth factors, immobilized to an extracellular matrix material, influence collective guidance of stem cell populations over extended time (>1 day) and length (>1 mm) scales in vitro. Patterns of low-to-high, high-to-low, and uniform concentrations of heparin-binding epidermal growth factor-like growth factor were inkjet printed at precise locations on fibrin substrates. Proliferation and migration responses of mesenchymal stem cells seeded at pattern origins were observed with time-lapse video microscopy and analyzed using both manual and automated computer vision-based cell tracking techniques. Based on results of established chemotaxis studies, we expected that the low-to-high gradient would most effectively direct cell guidance away from the cell source. All printed patterns, however, were found to direct net collective cell guidance with comparable responses. Our analysis revealed that collective “cell diffusion” down a cell-to-cell confinement gradient originating at the cell starting lines and not the net sum of directed individual cell migration up a growth factor concentration gradient is the principal driving force for directing mesenchymal stem cell population outgrowth from a cell source. These results suggest that simple uniform distributions of growth factors immobilized to an extracellular matrix material may be as effective in directing cell migration into a wound site as more complex patterns with concentration gradients. [Copyright &y& Elsevier]

Measurement of human body is useful for ergonomic design in manufacturing. We aim to accurately measure the shapes of human feet for the design of shoes, for which measuring the dynamic shape of the foot in motion is important because the foot deforms while walking or running. In this paper, we propose a method for measuring the anatomical feature cross-sections of the foot while walking. The dynamic shape of feature cross-section has never been measured, though those features are very basic and popular. Our proposed method is based on the multi-view stereo method. The target cross-sections are painted in individual colors (red, green, yellow and blue). Several nonlinear conditions are introduced in the process to find the consistent correspondence in all images. We show that the proposed method for dynamic measurement achieved the desired accuracy similar to existing 3D static foot scanners. [ABSTRACT FROM AUTHOR]

Dynamic weather such as rain and snow causes complex spatio-temporal intensity fluctuations in videos. Such fluctuations can adversely impact vision systems that rely on small image features for tracking, object detection and recognition. While these effects appear to be chaotic in space and time, we show that dynamic weather has a predictable global effect in frequency space. For this, we first develop a model of the shape and appearance of a single rain or snow streak in image space. Detecting individual streaks is difficult even with an accurate appearance model, so we combine the streak model with the statistical characteristics of rain and snow to create a model of the overall effect of dynamic weather in frequency space. Our model is then fit to a video and is used to detect rain or snow streaks first in frequency space, and the detection result is then transferred to image space. Once detected, the amount of rain or snow can be reduced or increased. We demonstrate that our frequency analysis allows for greater accuracy in the removal of dynamic weather and in the performance of feature extraction than previous pixel-based or patch-based methods. We also show that unlike previous techniques, our approach is effective for videos with both scene and camera motions. [ABSTRACT FROM AUTHOR]

Recovery of three dimensional (3D) shape and motion of non-static scenes from a monocular video sequence is important for applications like robot navigation and human computer interaction. If every point in the scene randomly moves, it is impossible to recover the non-rigid shapes. In practice, many non-rigid objects, e.g. the human face under various expressions, deform with certain structures. Their shapes can be regarded as a weighted combination of certain shape bases. Shape and motion recovery under such situations has attracted much interest. Previous work on this problem (Bregler, C., Hertzmann, A., and Biermann, H. 2000. In Proc. Int. Conf. Computer Vision and Pattern Recognition; Brand, M. 2001. In Proc. Int. Conf. Computer Vision and Pattern Recognition; Torresani, L., Yang, D., Alexander, G., and Bregler, C. 2001. In Proc. Int. Conf. Computer Vision and Pattern Recognition) utilized only orthonormality constraints on the camera rotations ( rotation constraints). This paper proves that using only the rotation constraints results in ambiguous and invalid solutions. The ambiguity arises from the fact that the shape bases are not unique. An arbitrary linear transformation of the bases produces another set of eligible bases. To eliminate the ambiguity, we propose a set of novel constraints, basis constraints, which uniquely determine the shape bases. We prove that, under the weak-perspective projection model, enforcing both the basis and the rotation constraints leads to a closed-form solution to the problem of non-rigid shape and motion recovery. The accuracy and robustness of our closed-form solution is evaluated quantitatively on synthetic data and qualitatively on real video sequences. [ABSTRACT FROM AUTHOR]

In Part I of this paper we developed the theory and algorithms for performing Shape-From-Silhouette (SFS) across time. In this second part, we show how our temporal SFS algorithms can be used in the applications of human modeling and markerless motion tracking. First we build a system to acquire human kinematic models consisting of precise shape (constructed using the temporal SFS algorithm for rigid objects), joint locations, and body part segmentation (estimated using the temporal SFS algorithm for articulated objects). Once the kinematic models have been built, we show how they can be used to track the motion of the person in new video sequences. This marker-less tracking algorithm is based on the Visual Hull alignment algorithm used in both temporal SFS algorithms and utilizes both geometric (silhouette) and photometric (color) information. [ABSTRACT FROM AUTHOR]

Shape-From-Silhouette (SFS) is a shape reconstruction method which constructs a 3D shape estimate of an object using silhouette images of the object. The output of a SFS algorithm is known as the Visual Hull (VH). Traditionally SFS is either performed on static objects, or separately at each time instant in the case of videos of moving objects. In this paper we develop a theory of performing SFS across time: estimating the shape of a dynamic object (with unknown motion) by combining all of the silhouette images of the object over time. We first introduce a one dimensional element called a Bounding Edge to represent the Visual Hull. We then show that aligning two Visual Hulls using just their silhouettes is in general ambiguous and derive the geometric constraints (in terms of Bounding Edges) that govern the alignment. To break the alignment ambiguity, we combine stereo information with silhouette information and derive a Temporal SFS algorithm which consists of two steps: (1) estimate the motion of the objects over time (Visual Hull Alignment) and (2) combine the silhouette information using the estimated motion (Visual Hull Refinement). The algorithm is first developed for rigid objects and then extended to articulated objects. In the Part II of this paper we apply our temporal SFS algorithm to two human-related applications: (1) the acquisition of detailed human kinematic models and (2) marker-less motion tracking. [ABSTRACT FROM AUTHOR]

Three-dimensional (3D) kinematic models are widely-used in video-based figure tracking. We show that these models can suffer from singularities when motion is directed along the viewing axis of a single camera. The single camera case is important because it arises in many interesting applications, such as motion capture from movie footage, video surveillance, and vision-based user-interfaces. We describe a novel two-dimensional scaled prismatic model (SPM) for figure registration. In contrast to 3D kinematic models, the SPM has fewer singularity problems and does not require detailed knowledge of the 3D kinematics. We fully characterize the singularities in the SPM and demonstrate tracking through singularities using synthetic and real examples. We demonstrate the application of our model to motion capture from movies. Fred Astaire is tracked in a clip from the film "Shall We Dance". We also present the use of monocular hand tracking in a 3D user-interface. These results demonstrate the benefits of the SPM in tracking with a single source of video. [ABSTRACT FROM AUTHOR]

The complete set of measurements that could ever be used by a passive 3D vision algorithm is the plenoptic function or light-field. We give a concise characterization of when the light-field of a Lambertian scene uniquely determines its shape and, conversely, when the shape is inherently ambiguous. In particular, we show that stereo computed from the light-field is ambiguous if and only if the scene is radiating light of a constant intensity (and color, etc.) over an extended region. [ABSTRACT FROM AUTHOR]

This article presents a method to recover the full-motion (3 rotations and 3 translations) of the head from an input video using a cylindrical head model. Given an initial reference template of the head image and the corresponding head pose, the head model is created and full head motion is recovered automatically. The robustness of the approach is achieved by a combination of three techniques. First, we use the iteratively reweighted least squares (IRLS) technique in conjunction with the image gradient to accommodate nonrigid motion and occlusion. Second, while tracking, the templates are dynamically updated to diminish the effects of self-occlusion and gradual lighting changes and to maintain accurate tracking even when the face moves out of view of the camera. Third, to minimize error accumulation inherent in the use of dynamic templates, we re-register images to a reference template whenever head pose is close to that in the template. The performance of the method, which runs in real time, was evaluated in three separate experiments using image sequences (both synthetic and real) for which ground truth head motion was known. The real sequences included pitch and yaw as large as 40° and 75°, respectively. The average recovery accuracy of the 3D rotations was about 3°. In a further test, the method was used as part of a facial expression analysis system intended for use with spontaneous facial behavior in which moderate head motion is common. Image data consisted of 1-min of video from each of 10 subjects while engaged in a two-person interview. The method successfully stabilized face and eye images allowing for 98% accuracy in automatic blink recognition. © 2003 Wiley Periodicals, Inc. Int J Imaging Syst Technol 13: 85–94, 2003; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/ima.10048 [ABSTRACT FROM AUTHOR]

To build a VR training system for visuomotor skills, an image displayed by a visual interface should be correctly registered to a haptic interface so that the visual sensation and the haptic sensation are both spatially and temporally consistent. In other words, it is desirable that what you see is what you feel (WYSIWYF). In this paper, we propose a method that can realize correct visual/haptic registration, namely WYSIWYF, by using a vision-based, object-tracking technique and a video-keying technique. Combining an encountered-type haptic device with a motioncommand-type haptic rendering algorithm makes it possible to deal with two extreme cases (free motion and rigid constraint). This approach provides realistic haptic sensations, such as free-to-touch and move-and-collide. We describe a first prototype and illustrate its use with several demonstrations. The user encounters the haptic device exactly when his or her hand reaches a virtual object in the display. Although this prototype has some remaining technical problems to be solved, it serves well to show the validity of the proposed approach. [ABSTRACT FROM AUTHOR]

Presents a neural network-based upright frontal face detection system. Information on the work of face detection researchers; Description of the system; Details on arbitration among multiple networks.

We present an approach to capture the 3D motion of a group of people engaged in a social interaction. The core challenges in capturing social interactions are: (1) occlusion is functional and frequent; (2) subtle motion needs to be measured over a space large enough to host a social group; (3) human appearance and configuration variation is immense; and (4) attaching markers to the body may prime the nature of interactions. The Panoptic Studio is a system organized around the thesis that social interactions should be measured through the integration of perceptual analyses over a large variety of view points. We present a modularized system designed around this principle, consisting of integrated structural, hardware, and software innovations. The system takes, as input, 480 synchronized video streams of multiple people engaged in social activities, and produces, as output, the labeled time-varying 3D structure of anatomical landmarks on individuals in the space. Our algorithm is designed to fuse the “weak” perceptual processes in the large number of views by progressively generating skeletal proposals from low-level appearance cues, and a framework for temporal refinement is also presented by associating body parts to reconstructed dense 3D trajectory stream. Our system and method are the first in reconstructing full body motion of more than five people engaged in social interactions without using markers. We also empirically demonstrate the impact of the number of views in achieving this goal. [ABSTRACT FROM AUTHOR]

An introduction to a section of the journal is presented which includes articles on topics including an heterogeneous two stream deep neural network to process spatial, facial attribute estimation and multi-task, deep learning architecture for the estimation of multiple facial attributes.

Automatic cell segmentation can hardly be flawless due to the complexity of image data particularly when time-lapse experiments last for a long time without biomarkers. To address this issue, we propose an interactive cell segmentation method by classifying feature-homogeneous superpixels into specific classes, which is guided by human interventions. Specifically, we propose to actively select the most informative superpixels by minimizing the expected prediction error which is upper bounded by the transductive Rademacher complexity, and then query for human annotations. After propagating the user-specified labels to the remaining unlabeled superpixels via an affinity graph, the error-prone superpixels are selected automatically and request for human verification on them; once erroneous segmentation is detected and subsequently corrected, the information is propagated efficiently over a gradually-augmented graph to un-labeled superpixels such that the analogous errors are fixed meanwhile. The correction propagation step is efficiently conducted by introducing a verification propagation matrix rather than rebuilding the affinity graph and re-performing the label propagation from the beginning. We repeat this procedure until most superpixels are classified into a specific category with high confidence. Experimental results performed on three types of cell populations validate that our interactive cell segmentation algorithm quickly reaches high quality results with minimal human interventions and is significantly more efficient than alternative methods, since the most informative samples are selected for human annotation/verification early. [ABSTRACT FROM AUTHOR]

Various object representations have been widely used for many tasks such as object detection, recognition, and tracking. Most of them requires an intensive training process on large database which is collected in advance, and it is hard to add models of a previously unobserved object which is not in the database. In this paper, we investigate how to create a representation of a new and unknown object online, and how to apply it to practical applications like object detection and tracking. To make it viable, we utilize a sensor fusion approach using a camera and a single-line scan LIDAR. The proposed representation consists of an approximated geometry model and a viewpoint-scale invariant appearance model which makes to extremely simple to match the model and the observation. This property makes it possible to model a new object online, and provides a robustness to viewpoint variation and occlusion. The representation has benefits of both an implicit model (referred to as a view-based model) and an explicit model (referred to as a shape-based model). Intensive experiments using synthetic and real data demonstrate the viability of the proposed object representation in both modeling and detecting/tracking objects. [ABSTRACT FROM AUTHOR]

We propose a data-driven approach to estimate the likelihood that an image segment corresponds to a scene object (its “objectness”) by comparing it to a large collection of example object regions. We demonstrate that when the application domain is known, for example, in our case activity of daily living (ADL), we can capture the regularity of the domain specific objects using millions of exemplar object regions. Our approach to estimating the objectness of an image region proceeds in two steps: $1)$ finding the exemplar regions that are the most similar to the input image segment; $2)$ calculating the objectness of the image segment by combining segment properties, mutual consistency across the nearest exemplar regions, and the prior probability of each exemplar region. In previous work, parametric objectness models were built from a small number of manually annotated objects regions, instead, our data-driven approach uses $5$ million object regions along with their metadata information. Results on multiple data sets demonstrates our data-driven approach compared to the existing model based techniques. We also show the application of our approach in improving the performance of object discovery algorithms. [ABSTRACT FROM AUTHOR]