Search

Suppose a set of arbitrary (unlabeled) images contains frequent occurrences of 2D objects from an unknown category. This paper is aimed at simultaneously solving the following related problems: 1) unsupervised identification of photometric, geometric, and topological properties of multiscale regions comprising instances of the 2D category, 2) learning a region-based structural model of the category in terms of these properties, and 3) detection, recognition, and segmentation of objects from the category in new images. To this end, each image is represented by a tree that captures a multiscale image segmentation. The trees are matched to extract the maximally matching subtrees across the set, which are taken as instances of the target category. The extracted subtrees are then fused into a tree union that represents the canonical category model. Detection, recognition, and segmentation of objects from the learned category are achieved simultaneously by finding matches of the category model with the segmentation tree of a new image. Experimental validation on benchmark data sets demonstrates the robustness and high accuracy of the learned category models when only a few training examples are used for learning without any human supervision. Index Terms--Object recognition, image segmentation tree, hierarchical object representation, unsupervised learning, graph matching, tree union.

A video containing multiple objects undergoing independent translational and rotational motions is analyzed through a combination of spatial- and frequency-domain representations. The Fourier transform of the sequence is used to estimate the multiple translations and rotations in a computationally efficient manner, which is also robust to local inaccuracies and global illumination changes. A novel algorithm is presented for the simultaneous extraction of all objects undergoing translation and the background via a least squares technique that takes place entirely in the Fourier domain. Spatial information is combined with the frequency domain object extraction results, to further refine them. For the case of rotational or combined, rotational and translational motions, the moving objects are segmented using purely spatial information. We show that the combined analysis takes advantage of the strengths of both representations, by providing reliable and computationally efficient motion estimates and object segmentation. The proposed algorithm is shown to be robust to local noise and occlusion, because of its global nature. Experiments are performed on synthetic and real video sequences to demonstrate the capabilities of our approach. Index Terms--Fourier transform (FT), motion segmentation, phase-based motion estimation, video analysis.

The analysis of periodic or repetitive motions is useful in many applications, such as the recognition and classification of human and animal activities. Existing methods for the analysis of periodic motions first extract motion trajectories using spatial information and then determine if they are periodic. These approaches are mostly based on feature matching or spatial correlation, which are often infeasible, unreliable, or computationally demanding. In this paper, we present a new approach, based on the time-frequency analysis of the video sequence as a whole. Multiple periodic trajectories are extracted and their periods are estimated simultaneously. The objects that are moving in a periodic manner are extracted using the spatial domain information. Experiments with synthetic and real sequences display the capabilities of this approach. Index Terms--Periodic motion analysis, time-frequency distributions, short term Fourier transform.

In augmented reality (AR) applications, accurately registering a virtual object with its real counterpart is a challenging problem. The size, depth, and geometry, as well as the physical attributes of the virtual object, have to be rendered precisely relative to a physical reference. This paper presents a systematic calibration process to address the registration challenge in a custom-designed AR system, which is based upon recent head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and our system configuration, we first present a computational model of the HMPD viewing system and requirements for the system calibration. Then, we describe, in detail, the calibration procedures to obtain estimates of unknown transformations, summarize the application of the estimates in a customized graphics rendering toolkit, and discuss the evaluation experiments and observations. Finally, the implementation of a testbed to demonstrate a successful registration is briefly described, and experimental results are presented. Index Terms--Augmented reality (AR), camera calibration, display calibration, head-mounted display (HMD), head-mounted projective display (HMPD).

Wide field of view (FOV) and high-resolution image acquisition is highly desirable in many vision-based applications. Several systems have reported the use of reflections off mirror pyramids to capture high-resolution, single-viewpoint, and wide-FOV images. Using a dual mirror pyramid (DMP) panoramic camera as an example, in this paper, we examine how the pyramid geometry, and the selection and placement of imager clusters can be optimized to maximize the overall panoramic FOV, sensor utilization efficiency, and image uniformity. The analysis can be generalized and applied to other pyramid-based designs. Index Terms--Panoramic camera, mirror pyramids, omnidirectional imaging, and catadioptric systems.

A mirror pyramid consists of a set of planar mirror faces arranged around an axis of symmetry and inclined to form a pyramid. By strategically positioning a number of conventional cameras around a mirror pyramid, the viewpoints of the cameras' mirror images can be located at a single point within the pyramid and their optical axes pointed in different directions to effectively form a virtual camera with a panoramic field of view. Mirror pyramid-based panoramic cameras have a number of attractive properties, including single-viewpoint imaging, high resolution, and video rate capture. It is also possible to place multiple viewpoints within a single mirror pyramid, yielding compact designs for simultaneous multiview panoramic video rate imaging. Nalwa [4] first described some of the basic ideas behind mirror pyramid cameras. In this paper, we analyze the general class of multiview panoramic cameras, provide a method for designing these cameras, and present experimental results using a prototype we have developed to validate single-pyramid multiview designs. We first give a description of mirror pyramid cameras, including the imaging geometry, and investigate the relationship between the placement of viewpoints within the pyramid and the cameras' field of view (FOV), using simulations to illustrate the concepts. A method for maximizing sensor utilization in a mirror pyramid-based multiview panoramic camera is also presented. Images acquired using the experimental prototype for two viewpoints are shown. Index Terms--Panoramic cameras, mirror pyramids, catadioptric systems, omnidirectional imaging and video capture, multiview panoramic imaging, stereoscopic cameras.

We describe a scheme that achieves spatially scalable coding of video by employing nonscalable video encoders (e.g., MPEG-2 main profile), along with a downsampler and an upsampler. The scheme is illustrated for the case of coding video at two resolutions. The enhancement layer is coded in two steps by first exploiting the spatial redundancy and then exploiting the temporal redundancy. Hence, the scheme has a separable implementation. Results are presented for five different sequences, coded for three different combinations of base and enhancement layer bit rates. When MPEG-2 main profile is used for the nonscalable encoders, the results obtained are comparable to the performance of MPEG-2 spatial scalability profile. Index Terms--Discrete cosine transform (DCT), HDTV, MPEG-2, multiresolution coding, scalable video compression, spatial scalability, subband decomposition.

Development of a lossless image-compression method is discussed. The method uses an optimal amount of segmentation information to exploit spatial redundancies in imaging data.

We present an algorithm for extracting and classifying two-dimensional motion in an image sequence based on motion trajectories. First, a multiscale segmentation is performed to generate homogeneous regions in each frame. Regions between consecutive frames are then matched to obtain two-view correspondences. Affine transformations are computed from each pair of corresponding regions to define pixel matches. Pixels matches over consecutive image pairs are concatenated to obtain pixel-level motion trajectories across the image sequence. Motion patterns are learned from the extracted trajectories using a time-delay neural network. We apply the proposed method to recognize 40 hand gestures of American Sign Language. Experimental results show that motion patterns of hand gestures can be extracted and recognized accurately using motion trajectories. Index Terms--Motion segmentation, motion analysis, motion trajectory, American Sign Language, hand gesture recognition, time-delay neural network.

Images containing faces are essential to intelligent vision-based human computer interaction, and research efforts in face processing include face recognition, face tracking, pose estimation, and expression recognition. However, many reported methods assume that the faces in an image or an image sequence have been identified and localized. To build fully automated systems that analyze the information contained in face images, robust and efficient face detection algorithms are required. Given a single image, the goal of face detection is to identify all image regions which contain a face regardless of its three-dimensional position, orientation, and lighting conditions. Such a problem is challenging because faces are nonrigid and have a high degree of variability in size, shape, color, and texture. Numerous techniques have been developed to detect faces in a single image, and the purpose of this paper is to categorize and evaluate these algorithms. We also discuss relevant issues such as data collection, evaluation metrics, and benchmarking. After analyzing these algorithms and identifying their limitations, we conclude with several promising directions for future research. Index Terms--Face detection, face recognition, object recognition, view-based recognition, statistical pattern recognition, machine learning.

In this paper, we report on our efforts toward developing a multiscale segmentation based video compression algorithm aimed at very low bid-rate applications such as video teleconferencing and video phones. We introduce novel techniques for multiscale segmentation based motion compensation and residual coding. Our region based forward motion compensation strategy (in terms of direction of motion vector, which is from the previous frame to the current frame) regulates the size and number of regions used, by pruning a multiscale segmentation of video frames. Since regions used for motion compensation are obtained by segmenting the previous decoded frame, the shape of the regions need not be transmitted to the decoder. Furthermore, our hierarchical motion compensation strategy refines an initial region level, coarse motion field to obtain a dense motion field which provides pixel level motion vectors. The refinement procedure does not require any additional information to be transmitted. This motion compensation technique effectively addresses the problem of dealing with 'holes' and 'overlapping regions' which are inherent to forward motion compensation. Residual coding is performed using a novel method which exploits the fact that the energy of the residual resulting from motion compensations is concentrated in a priori predictable positions. We will show that this residual coding technique can also be extrapolated to improve the performance of coders using a block based motion compensation strategy. A fusion of these concepts leads to a gain of 2-3 dB in peak signal-to-noise ratio, apart from significant perceptual improvement, over a generic video coding algorithm using a block based motion compensation strategy (such as H.261 or H.263) for a variety of test sequences. Index Terms: Hierarchical motion compensation, low bid-rate video compression, multiscale segmentation, residual coding

An analytically tractable potential field model of free space is presented in this paper. The model assumes that the border of every two-dimensional (2-D) region is uniformly charged. It is shown that the potential and the resulting repulsion (force and torque) between polygonal regions can be calculated in closed form. By using the Newtonian potential function, collision avoidance between object and obstacle thus modeled is guaranteed in a path planning problem. A local planner is developed for finding object paths going through narrow areas of free space where the obstacle avoidance is most important. Simulation results show that not only does individual object configuration of a path obtained with the proposed approach avoid obstacles effectively, the configurations also connect smoothly into a path. Index Terms - Artificial potential field, free-space representation, obstacle avoidance, path planning.

This paper presents a new approach to hierarchical clustering of point patterns. Two algorithms for hierarchical location- and density-based clustering are developed. Each method groups points such that maximum intracluster similarity and intercluster dissimilarity are achieved for point locations or point separations. Performance of the clustering methods is compared with four other methods. The approach is applied to a two-step texture analysis, where points represent centroid and average color of the regions in image segmentation. Index Terms - Point patterns, clustering, hierarchy of clusters, spatially interleaved clusters, density-based clustering, location-based clustering.

A transitory image sequence is one in which no scene element is visible through the entire sequence. When a camera system scans a scene which cannot be covered by a single view, the image sequence is transitory. This article deals with some major theoretical and algorithmic issues associated with the task of estimating structure and motion from transitory image sequences. It is shown that integration with a transitory sequence has properties that are very different from those with a non-transitory one. Two representations, world-centered (WC) and camera-centered (CC), behave very differently with a transitory sequence. The asymptotic error rates derived in this article indicate that one representation is significantly superior to the other, depending on whether one needs camera-centered or world-centered estimates. To establish the tightness of these error rates, it has been shown that these reachable error rates are in fact the lowest possible given by a theoretical lower error bound, the Cramer-Rao error bound. Based on these results, we introduce an efficient "cross-frame" estimation technique for the CC representation. For the WC representation, our analysis indicates that a good technique should be based on camera global pose instead of interframe motions. In addition to testing with synthetic data, rigorous experiments were conducted with real-image sequences taken by a fully calibrated camera system. The comparison of the experimental results with the ground truth has demonstrated that a good accuracy can be obtained from transitory image sequences. Index Terms - Motion analysis, struction from motion, image sequences, optimal estimation, Cramer-Rao bound, optical flow.

A new nonlinear transform has been developed to solve the problem of multiscale image segmentation. All of its parameters are automatically selected, therefore eliminating the need for user-specified parameters. The automatic selection of both homogeneity and spatial scales also eliminates the need to make restrictive assumptions about the geometric of homogeneity properties of the structure.

This paper is concerned with efficient derivation of the medial axis transform of a two-dimensional polygonal region. Instead of using the shortest distance to the region border, a potential field model is used for computational efficiency. The region border is assumed to be charged and the valleys of the resulting potential field are used to estimate the axes for the medial axis transform. The potential valleys are found by following force field, thus, avoiding two-dimensional search. The potential field is computed in closed form using the equations of the border segments. The simple Newtonian potential is shown to be inadequate for this purpose. A higher order potential is defined which decays faster with distance than as inverse of distance. It is shown that as the potential order becomes arbitrarily large, the axes approach those computed using the shortest distance to the border. Algorithms are given for the computation of axes, which can run in linear parallel time for part of the axes having initial guesses. Experimental results are presented for a number of examples. Index Terms - Generalized potential, Newtonian potential, topology, medial axis, symmetric axis transform, skeletonization, distance transform.

This paper describes a new transform to extract image regions at all geometric and photometric scales. It is argued that linear approaches such as convolution and matching have the fundamental shortcoming that they require a priori models of region shape. The proposed transform avoids this limitation by letting the structure emerge, bottom-up, from interactions among pixels, in analogy with statistical mechanics and particle physics. The transform involves global computations on pairs of pixels followed by vector integration of the results, rather than scalar and local linear processing. An attraction force field is computed over the image in which pixels belonging to the same region are mutually attracted and the region is characterized by a convergent flow. It is shown that the transform possesses properties that allow multiscale segmentation, or extraction of original, unblurred structure at all different geometric and photometric scales present in the image. This is in contrast with much of the previous work wherein multiscale structure is viewed as the smoothed structure in a multiscale decimation of image signal. Scale is an integral parameter of the force computation, and the number and values of scale parameters associated with the image can be estimated automatically. Regions are detected at all, a priori unknown, scales resulting in automatic construction of a segmentation tree, in which each pixel is annotated with descriptions of all the regions it belongs to. Although some of the analytical properties of the transform are presented for piecewise constant images, it is shown that the results hold for more general images, e.g., those containing noise and shading. Thus the proposed method is intended as a solution to the problem of multiscale, integrated edge and region detection, or low-level image segmentation. Experimental results with synthetic and real images are given to demonstrate the properties and segmentation performance of the transform. Index Terms - Image segmentation, representation, scale-space, edge detection, region detection, perceptual structure, pyramids, medial axis, nonlinear image analysis, texture.

This paper is concerned with three-dimensional interpretation of image sequences showing multiple objects in motion. Each object exhibits smooth motion except at certain time instants when a motion discontinuity may occur. The objects are assumed to contain point features which are detected as the images are acquired. Estimating feature trajectories in the first two frames amounts to feature matching. As more images are acquired, existing trajectories are extended. Both initial detection and extension of trajectories are done by enforcing pertinent constraints from among the following: similarity of image plane arrangement of neighboring features, smoothness of three dimensional motion, and smoothness of image plane motion. The constraints are incorporated into energy functions which are minimized using two-dimensional Hopfield networks. Wrong matches that result from convergence to local minima are eliminated using a one-dimensional Hopfield like network. Experimental results on several image sequences are shown.

This paper presents necessary and sufficient conditions for uniquely determining the motion and structure of a planar surface from any number of monocular images. First, we show an uncertain situation for the two-view plane motion problem in which an infinite number of solutions may result. Although this situation rarely occurs in real situations, it does raise visual ambiguities under certain circumstances, e.g., when some images are observed through a mirror. The understanding of this situation therefore help analyze reflected images which are often seen in daily life. Then, we present necessary and sufficient conditions for a unique solution of plane motion and structure from any number of views. These algorithm-independent conditions enhance our understanding about the problem of estimating plane motion and structure from image sequences and may be used to determine the uniqueness of the solution of the motion and structure of a planar surface.

This correspondence presents a matching algorithm for obtaining feature point correspondences across images containing rigid objects undergoing different motions. First point features are detected using newly developed feature detectors. Then a variety of constraints are applied starting with simplest and following with more informed ones. First, an intensity-based matching algorithm is applied to the feature points to obtain unique point correspondences. This is followed by the application of a sequence of newly developed heuristic tests involving geometry, rigidity, and disparity. The geometric tests match two-dimensional geometrical relationships among the feature points, the rigidity test enforces the three dimensional rigidity of the object, and the disparity test ensures that no matched feature point in an image could be rematched with another feature, if reassigned another disparity value associated with another matched pair or an assumed match on the epipolar line. The computational complexity is proportional to the numbers of detected feature points in the two images. Experimental results with indoor and outdoor images are presented, which show that the algorithm yields only correct matches for scenes containing rigid objects. Index Terms - Disparity, feature detection, geometric consistency, matching, multiresolution, point features, rigidity.

Much research has emphasized stereo disparity as a source of depth information. To a lesser extent, camera vergence and lens focus have also been investigated for their utility in depth recovery. Each of these visual cues exhibits shortcomings when used individually in the sense that none alone can be used to reconstruct surfaces for real scenes that often cover a wide field of view and a large range of depth. This paper presents an approach to integration of these cues that attempts to exploit their complementary strengths and weaknesses through active control of camera focus and orientations. In addition, the aperture and zoom settings of the cameras are controlled. The result is an active vision system that dynamically and cooperatively interleaves image acquisition with surface estimation. A dense composite map of a single contiguous surface is synthesized by automatically scanning the surface and combining estimates of adjacent, local surface patches. This problem is formulated as one of minimizing a pair of objective functions. The first such function is concerned with the selection of a target for fixation. The second objective function guides the surface estimation process in the vicinity of the fixation point. Calibration parameters of the cameras are treated as variables during optimization, thus making camera calibration an integral, flexible component of surface estimation. An implementation of this method is described, and a performance evaluation of the system is presented. An average absolute error of less than 0.15% in estimated depth was achieved for a large surface having a depth of approximately 2 m.