Search

In surveillance applications, keyframes are usually used to summarize important video contents. However, most traditional keyframe extraction approaches just select some video frames from the input video, while the information in these selected video frames are insufficient and cannot let the users perceive the events happened in the original video easily. In this paper, we propose a novel keyframe generating technique to condense the contents of a surveillance video clip captured by a static camera into a still picture (augmented keyframe). The augmented keyframe is a more meaningful keyframe augmented with representative objects, important contents (human faces, license plate, etc.), motion status (represented as icons) and some marks of the moving objects in a still picture. This new technique consists of two major phases: content extraction and content synthesis. Some testing results of the proposed augmented keyframe are also presented and compared with results obtained by the traditional keyframe approach.

Watershed segmentation has recently become a popular tool for image segmentation. There are two approaches to implementing watershed segmentation: immersion approach and toboggan simulation. Conceptually, the immersion approach can be viewed as an approach that starts from low altitude to high altitude and the toboggan approach as an approach that starts from high altitude to low altitude. The former seemed to be more popular recently (e.g., Vincent and Soille), but the latter had its own supporters (e.g., Mortensen and Barrett). It was not clear whether the two approaches could lead to exactly the same segmentation result and which approach was more efficient. In this paper, we present two "order-invariant" algorithms for watershed segmentation, one based on the immersion approach and the other on the toboggan approach. By introducing a special RIDGE label to achieve the property of order-invariance, we find that the two conceptually opposite approaches can indeed obtain the same segmentation result. When running on a Pentium-III PC, both of our algorithms require only less than 1/30 s for a 256 /spl times/ 256 image and 1/5 s for a 512 /spl times/ 512 image, on average. What is more surprising is that the toboggan algorithm, which is less well known in the computer vision community, turns out to run faster than the immersion algorithm for almost all the test images we have used, especially when the image is large, say, 512 /spl times/ 512 or larger. This paper also gives some explanation as to why the toboggan algorithm can be more efficient in most cases.

Our goal is to augment a panorama with object movies in a visually 3D-consistent way. Notice that a panorama is recorded as one single 2D image and an object movie (OM) is composed of a set of 2D images taken around a 3D object. The challenge is how to integrate the above two sources of 2D images in a 3D-consistent way so that the user can easily manipulate object movies in a panorama. To solve this problem, we adopt a purely image-based approach that does not have to reconstruct the geometric models of the 3D objects to be inserted in the panorama. A critical issue of this method is how to generate the novel views required for showing an OM in different places of a panorama, and we have proposed a view morphing technique, called t-DBVM, to solve this problem. Our experiments have shown that this purely image-based approach can effectively generate visually convincing OM-augmented panoramas. This method has great potential for many applications that require integration of panoramas and object movies, such as virtual malls, virtual museum, and interior design. Copyright © 2002 John Wiley & Sons, Ltd.

For showing 2D video contents on 3DTVs, 2D-to-3D conversion is required to convert 2D contents to 3D ones. In general, the conversion process consists of depth map generation, which estimates the 3D geometry of the scene, and rendering, which produces output stereo images. We propose a depth map refinement algorithm which uses adaptive decimation and guided interpolation to refine the depth map. The proposed approach eliminates unnecessary textures and keeps object boundaries on the depth map. In the refined depth map, the transition of depth values within an object is smooth, the object boundaries on the depth map are aligned to those on the input image, and the foreground can be clearly separated from the background. These characteristics on the depth map help to achieve better 3D visual perception. Compared to the state-of-the-art approaches, the blurriness and object boundary alignment are easier to be adjusted by the proposed two-stage operations, and the computational complexity of the proposed algorithm is much lower.

Hand tracking is a fundamental task in a gesture recognition system. Most previous works tracked the hand position on color images and relied heavily on skin color information. However, color information is very vulnerable to lighting variations and skin color varies across difference human races. Furthermore, one can not effectively discriminate faces or other skin-color-like objects from hands when using skin color detection. In this paper, we propose a hand tracking algorithm that uses depth images only, and also a hand click detection method to initialize the hand tracking automatically. We show that depth images suffice and are advantageous to real-time hand tracking. A region growing technique is applied to segment the hand region on depth images. Then a mean-shift based algorithm accurately locates the hand center in the segmented hand region. The experimental results show that the proposed tracking algorithm runs at 300+ FPS, and the average error of the tracked 3D hand positions is less than 1 centimeter. The proposed method enables a plethora of potential applications to natural Human-Computer Interaction (HCI), and is adequate for embedded systems of consumer electronics because of its low complexity and low bandwidth requirement.

High Efficiency Video Coding (HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on Video Coding (JCT-VC). In this paper, an overview of the motion vector coding techniques for HEVC is presented. Our three proposed coding tools for the motion vector predictor (MVP) in the Inter, Skip and Merge modes of HEVC are also presented, which includes a new location of the temporal MVP, a priority-based derivation method of spatial MVPs, and a derivation method of temporal MVPs. A combination of these three tools can achieve on average 1.3%, 1.8%, 1.2% and 2.2% bit rate reductions for high efficiency random access, low complexity random access, high efficiency low delay, and low complexity low delay, respectively.

Image-based rendering has been successfully used to display 3-D objects for many applications. A well-known example is the object movie, which is an image-based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object. In order to integrate image-based 3-D objects into a chosen scene (e.g., a panorama), one has to meet a hard challenge--to efficiently and effectively remove the background from the foreground object. This problem is referred to as multiview images (MVIs) segmentation. Another task requires MVI segmentation is image-based 3-D reconstruction using multiview images. In this paper, we propose a new method for segmenting MVI, which integrates some useful algorithms, including the well-known graph-cut image segmentation and volumetric graph-cut. The main idea is to incorporate the shape prior into the image segmentation process. The shape prior introduced into every image of the MVI is extracted from the 3-D model reconstructed by using the volumetric graph cuts algorithm. Here, the constraint obtained from the discrete medial axis is adopted to improve the reconstruction algorithm. The proposed MVI segmentation process requires only a small amount of user intervention, which is to select a subset of acceptable segmentations of the MVI after the initial segmentation process. According to our experiments, the proposed method can provide not only good MVI segmentation, but also provide acceptable 3-D reconstructed models for certain less-demanding applications.

Object movie (OM) is a popular technique for producing interactive 3D artifacts because of its simplicity in production and it photo-realistic ability to present the artifacts. At the same time, many stereoscopic vision techniques are developed for a variety of applications. However, the traditional approach for generating binocular object movies require duplicate effort compared with monocular ones both in the process of acquisition and image processing. Therefore, we propose a framework to generate stereo OMs from monocular ones with the help of an automatically constructed 3D model from the monocular OM. Here, a new representation of the 3D model, named billboard clusters, is proposed for efficient generating binocular views. In order to obtain better results, a novel approach to extract view-independent texture is developed in this work. Besides, billboard clusters can be used to compress the storage capacity of OMs, and to perform relighting so that the binocular OMs can be well augmented into virtual environments with different lighting conditions. This paper describes the methods in detail and reports on its wide applications.

Research on creating a large, high-resolution, low-cost display system has become increasingly important due to the growing desire in many fields for bigger and better displays. The goal of this research is to build a seamless large-scale display system by tiling multiple projectors with the help of a pan-tilt-zoom camera. In order to achieve this goal, we took two tasks into consideration: geometric calibration and photometric calibration. Compared to the previous work, our method for geometric calibration is more accurate, thanks to the much higher resolution images acquired by combining several zoom-in images and doing lens correction for the camera at first. For photometric calibration, we achieve color uniformity among projectors by utilizing the same PTZ camera, which is calibrated once with a colorimeter in a factory. Furthermore, we adopted the technique of producing a high dynamic range image from several images of different exposures to increase the measurement accuracy of the camera. In our experiments, the average error of photometric measurement achieved by our method is less than the mean perceptibility tolerance. Our method has great potential for many applications that require large and high-resolution displays.

Panoramas and object movies are popular image-based techniques for modeling and rendering 3D scenes and objects. In this paper, we present a method which allows the authors of virtual exhibition systems to produce an augmented stereo panorama by interactively integrating stereo object movies into a stereo panorama. When navigating in the augmented stereo panorama with a stereoscopic display, the user can directly browse the stereo object movies that he is interested in. With augmented stereo panoramas, the user can enjoy more persuasive interaction with better depth perception. To our best knowledge, this paper is the first work to integrate stereoscopic image-based scenes with stereo object movies.

This paper proposes a new object movie (OM) segmentation method that incorporates shape priors into the segmentation algorithm. The shape prior introduced into every image of the OM is learned from the 3D model reconstructed by the volumetric graph cuts. Here, the constraint derived from the discrete medial axis is used to improve the reconstruction algorithm. Our segmentation method requires only a small amount of user intervention, which is to select a subset of acceptable segmentations of the OM after the initial segmentation process. Compared to other techniques, our method provides not only the better segmentation result but also the better 3D reconstruction result.

Object Movies (OMs) have been successfully used in many applications. However, the techniques for acquiring OMs still need to be improved if high-quality and efficient OMs are desired. In this paper, we present a method for calibrating a motorized object rig to facilitate the acquisition of OMs. We first apply the CPC kinematic model to formulate the 3D configuration of the device, and then propose a method to estimate the parameters of the CPC model of the device. Furthermore, a visual tool is provided for users to adjust the controllable axes of the rig according to the estimated results. After this calibration, more accurate camera parameters can be obtained and then be used for different purposes. In this work, we use the parameters to reconstruct, from an OM, the 3D model of the object, and then adjust the OM according to the center of the 3D model so that a high-quality OM can be obtained for rendering.

Many wall-size display systems are built to provide large-scale visualization. These systems may be quite successful for some limited applications, but are very inflexible, since these systems only have fixed display regions. This paper integrates steerable projectors whose beam can be moved under computer control onto a wall-size display system to strengthen its display ability. With the steerable projectors, the integrated display system, named Flexible Display, provide an extendable display region. This consists of a large-scale display region and several movable display regions, such that the integrated display system has great potential in the area of human-computer interaction and information visualization. This paper applies the Flexible Display to a virtual museum application to give the users fluent navigation experience. For the application, the Flexible Display provides the following functions: 1) intensity and resolution enhancement of sub-region of display wall, 2) information augmentation, and 3) “stepping user interfaces” for its viewers interacting with display wall.

We propose a Bayesian approach to video object segmentation, which consists of two stages. In the first stage, we partition the video data into a set of 3D watershed volumes, where each watershed volume is a series of corresponding 2D image regions. These 2D image regions are obtained by applying to each image frame the marker-controlled watershed segmentation. In the second stage, we use a Markov random field to model the spatio-temporal relationship among the 3D watershed volume. Then, the desired video objects can be extracted by merging watershed volumes having similar motion characteristics within a Bayesian framework Our experiments have shown that the proposed method has great potential in extracting moving objects from a video sequence.