Search

We propose a novel framework for upconversion of depth video resolution in both spatial and time domains considering spatial and temporal coherences. Although the Time-of-Flight (TOF) sensor which is widely used in computer vision fields provides depth video in realtime, it also provides a low resolution and a low frame-rate depth video. We propose a cheaper solution that enhances depth video obtained from a TOF sensor by combining it with a Charge-coupled Device (CCD) camera in 3D contents which consist of 2D-plus-depth. Temporal fluctuation problems are also considered for temporally consistent framerate up-conversion. It is important to maintain temporal coherence in depth video, because temporal fluctuation problems may cause eye fatigue and increase bit rates on video coding. We propose a Motion Compensated Frame Interpolation (MCFI) using reliable and rich motion information from a CCD camera and 3-dimensional Joint Bilateral Up-sampling (3D JBU) extended into the temporal domain of depth video. Experimental results show that depth video obtained by the proposed method provides satisfactory quality.

Fast-forward playback enables viewers to scan through the video scene of interest efficiently. One approach to realize fast-forward playback is to employ a frame-skipping transcoder which transcodes only the frames required for playback at the desired fast speed. Various motion vector composition algorithms are used to compose the new motion vectors with reduced complexity. These algorithms do not work well for dropping a large number of frames, which is very common in fast-forward playback. In this paper, a new multiple-candidate vector selection algorithm (MCVS) is proposed to select a composed motion vector from a set of candidate motion vectors, which utilizes relevant areas in the target macroblock to ensure a reliable tracking process for motion vector composition. Experimental results show that the proposed MCVS can provide fast-forward playback through video transcoding with significant gain, in terms of rate-distortion performance, especially when a large speed-up factor is required.

Video frame resizing has received more and more research attention as contemporary video distributions need to deliver video to various receiving devices with different display resolutions. It is now becoming necessary for video distribution systems to be able to generate higher resolution video from lower resolution one for some end users. Current schemes in video up-sampling either only focused on improving visual quality with little or no improvement in objective quality or increasing the up-sampling accuracy by adaptively optimizing interpolation methods. Many such algorithms are typically quite complex and require substantial extra implementation efforts. In this letter, we propose a new hybrid DCT-Wiener-based interpolation scheme for video intra frame up-sampling without referencing the original high resolution video frames. This scheme takes full advantage of interpolation in both DCT domain and spatial domain and seamlessly integrates these two approaches to design an improved up-sampling filter. Experiments have been carried out to demonstrate that noticeable improvement in both objective and visual quality are obtained. With similar complexity, the proposed algorithm can achieve up to 4 dB gain in PSNR over fixed parameter Wiener filter-based interpolation, 4 dB gain over popular bicubic interpolation, and 1 dB gain over up-sampling scheme in DCT domain.

To manage large database system with video, effective video indexing and retrieval are required. A large number of video retrieval algorithms have been presented for frame-wise user query or video content query, whereas a few video identification algorithms have been proposed for video sequence query. In this paper, we propose an effective video identification algorithm for video sequence query that employs the Cauchy function of histograms between successive frames and the modified Hausdorff distance. To effectively match the video sequences with a low computational load, we make use of the key frames extracted by the cumulative Cauchy function and compare the set of key frames using the modified Hausdorff distance. Experimental results with several color video sequences show that the proposed algorithm for video identification yields remarkably higher performance than conventional algorithms such as Euclidean metric, and directed divergence methods.

We present an image-based rendering system to viewpoint-navigate through space and time of complex real-world, dynamic scenes. Our approach accepts unsynchronized, uncalibrated multivideo footage as input. Inexpensive, consumer-grade camcorders suffice to acquire arbitrary scenes, for example in the outdoors, without elaborate recording setup procedures, allowing also for hand-held recordings. Instead of scene depth estimation, layer segmentation or 3D reconstruction, our approach is based on dense image correspondences, treating view interpolation uniformly in space and time: spatial viewpoint navigation, slow motion or freeze-and-rotate effects can all be created in the same way. Acquisition simplification, integration of moving cameras, generalization to difficult scenes and space–time symmetric interpolation amount to a widely applicable virtual video camera system.

Automated video analysis lacks reliability when searching for unknown events in video data. The practical approach is to watch all the recorded video data, if applicable in fast-forward mode. In this paper we present a method to adapt the playback velocity of the video to the temporal information density, so that the users can explore the video under controlled cognitive load. The proposed approach can cope with static changes and is robust to video noise. First, we formulate temporal information as symmetrized Renyi divergence, deriving this measure from signal coding theory. Further, we discuss the animated visualization of accelerated video sequences and propose a physiologically motivated blending approach to cope with arbitrary playback velocities. Finally, we compare the proposed method with the current approaches in this field by experiments and a qualitative user study, and show its advantages over motion-based measures.

In 3D TV research, one approach is to employ multiple cameras for creating a 3D multi-view signal with the aim to make interactive free-viewpoint selection possible in 3D TV media. This paper explores a new rendering algorithm that enables to compute a free-viewpoint between two reference views from existing cameras. A unique property is that we perform forward warping for both texture and depth simultaneously. Advantages of our rendering are manyfold. First, resampling artifacts are filled in by inverse warping. Second, disocclusions are processed while omitting warping of edges at high discontinuities. Third, our disocclusion inpainting approach explicitly uses depth information. We obtain an average PSNR gain of 3 dB and 4.5 dB for the ‘Breakdancers’ and ‘Ballet’ sequences, respectively, compared recently published results. Moreover, experiments are performed using compressed video from surrounding cameras. The overall system quality is dominated by rendering quality and not by coding.

As high-definition video is broadly used in various system such as broadcast system and digital camcorder the proper method in order to improve the quality of high-definition video is needed. In this paper, we propose an efficient method to improve color and contrast of high-definition video. In order to apply the image enhancement method to high-definition video, scale-down video of high-definition video is used and the parameter for image enhancement method is computed from small size video. To enhance the color of high-definition video, we apply color constancy method. First, we separate the video into several scenes by cut detection method. Then, we apply color constancy to each scene with same parameter. To improve the contrast of high-definition video, we use union of original image and histogram equalized image, and weight is calculated based on sorting of histogram bins. Finally, the performance of proposed method is demonstrated in experiment section.

For the convenience of content-based video retrieval in large storage device, a new efficient video similarity search approach is proposed in this paper. To solve two challenging problems: similarity measurement and search method, a novel video feature computation of image characteristic code was proposed based on spatial-temporal distribution of video frame sequences. The video similarity is measured based on the calculation of the number of similar video components. For the scalable computing requirement, a new search method according to clustering index table was presented by index clustering. The experimental results in large database query tests show this method is efficient and effective for similar video search.

Local processing, which is a dominant type of processing in image and video applications, requires a huge computational power to be performed in real-time. However, processing locality, in space and/or in time, allows to exploit data parallelism and data reusing. Although it is possible to exploit these properties to achieve high performance image and video processing in multi-core processors, it is necessary to develop suitable models and parallel algorithms, in particular for non-shared memory architectures. This paper proposes an efficient and simple model for local image and video processing on non-shared memory multi-core architectures. This model adopts a single program multiple data approach, where data is distributed, processed and reused in an optimal way, regarding the data size, the number of cores and the local memory capacity. The model was experimentally evaluated by developing video local processing algorithms and programming the Cell Broadband Engine multi-core processor, namely for advanced video motion estimation and in-loop deblocking filtering. Furthermore, based on these experiences it is also addressed the main challenges of vectorization, and the reduction of branch mispredictions and computational load imbalances. The limits and advantages of the regular and adaptive algorithms are also discussed. Experimental results show the adequacy of the proposed model to perform local video processing, and that real-time is achieved even to process the most demanding parts of advanced video coding. Full-pixel motion estimation is performed over high resolution video (720×576 pixels) at a rate of 30 frames per second, by considering large search areas and five reference frames.

As the next generation of television, three-dimensional television (3DTV) is enkindling various new applications. Since the representation and coding of three-dimensional video has direct effect on successive transmission, synthesis and display, it becomes crucially important and receives a wide publicity. This paper gives an overview of existing 3D video representation, relevantly state-of-the-art coding standards and various coding approaches consisting of conventional stereo video, multi-view video, video plus depth, multi-view video plus depth and layered depth video.

Ultrahigh-definition television (UHDTV) is the next-generation ultrahigh-resolution video system with more than 4000 lines, and 7680 horizontal pixels × 4320 vertical lines at 60 f/s (frames per second) in video format. We propose a method of converting high-quality UHDTV video to digital cinema video that is 3840 horizontal pixels × 2160 vertical lines at 24 f/s. Here, it is important to detect motion vectors extremely accurately to convert the frame rate. We introduced two new algorithms to increase the accuracy of detecting motion vectors. First, we introduced a method of interpolation frames using 1/2 block-shift bi-directional motion estimation by expanding the time axis. The experimental results revealed that our new method produced higher quality video images than conventional methods of video conversion using forward-directional and overlapped block bi-directional motion estimation. Next, we focused on Bayer pattern sampling in existing UHDTV, and introduced a motion-compensated method of interpolation where the first algorithm utilizes the high-frequency components from green signals. The experimental results demonstrated that our proposed method achieves higher quality video conversions than conventional methods using only the first algorithm for motion-compensated interpolation, linear interpolation, and 5:2 frame skipping.

In modern times, video object segmentation has emerged as one of the most imperative and challenging area of research. The principal objective of video object segmentation is to facilitate content-based representation by extracting objects of interest from a series of consecutive video frames. Recently, a number of video object segmentation algorithms have been discussed and unfortunately most existing segmentation algorithms are not adequate and robust enough to process noisy video sequences. Competence of most segmentation techniques is affected by the presence of noise in frames which is a critical issue of edge preservation. This paper presents a novel video object segmentation approach for noisy color video sequences towards effective video retrieval. Initially, the noisy video frames are denoised using a strategy based on an enhanced sparse representation in transform domain. Afterwards, the background is estimated from the denoised frames using the Expectation Maximization (EM) algorithm. Then, the foreground objects i.e.) moving video objects are segmented with the aid of the novel approach presented. The biorthogonal wavelet transform and the L2 norm distance measure are employed in the foreground object segmentation. The experimental results demonstrate the effectiveness of the presented approach in segmenting the video objects from noisy color video sequences.

A technique for film grain noise extraction, modeling and synthesis is studied and applied to high-definition video coding in this paper. Film grain noise enhances the natural appearance of pictures in high-definition video and should be preserved in coded video. However, the coding of video contents with film grain noise is expensive. In previous art, it was proposed to enhance the coding performance by extracting film grain noise from the input video at the encoder as a preprocessing step, and by resynthesizing and adding it back to the decoded video at the decoder as a postprocessing step. In a novel implementation of this approach, we first remove film grain noise from image/video with a variational denoising approach without distorting its original content. Then, we present a parametric model (consisting of a small set of parameters) to generate film grain noise that is close to the actual one in terms of a couple of observed statistical properties, such as the power spectral density and the crosschannel spectral correlation. Under this framework, the coding gain of denoised video is higher while the visual quality of the final reconstructed video is well preserved. Experimental results are provided to demonstrate the superior performance of the proposed approach.

Transmission distortion analysis for video streams is a considerably challenging task. In this letter, a compressed-domain-based (CDB) transmission distortion model for precoded H.264/advanced video coding video streams is developed. Unlike the earlier schemes, which were based on pixel domain and required a complete decoding of the compressed video streams, the CDB model only requires some information on the video features, which can be directly extracted from the compressed video streams. Therefore, the complexity of the calculations is substantially reduced, which is well suited for real-time applications. More specifically, the model is applicable to the real-time transmission for precoded video streams, such as video on demand and mobile video. The experimental results demonstrate high accuracy of the model. Furthermore, an application example using the CDB model in resource allocation in real-time multiuser video communication reveals the applicability and effectiveness of the model.

Efficient browsing of long video sequences is a key tool in visual surveillance, e.g., for postevent video forensics, but can also be used for fast review of motion pictures and home videos. While frame skipping (fixed or adaptive) is straightforward to implement, its performance is quite limited. Although more efficient techniques have been developed, such as video summarization and video montage, they lose either the temporal or semantic context of events. A recently proposed method called video synopsis deals with some of these issues but involves multiple processing stages and is fairly complex. Video condensation, that we propose here, is novel in the way information is removed from the space-time video volume, is conceptually simple and relatively easy to implement. We introduce the concept of a video ribbon inspired by that of a seam recently proposed for image resizing. We recursively carve ribbons out by minimizing an activity-aware cost function using dynamic programming. The ribbon model we develop is flexible and permits an easy adjustment of the compromise between temporal condensation ratio and anachronism of events. We also propose sliding-window ribbon carving to handle streaming video and demonstrate the method's efficiency on motor and pedestrian traffic data.

With the phenomenal growth of the online and personal video repositories, an efficient and robust example-based video search solution is required to support applications like query by clip, query by capture, and repeated clip detection. In this letter, video sequences are represented as temporal trajectories via scaling and lower dimensional representation of the video frame luminance field, and a video trajectory indexing and matching scheme is developed to support video clip search. Simulation results demonstrate that the proposed approach achieves excellent performance in both response speed and precision-recall accuracy.

A depth image represents three-dimensional (3-D) scene information and is commonly used for depth image-based rendering (DIBR) to support 3-D video and free-viewpoint video applications. The virtual view is generally rendered by the DIBR technique and its quality depends highly on the quality of depth image. Thus, efficient depth coding is crucial to realize the 3-D video system. In this letter, we propose a depth reconstruction filter and depth down/up sampling techniques to improve depth coding performance. Experimental results demonstrate that the proposed methods reduce the bit-rate for depth coding and achieve better rendering quality.

Video shot transition identification constitutes an important computer vision research field, being applied, as an essential step, in many other digital video analysis domains: video scene detection, video compression, video indexing, video content retrieval and video object tracking. This paper approaches the video cut transition detection domain, providing a novel feature-based automatic identification method. We propose a feature extraction technique that uses 2D Gabor filtering, computing tridimensional image feature vectors for the video frames. Most shot cut detection techniques use a thresholding operation to discriminate between the inter-frame difference metric values and thus identify the video break points. Our identification approach is not threshold-based, using an automatic unsupervised distance classification procedure instead of a threshold. Thus, we provide a region-growing based classification approach, that proves to be very efficient in clustering the distances between feature vectors of consecutive frames. The two resulted distance classes determine a satisfactory video shot detection.

A novel adaptive interpolation filter structure for video coding with motion-compensated prediction is presented in this letter. The proposed scheme uses an independent directional adaptive interpolation filter for each sub-pixel location. The Wiener interpolation filter coefficients are computed analytically for each inter-coded frame at the encoder side and transmitted to the decoder. Experimental results show that the proposed method achieves up to 1.1 dB coding gain and a 15% average bit-rate reduction for high-resolution video materials compared to the standard nonadaptive interpolation scheme of H.264/AVC, while requiring 36% fewer arithmetic operations for interpolation. The proposed interpolation can be implemented in exactly 16-bit arithmetic, thus it can have important use-cases in mobile multimedia environments where the computational resources are severely constrained.

We propose a new approach to quantitatively analyze the rendering quality of image-based rendering (IBR) algorithms with depth information. The resulting error bounds for synthesized views depend on IBR configurations including the depth and intensity estimate errors, the scene geometry and texture, the number of actual cameras, their positions and resolution. Specifically, the IBR error is bounded by the summation of three terms, highlighting the impact of using multiple actual cameras, the impact of the noise level at the actual cameras, and the impact of the depth accuracy. We also quantify the impact of occlusions and intensity discontinuities. The proposed methodology is applicable to a large class of common IBR algorithms and can be applied locally. Experiments with synthetic and real scenes show that the developed error bounds accurately characterize the rendering errors. In particular, the error bounds correctly characterize the decay rates of synthesized views' mean absolute errors as O(lambda(-1)) and O(lambda(-2)), where lambda is the local density of actual samples, for 2-D and 3-D scenes, respectively. Finally, we discuss the implications of the proposed analysis on camera placement, budget allocation, and bit allocation.

Browsing video scenes is just the process to unfold the story scenarios of a long video archive, which can help users to locate their desired video segments quickly and efficiently. Automatic scene detection of a long video stream file is hence the first and crucial step toward a concise and comprehensive content-based representation for indexing, browsing and retrieval purposes. In this paper, we present a novel scene detection scheme for various video types. We first detect video shot using a coarse-to-fine algorithm. The key frames without useful information are detected and removed using template matching. Spatio-temporal coherent shots are then grouped into the same scene based on the temporal constraint of video content and visual similarity of shot activity. The proposed algorithm has been performed on various types of videos containing movie and TV program. Promising experimental results shows that the proposed method makes sense to efficient retrieval of video contents of interest.

Driven by a high demand for user-centred video interfaces and recent advances in scalable video coding technology, this work introduces a novel framework for video browsing by utilising inherently hierarchical compressed-domain features of scalable video and e cient dynamic video summarisation. This approach enables instant adaptability of generated video summaries to user requirements, available channel bandwidth as well as display size. By utilising compressed domain features an e cient hierarchical analysis of motion activity at di erent layers of complexity is achieved. Exploiting a contour evolution algorithm, a scale space of temporal video descriptors is generated, enabling dynamic video summarisation in real-time. Given the spatial resources of terminal display and generated video summary, the nal browsing layout is generated utilising an unsupervised robust spectral clus- tering technique and a fast discrete optimisation algorithm. Results show excellent scalability of the video browsing interface and good algorithm efficiency.

Image inpainting or image completion is the technique that automatically restores/completes removed areas in an image. When dealing with a similar problem in video, not only should a robust tracking algorithm be used, but the temporal continuity among video frames also needs to be taken into account, especially when the video has camera motions such as zooming and tilting. In this paper, we extend an exemplar-based image inpainting algorithm by incorporating an improved patch matching strategy for video inpainting. In our proposed algorithm, different motion segments with different temporal continuity call for different candidate patches, which are used to inpaint holes after a selected video object is tracked and removed. The proposed new video inpainting algorithm produces very few ldquoghost shadows,rdquo which were produced by most image inpainting algorithms directly applied on video. Our experiments use different types of videos, including cartoon, video from games, and video from digital camera with different camera motions. Our demonstration at http://member.mine.tku.edu.tw/www/T_CSVT/web/shows the promising results.

Frequency domain analysis is a fundamental procedure for understanding the characteristics of visual data. Several studies have been conducted with 2D videos, but analysis of stereoscopic 3D videos is rarely carried out. In this paper, we derive the Fourier transform of a simplified 3D video signal and analyze how a 3D video is influenced by disparity and motion in terms of temporal aliasing. It is already known that object motion affects temporal frequency characteristics of a time-varying image sequence. In our analysis, we show that a 3D video is influenced not only by motion but also by disparity. Based on this conclusion, we present a temporal anti-aliasing filter for a 3D video. Since the human process of depth perception mainly determines the quality of a reproduced 3D image, 2D image processing techniques are not directly applicable to 3D images. The analysis presented in this paper will be useful for reducing undesirable visual artifacts in 3D video as well as for assisting the development of relevant technologies.

Delivering video streaming service over the Internet encounters some challenges. Two of them are heterogeneity of networks capacity and variability of video data rate. The capacity of network segments are constrained. Meanwhile, the rate of video data to be transmitted is highly variable in order to get near-constant images quality. Therefore, to send variable bit rate (VBR) video data over capacity-constrained network, its bit rate should be adjusted. In this paper a system to adjust the transmission bit rate of VBR MPEG video data called Transcoding-after-Smoothing (TaS), which is a combination of bit rate transcoding and bit rate smoothing algorithm, is proposed. The system smoothes out transmission rate of video data while at the same time also performs transcoding on some video frames when necessary in order to keep the transmission bit rate below a certain threshold value. Two kinds of TaS methods are proposed. One method does not have transcoding preference, while the other method uses frame type preference where an intra-coded frame is the last one that will be transcoded. These methods are implemented in our video server where a VBR video data is accessed by a client. Our experiment results show that the first TaS method yields significant reduction in the number of transcoded frames compared with the second TaS method and conventional frame-level transcoding. However, the second method performs better in minimizing the quality distortion.

We present a real-time video-based rendering system using a network camera array. Our system consists of 64 commodity network cameras that are connected to a single PC through a gigabit Ethernet. To render a high-quality novel view, our system estimates a view-dependent per-pixel depth map in real time by using a layered representation. The rendering algorithm is fully implemented on the GPU, which allows our system to efficiently perform capturing and rendering processes as a pipeline by using the CPU and GPU independently. Using QVGA input video resolution, our system renders a free-viewpoint video at up to 30 frames per second, depending on the output video resolution and the number of depth layers. Experimental results show high-quality images synthesized from various scenes.

We propose a method to improve performance of video coding using an adaptive interpolation filter technique. The adaptive interpolation technique was based on clustering using the k-means method and did not need filter selection information. To improve the performance of the clustering, we introduced autocorrelation coefficients of the pixel values and the directions of the motion vectors as the new features and the Mahalanobis distance as the distance scale in the k-means method. We also used vector quantization to reduce the number of interpolation filter coefficients. In the simulation, our proposed method was implemented in the MPEG-2 based video codec. The simulation results show that the proposed method can reduce the bit rate by up to nearly 7 % compared to conventional adaptive interpolation filter methods.

Video retrieval and indexing research aims to efficiently and effectively manage very large video databases, e.g., CCTV records, which is a key component in video-based object and event analysis. In this paper, for the purpose of video retrieval, we propose a novel method to represent video data by developing an optical flow tensor (OFT) and incorporating hidden Markov models (HMMs). As video is content-sensitive and normally carries rich motion information of objects, optical flow field is first employed to estimate such motion. Then, a shot HMMs tree is built to model video clips in different levels in a database. Experimental results demonstrate that the newly developed method inherits advantages of both optical flow and HMMs in video representation. With the newly developed video representation, in video retrieval and indexing tasks, no need to exhaustively compare a query video shot with all video shot records in the database. Moreover, the novel representation method works well when linear discriminant analysis (LDA) is utilized to reduce the feature dimensionality and further speed up the retrieval procedure. © 2008 Elsevier B.V. All rights reserved.

Free-viewpoint video allows interactive control of the camera viewpoint in video playback using a three-dimensional (3D) scene representation. This paper describes the challenges and the state-of-the-art techniques adopted in developing a free-viewpoint renderer for 3D video production of people. The system requires video streams from a set of fixed or dynamic real-world video cameras and synthesizes arbitrary viewpoints on the GPU using a dynamic 3D scene model. We have released the renderer as an open source application for the research community (see the web information at the end of this paper). The renderer provides (i) the facility to simulate camera viewpoints from publicly available multiple-view video datasets and (ii) a baseline technique for free-viewpoint video synthesis in the development of interactive 3D video and 3DTV applications.

This paper presents a Reduced-reference based video-quality estimation method suitable for individual end-user quality monitoring of IPTV services. With the proposed method, the activity values for individual given-size pixel blocks of an original video are transmitted to end-user terminals. At the end-user terminals, the video quality of a received video is estimated on the basis of the activity-difference between the original video and the received video. Psychovisual weightings and video-quality score adjustments for fatal degradations are applied to improve estimation accuracy. In addition, low-bit-rate transmission is achieved by using temporal sub-sampling and by transmitting only the lower six bits of each activity value. The proposed method achieves accurate video quality estimation using only low-bit-rate original video information (15kbps for SDTV). The correlation coefficient between actual subjective video quality and estimated quality is 0.901 with 15kbps side information. The proposed method does not need computationally demanding spatial and gain-and-offset registrations. Therefore, it is suitable for real-time video-quality monitoring in IPTV services.

Video playback on a mobile device is a resource-intensive task. Since the battery life of a mobile device decreases with time, it is desirable to have a video representation which adapts dynamically to the available battery life during the playback process. A novel Hybrid Layered Video (HLV) encoding scheme is proposed, which comprises of content-aware, multi-layer encoding of texture and a generative sketch-based representation of the object outlines. Different combinations of the texture- and sketch-based representations are shown to result in distinct video states, each with a characteristic power consumption profile. Further, a smart content-aware caching scheme is proposed which is suitable for low-latency dissemination of HLV over the Internet. The proposed HLV representation, combined with the proposed caching scheme, is shown to be effective for video playback and dissemination on power-constrained mobile devices.

To optimize scarce network resources and present the highest quality video, streaming video systems need adapt to the video content as well as the network conditions. This paper presents ARMOR, a video streaming system that dynamically adjusts repair and media scaling to meet current video and network conditions. In order to adapt effectively, ARMOR, and any dynamic video adaptation system, needs to predict the video quality as perceived by end users over the range of scaling and repair choices. Thus, this paper first proposes a novel video quality metric called distorted playable frame rate that provides estimation of user perceptual quality considering temporal and quality degradations. Comprehensive user studies show distorted playable frame rate is more accurate than other video quality metrics. Analytic experiments with distorted playable frame rate and the ARMOR optimization algorithm illustrate the predictive power of the metric in a dynamic, streaming video system. Lastly, implementation and experiments of a complete, fully-functioning ARMOR system show the effective practicality of the proposed approach.

In this article we present three key ideas which together form a flexible framework for maximizing user-perceived quality under given resources with modern video codecs (H.264). First, we present a method to predict resource usage for video decoding online. For this, we develop and discuss a video decoder model using key metadata from the video stream. Second, we explain a light-weight method for providing replacement content for a given region of a frame. We use this method for online adaptation. Third, we select a metric modeled after human image perception which we extend to quantify the consequences of available online adaptation decisions. Together, these three parts allow us, to the best of our knowledge for the first time, to maximize user-perceived quality in video playback under given resource constraints.

The mushroom growth of video information, consequently, necessitates the progress of content-based video analysis techniques. Video summarization, aiming to provide a short video summary of the original video document, has drawn much attention these years. In this paper, we propose an algorithm for video summarization with a two-level redundancy detection procedure. By video segmentation and cast indexing, the algorithm first constructs story boards to let users know main scenes and cast (when this is a video with cast) in the video. Then it removes redundant video content using hierarchical agglomerative clustering in the key frame level. The impact factors of scenes and key frames are defined, and parts of key frames are selected to generate the initial video summary. Finally, a repetitive frame segment detection procedure is designed to remove redundant information in the initial video summary. Results of experimental applications on TV series, movies and cartoons are given to illustrate the proposed algorithm.

A novel method is given for content-aware video resizing, i.e. targeting video to a new resolution (which may involve aspect ratio change) from the original. We precompute a per-pixel cumulative shrinkability map which takes into account both the importance of each pixel and the need for continuity in the resized result. (If both x and y resizing are required, two separate shrinkability maps are used, otherwise one suffices). A random walk model is used for efficient offline computation of the shrinkability maps. The latter are stored with the video to create a multi-sized video, which permits arbitrary-sized new versions of the video to be later very efficiently created in real-time, e.g. by a video-on-demand server supplying video streams to multiple devices with different resolutions. These shrinkability maps are highly compressible, so the resulting multi-sized videos are typically less than three times the size of the original compressed video. A scaling function operates on the multi-sized video, to give the new pixel locations in the result, giving a high-quality content-aware resized video. Despite the great efficiency and low storage requirements for our method, we produce results of comparable quality to state-of-the-art methods for content-aware image and video resizing.

Video abstraction is defined as creating a video abstract which includes only important information in the original video streams. There are two general types of video abstracts, namely the dynamic and static ones. The dynamic video abstract is a 3-dimensional representation created by temporally arranging important scenes while the static video abstract is a 2-dimensional representation created by spatially arranging only keyframes of important scenes. In this paper, we propose a unified method of automatically creating these two types of video abstracts considering the semantic content targeting especially on broadcasted sports videos. For both types of video abstracts, the proposed method firstly determines the significance of scenes. A play scene, which corresponds to a play, is considered as a scene unit of sports videos, and the significance of every play scene is determined based on the play ranks, the time the play occurred, and the number of replays. This information is extracted from the metadata, which describes the semantic content of videos and enables us to consider not only the types of plays but also their influence on the game. In addition, user's preferences are considered to personalize the video abstracts. For dynamic video abstracts, we propose three approaches for selecting the play scenes of the highest significance: the basic criterion, the greedy criterion, and the play-cut criterion. For static video abstracts, we also propose an effective display style where a user can easily access target scenes from a list of keyframes by tracing the tree structures of sports games. We experimentally verified the effectiveness of our method by comparing our results with man-made video abstracts as well as by conducting questionnaires.

The explosive growth of video data demands the video presentation technique which supports fast browsing of video content. In this paper, we present an automatic procedure for constructing a compact synthesized collage from a video sequence. The synthesized image, called “Video Collage”, is a kind of static video summary—to select the most representative images from video, to extract salient regions of interest (ROIs) from these images, and to seamlessly arrange ROIs on a given canvas with the temporal structure of video content preserved. We formulate the generation of Video Collage as a unified energy minimization problem in which each of above desirability is represented by an energy term. We start from the basic setting of Video Collage in which both the shape of ROIs and collage are fixed as rectangular, and then show how it can support arbitrary shapes of ROIs, as well as a variety of collage templates and region of interest (ROI) arrangement layouts (i.e., book, diagonal, and spiral). The experiments show its effectiveness to present a video in a very compact and visually appealing form while preserving the necessary information to understand the video.

One of the most common tasks in image and video editing is the local adjustment of various properties (e.g., saturation or brightness) of regions within an image or video. Edge-aware interpolation of user-drawn scribbles offers a less effort-intensive approach to this problem than traditional region selection and matting. However, the technique suffers a number of limitations, such as reduced performance in the presence of texture contrast, and the inability to handle fragmented appearances. We significantly improve the performance of edge-aware interpolation for this problem by adding a boosting-based classification step that learns to discriminate between the appearance of scribbled pixels. We show that this novel data term in combination with an existing edge-aware optimization technique achieves substantially better results for the local image and video adjustment problem than edge-aware interpolation techniques without classification, or related methods such as matting techniques or graph cut segmentation.

The efficiency of H.264 video compression together with the wide use of MPEG-2 today necessitates transcoding between these two technologies. The combination of bandwidth efficient technology such as H.264 and the widely deployed MP EG-2 allows for innovative video delivery services that can exploit both these technologies. Using H.264 for video delivery and MPEG-2 for playback on the end user equipment fully leverages the benefits of these technologies. This architecture requires transcoding H.264 video delivered over networks to MPEG-2 format for playback on the end- user device. We present algorithms for transcoding H.264 to MPEG-2 and evaluate the performance of these algorithms. The performance is compared with a baseline transcoder with a full decoding stage and a full encoding stage. The proposed algorithms reduce the complexity by reducing the search space for motion estimation. The results show that the proposed algorithms reduce the motion estimation complexity by over 80% with a negligible loss in PSNR.

Storage of video information is experiencing rapid growth as digital broadcasting services have developed. This has rapidly increased video data storage requirements in DVD and hard disc recorders in recent years. We design a video summarization tool for personal video recorders. This tool provides a video summarization function to grasp the original long video content quickly. This tool is with a two- level redundancy detection procedure. After the video parsing step, it removes redundant video content with the hierarchical agglomerative cluster method in the shot level. Impact factors of scenes and key frames are defined, and parts of shots are selected to generate the initial video summary. Finally a repetitive frame segment detection step is used to remove redundant information. We have developed a prototype model and tested this tool on TV series, movies and cartoons data. The experiment results are given to illustrate the proposed tool.

The characterization of a video segment by a digital signature is a fundamental task in video processing. It is necessary for video indexing and retrieval, copyright protection, and other tasks. Semantic video signatures are those that are based on high-level content information rather than on low-level features of the video stream. The major advantage of such signatures is that they are highly invariant to nearly all types of distortion. A major semantic feature of a video is the appearance of specific persons in specific video frames. Because of the great amount of research that has been performed on the subject of face detection and recognition, the extraction of such information is generally tractable, or will be in the near future. We have developed a method that uses the pre-extracted output of face detection and recognition to perform fast semantic query-by-example retrieval of video segments. We also give the results of the experimental evaluation of our method on a database of real video. One advantage of our approach is that the evaluation of similarity is convolution-based, and is thus resistant to perturbations in the signature and independent of the exact boundaries of the query segment.

The new H.264 video coding standard achieves significantly higher compression performance than MPEG-2. As the MPEG-2 is popular in digital TV, DVD, etc., bandwidth or memory space can be saved by transcoding those streams into H.264 in these applications. Unfortunately, the huge complexity keeps transcoding from being widely used in practical applications. This paper proposes an efficient transcoding architecture with a smart downscaling decoder and a fast mode decision algorithm. Using the proposed architecture, huge buffering memory space is saved and the transcoding complexity is reduced. Performance of the proposed fast mode decision algorithm is validated by experiments.

Currently 3D animation rendering and video compression are completely independent processes even if rendered frames are streamed on-the-fly within a client-server platform. In such scenario, which may involve time-varying transmission bandwidths and different display characteristics at the client side, dynamic adjustment of the rendering quality to such requirements can lead to a better use of server resources. In this work, we present a framework where the renderer and MPEG codec are coupled through a straightforward interface that provides precise motion vectors from the rendering side to the codec and perceptual error thresholds for each pixel in the opposite direction. The perceptual error thresholds take into account bandwidth-dependent quantization errors resulting from the lossy com-pression as well as image content-dependent luminance and spatial contrast masking. The availability of the discrete cosine transform (DCT) coefficients at the codec side enables to use advanced models of the human visual system (HVS) in the perceptual error threshold derivation without incurring any significant cost. Those error thresholds are then used to control the rendering quality and make it well aligned with the compressed stream quality. In our prototype system we use the lightcuts technique developed by Walter et al., which we enhance to handle dynamic image sequences, and an MPEG-2 implementation. Our results clearly demonstrate many advantages of coupling the rendering with video compression in terms of faster rendering. Furthermore, temporally coherent rendering leads to a reduction of temporal artifacts.

Ray-space based arbitrary viewpoint rendering without complex object segmentation or model construction is the main technology to realize Free Viewpoint Video (FVV) system for complex scenes. Ray-space interpolation and compression are two key techniques for the solution. In this paper, correlation among multiple epipolar lines in ray-space data is analyzed, and a new method of ray-space interpolation with multi-epipolar lines matching is proposed. Comparing with the pixel-based matching interpolation method and the block-based matching interpolation method, the proposed method can achieve higher Peak Signal to Noise Ratio (PSNR) in interpolating ray-space data and rendering arbitrary viewpoint images.

Recent developments have given birth to H.264/AVC: a video coding standard offering better bandwidth to video quality ratios than previous standards (such as H.263, MPEG-2, MPEG-4, etc.), due to its improved inter- and intraprediction modes at the expense of higher computation complexity. It is expected that H.264/AVC will take over the digital video market, replacing the use of previous standards in most digital video applications. This creates an important need for heterogeneous video transcoding technologies from older standards to H.264. In this paper, we focus our attention on the interframe prediction, the most computationally intensive task involved in the heterogeneous video transcoding process. This paper presents a novel macroblock (MB) mode decision algorithm for interframe prediction based on data mining techniques to be used as part of a very low complexity heterogeneous video transcoder. The proposed approach is based on the hypothesis that MB coding mode decisions in H.264 video have a correlation with the distribution of the motion compensated residual in the decoded video. We use data mining tools to exploit the correlation and derive decision trees to classify the incoming decoded MBs into one of the several coding modes in H.264. The proposed approach reduces the H.264 MB mode computation process into a decision tree lookup with very low complexity. For general validation purposes, we apply our algorithm to two of the most important heterogeneous video transcoders: MPEG-2 to H.264 and H.263 to H.264. Our results show that the our data-mining based transcoding algorithm is able to maintain a good video quality while considerably reducing the computational complexity by 72% on average when applied in MPEG-2 to H.264 transcoders, and by 62% on average when applied in H.263 to H.264 transcoders. Finally, we conduct a comparative study with some of the most prominent fast interprediction methods for H.264 presented in the literature. Our results show that the proposed data mining-based approach achieves the best results for video transcoding applications.

Successful proliferation of multimedia-enabled devices and advances in very large-scale integration (VLSI) technology has spawned new research efforts in migrating video processing applications onto ever smaller and more inexpensive devices. This article focuses on the technical challenges associated with that migration. Due to limitations in size, battery lifetime, and, ultimately, cost, mapping complex video applications onto resource-constrained systems is a very challenging proposition. To this end, we first consider a technique, region-of-interest (ROI) processing, of defining a window within a video frame and only operating on the data inside that window, ignoring the rest of the frame. By using this lossy technique, the processing requirements can be reduced by roughly 80% while the error introduced in the quality of the results is roughly 10%. The other technique is adaptive data partitioning (ADP) combined with a content-based power management algorithm. By distributing video processing among multiple processors and shutting them down when they are not needed, the energy consumed per processor can be reduced by 60% without sacrificing the performance of the underlying video-based application. Taken together, these novel techniques enable ambient multimedia systems and maintain the needed overall efficiency in video processing.

Video structure analysis is a basic requirement for most content-based video editing and processing systems. This paper presents a fast video structure analysis method based on image segmentation in each frame, with region matching between frames. The structure analysis decomposes the video into several moving objects, including information about their colors, positions, shapes, movements, and lifetimes. The method also supports user interactions to improve the results. The result shows that this method is fast and stable and can complete video analyzing interactively.