Search

We formulate a just noticeable rendering distortion (JNRD) model.Spatial and temporal correlation of depth video is enhanced based on JNRD model.Bit rate of the depth video is greatly saved while quality of virtual view is maintained. Spatial and temporal inconsistency of depth video deteriorates encoding efficiency in three dimensional video systems. A depth video processing algorithm based on human perception is presented. Firstly, a just noticeable rendering distortion (JNRD) model is formulated by combining the analyses of the influence of depth distortion on virtual view rendering with human visual perception characteristics. Then, depth video is processed based on the JNRD model from two aspects, spatial and temporal correlation enhancement. During the process of spatial correlation enhancement, depth video is segmented into edge, foreground, and background regions, and smoothened by Gaussian and mean filters. The operations of the temporal correlation enhancement include temporal-spatial transpose (TST), temporal smoothing filter and inverse TST. Finally, encoding and virtual view rendering experiments are conducted to evaluate the proposed algorithm. Experimental results show that the proposed algorithm can greatly reduce the bit rate while it maintains the quality of virtual view.

We propose a playback length changeable 3D video data chunk segmentation algorithm.A novel hybrid-priority based 3D video P2P data scheduling algorithm is presented.Bandwidth-adaptive 3D video P2P streaming for heterogeneous networks is implemented.The proposed method obtains superior error-resilience and network utilization performance. 3D video distribution over P2P networks has been thought as a promising way for 3D video entering home. The convergence of scalable 3D video coding and P2P streaming can provide diverse 3D experiences for heterogeneous clients with high distribution efficiencies. However, the conventional chunk segmentation and scheduling algorithms originally aiming at the non-scalable 2D video streaming are not very efficient for scalable 3D video streaming over P2P networks due to the particular data characteristics of scalable 3D video. Based on this motivation, this paper first presents a playback length changeable 3D video chunk segmentation (PLC3DCS) algorithm to provide different error resilience strengths to video and depth as well as layers with different importance levels in the 3D video transmission. Then, a hybrid-priority based chunk scheduling (HPS) algorithm is proposed to be tied in with the proposed chunk segmentation algorithm to further promote the overall 3D video P2P streaming performance. The simulation results show that the proposed PLC3DCS algorithm with the corresponding HPS can increase the success delivery rates of chunks with more important levels, and further improve the user's quality of 3D experience.

For a variety of applications such as video surveillance and event annotation, the spatial–temporal boundaries between video objects are required for annotating visual content with high-level semantics. In this paper, we define spatial–temporal sampling as a unified process of extracting video objects and computing their spatial–temporal boundaries using a learnt video object model. We first provide a computational approach for learning an optimal key-object codebook sequence from a set of training video clips to characterize the semantics of the detected video objects. Then, dynamic programming with the learnt codebook sequence is used to locate the video objects with spatial–temporal boundaries in a test video clip. To verify the performance of the proposed method, a human action detection and recognition system is constructed. Experimental results show that the proposed method gives good performance on several publicly available datasets in terms of detection accuracy and recognition rate.

3D video services are emerging in various application domains including cinema, TV broadcasting, Blu-ray discs, streaming and smartphones. A majority of the 3D video content in market is still based on stereo video, which is typically coded with the multiview video coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard or as frame-compatible stereoscopic video. However, the 3D video technologies face challenges as well as opportunities to support more demanding application scenarios, such as immersive 3D telepresence with numerous views and 3D perception adaptation for heterogeneous 3D devices and/or user preferences. The Multiview Video plus Depth (MVD) format enables depth-image-based rendering (DIBR) of additional viewpoints in the decoding side and hence helps in such advanced application scenarios. This paper reviews the MVC+D standard, which specifies an MVC-compatible MVD coding format.

In this paper, we study the impact of quantization, frame dropping and spatial down-sampling on the perceived quality of compressed video streams. Based on the analysis of quality ratings obtained from extensive subjective tests, we propose a no-reference metric (named MDVQM) for video quality estimation in the presence of both spatial and temporal quality impairments. The proposed metric is based on the per-pixel bitrate of the encoded stream and selected spatial and temporal activity measures extracted from the video content. All the values required to compute the proposed video quality metric can be obtained without using the original reference video which makes the metric for instance useful for making transcoding decisions in a wireless video transmission scenario. Different from comparable metrics in the literature, we have also considered the case when both frame rate and frame size are changed simultaneously. The validation results show that the proposed metric provides more accurate estimation of the video quality than the state of the art metrics.

Video summarization is a method to reduce redundancy and generate succinct representation of the video data. One of the mechanisms to generate video summaries is to extract key frames which represent the most important content of the video. In this paper, a new technique for key frame extraction is presented. The scheme uses an aggregation mechanism to combine the visual features extracted from the correlation of RGB color channels, color histogram, and moments of inertia to extract key frames from the video. An adaptive formula is then used to combine the results of the current iteration with those from the previous. The use of the adaptive formula generates a smooth output function and also reduces redundancy. The results are compared to some of the other techniques based on objective criteria. The experimental results show that the proposed technique generates summaries that are closer to the summaries created by humans.

We propose no-reference analysis and processing of DCT (Discrete Cosine Transform) coded images based on estimation of selected MPEG parameters from the decoded video. The goal is to assess MPEG video quality and perform post-processing without access to neither the original stream nor the code stream. Solutions are presented for MPEG-2 video. A method to estimate the quantization parameters of DCT coded images and MPEG I-frames at the macro-block level is presented. The results of this analysis is used for deblocking and deringing artifact reduction and no-reference PSNR estimation without code stream access. An adaptive deringing method using texture classification is presented. On the test set, the quantization parameters in MPEG-2 I-frames are estimated with an overall accuracy of 99.9% and the PSNR is estimated with an overall average error of 0.3dB. The deringing and deblocking algorithms yield improvements of 0.3dB on the MPEG-2 decoded test sequences.

This paper proposes a new method for presenting sports videos. Tennis videos are used as an example for the implementation of a viewing program called as Tennis Video 2.0. For the methods in video analysis, background generation by considering the pixels in temporal and spatial distribution is proposed; foreground segmentation combining automatic trimap generation and matting model is proposed. To provide more functions in watching videos, the rendering flow of video contents and the semantic Scalability are proposed. With the new analysis and rendering tools, the presentation of sports videos has three properties-Structure, Interactivity, and Scalability. The experiments show that several broadcasting game videos are employed to evaluate the robustness and performance of the proposed system. For user study, 20 evaluators highly identify that Tennis Video 2.0 is a new presentation of sports videos and give people better viewing experience.

Fast-forward playback enables viewers to scan through the video scene of interest efficiently. One approach to realize fast-forward playback is to employ a frame-skipping transcoder which transcodes only the frames required for playback at the desired fast speed. Various motion vector composition algorithms are used to compose the new motion vectors with reduced complexity. These algorithms do not work well for dropping a large number of frames, which is very common in fast-forward playback. In this paper, a new multiple-candidate vector selection algorithm (MCVS) is proposed to select a composed motion vector from a set of candidate motion vectors, which utilizes relevant areas in the target macroblock to ensure a reliable tracking process for motion vector composition. Experimental results show that the proposed MCVS can provide fast-forward playback through video transcoding with significant gain, in terms of rate-distortion performance, especially when a large speed-up factor is required.

In 3D TV research, one approach is to employ multiple cameras for creating a 3D multi-view signal with the aim to make interactive free-viewpoint selection possible in 3D TV media. This paper explores a new rendering algorithm that enables to compute a free-viewpoint between two reference views from existing cameras. A unique property is that we perform forward warping for both texture and depth simultaneously. Advantages of our rendering are manyfold. First, resampling artifacts are filled in by inverse warping. Second, disocclusions are processed while omitting warping of edges at high discontinuities. Third, our disocclusion inpainting approach explicitly uses depth information. We obtain an average PSNR gain of 3 dB and 4.5 dB for the ‘Breakdancers’ and ‘Ballet’ sequences, respectively, compared recently published results. Moreover, experiments are performed using compressed video from surrounding cameras. The overall system quality is dominated by rendering quality and not by coding.

Video playback on a mobile device is a resource-intensive task. Since the battery life of a mobile device decreases with time, it is desirable to have a video representation which adapts dynamically to the available battery life during the playback process. A novel Hybrid Layered Video (HLV) encoding scheme is proposed, which comprises of content-aware, multi-layer encoding of texture and a generative sketch-based representation of the object outlines. Different combinations of the texture- and sketch-based representations are shown to result in distinct video states, each with a characteristic power consumption profile. Further, a smart content-aware caching scheme is proposed which is suitable for low-latency dissemination of HLV over the Internet. The proposed HLV representation, combined with the proposed caching scheme, is shown to be effective for video playback and dissemination on power-constrained mobile devices.

To optimize scarce network resources and present the highest quality video, streaming video systems need adapt to the video content as well as the network conditions. This paper presents ARMOR, a video streaming system that dynamically adjusts repair and media scaling to meet current video and network conditions. In order to adapt effectively, ARMOR, and any dynamic video adaptation system, needs to predict the video quality as perceived by end users over the range of scaling and repair choices. Thus, this paper first proposes a novel video quality metric called distorted playable frame rate that provides estimation of user perceptual quality considering temporal and quality degradations. Comprehensive user studies show distorted playable frame rate is more accurate than other video quality metrics. Analytic experiments with distorted playable frame rate and the ARMOR optimization algorithm illustrate the predictive power of the metric in a dynamic, streaming video system. Lastly, implementation and experiments of a complete, fully-functioning ARMOR system show the effective practicality of the proposed approach.

In this article we present three key ideas which together form a flexible framework for maximizing user-perceived quality under given resources with modern video codecs (H.264). First, we present a method to predict resource usage for video decoding online. For this, we develop and discuss a video decoder model using key metadata from the video stream. Second, we explain a light-weight method for providing replacement content for a given region of a frame. We use this method for online adaptation. Third, we select a metric modeled after human image perception which we extend to quantify the consequences of available online adaptation decisions. Together, these three parts allow us, to the best of our knowledge for the first time, to maximize user-perceived quality in video playback under given resource constraints.

Video transcoding is a key technology to support video communications over heterogeneous networks. Although quite a bit of research effort has been made in video transcoding due to its wide applications, most video transcoding techniques proposed in the literature are optimized based on the simple mean squared error (MSE) metric which does not correlate well with the human visual perception. In this paper, foveation, a property of the HVS, is exploited in video transcoding. The proposed foveation embedded DCT domain video transcoding can reduce the bit rate without compromising visual quality or achieve better subjective quality for a given bit rate by shaping the compression distortion according to the foveated contrast sensitivity function of the HVS. In addition, fast algorithms for video foveation filtering and DCT domain inverse motion compensation are developed, which significantly improve the efficiency of video transcoding.

Network-based computing is becoming an increasingly important area of research, whereby computational elements within a distributed infrastructure process/enhance the data that traverses through its path. We refer to these computations as online processing and this paper investigates scene change detection in the context of MBone-based proxies in the network. On-line processing varies from traditional offline processing schemes, where for example, the whole video scope is known as a priori, which allows multiple scans of the stored video files during video processing. The on-line processing system is designed to meet the requirements of real-time video multicasting over the Internet and to utilize the successful video parsing techniques available today. The proposed algorithms do scene change detection and extract key frames from video bitstreams sent through the MBone network. We study several algorithms based on histogram differences and evaluate them with respect to precision, recall, and processing latency.

Changing between frames is one of the most obvious information in video data. This frame-by-frame time series data is essential to many application areas. However, how to extract and compare video contents in a video information system is still an important problem to be solved. In this paper, we focus on the problem of design a fast searching method in a video information system to locate video segments that match a content-based query, approximately by time series feature values. The idea is to extract video contents via low-level feature extraction and/or high level semantic retrieval mechanisms according to a specific point of view, then segment video contents into bounding boxes via a box segmentation mechanism by their time series feature values. Video content indexing is constructed by the characteristics of prominent points that accompany bounding boxes. We also propose an efficient and effective video content matching algorithm to find similar sequences. With the help of the video indexing and matching mechanisms, several high level box-to-box and low level point-to-point query types can be requested. The implementation and performance evaluation of our video information prototype system is described.

Indexing is an important aspect of video database management. Video indexing involves the analysis of video sequences, which is a computationally intensive process. However, effective management of digital video requires highly efficient, yet robust indexing techniques. Based on the properties of the generalized color ratio model, the paper presents techniques for robust and efficient video indexing, by analyzing the video sequence directly in the compressed domain. Illumination and motion invariant color ratio features, appropriate for block-based transform domain video indexing are presented. Robustness is achieved by use of (i) illumination and motion invariant color ratios; (ii) formally derived thresholds with predictable performance; and (iii) both DC and AC coefficients. Gains in efficiency accrue from: (i) operating only on a formally selected minimal set of blocks; (ii) performing all operations directly in the compressed domain; and (iii) avoiding computationally expensive motion compensation procedures. Some experimental results are presented which show the superiority of the proposed methods over traditional approaches.