Search

Motion compensation with multiple reference pictures has been widely used during the development of the emerging High Efficiency Video Coding (HEVC) standard, which greatly helps to improve the coding efficiency. Usually, a heuristic strategy is exploited to use the nearest reconstructed pictures as references. However, such a strategy may not be efficient on all occasions, especially when different content characteristics and coding settings are considered. In this paper, we investigate how to manage reference pictures so as to achieve better rate-distortion performance under the memory constraint of the decoded picture buffer at the decoder. We formulate the reference picture management as an optimization problem and approximate its optimal solution. Moreover, we explore how to adjust quality for each picture according to the reference structure to further improve coding efficiency. For some coding cases, where a complicated encoder optimization is unaffordable, we also develop fast algorithms to get the most benefit from reference picture selection. Among them, one strategy has been adopted by the HEVC software and common test conditions to generate the anchor. Experimental results show that the proposed full search algorithm and fast search algorithms achieve significant bitrate reduction.

Visually summarizing web pages is an attractive approach that provides users an effective and friendly interface to identify desired contents at a first glance for search and re-finding tasks. Using dominant images in web pages is generally reliable for this purpose. However, dominant images are often unavailable in many web pages. To solve this problem, we first propose a new approach to summarize those web pages without any dominant images by retrieving relevant external images from the Internet. However, relevant external images are sometimes unreliable. To take the advantages of these two kinds of images, we further propose a clustering based algorithm to select the best summarization among all of internal and external images. This algorithm leverages relevance and dominance of images as the prior information. Experimental results show that our approach achieves 0.098 and 0.082 NDCG1 gain on a human labeled data set, compared with relevant external image and dominant image, respectively. Our user study also indicates that the images selected by our algorithm are useful as the summarization of web pages.

This paper presents diagonal motion partitions (DMP) for inter prediction in HEVC. In addition to the square and rectangular partitions, we propose to add diagonal shaped partitions to match different motion parts with oblique boundaries. Considering the overlap of pixels along the partition boundaries, the calculation of sum of absolute differences (SAD) and the motion compensation for the pixels on the boundaries are weighted. Besides, the residues of a diagonal prediction unit (PU) are augmented to form a rectangular one, so as to perform Hadamard transform on the residues. We also revise the advanced motion vector prediction (AMVP) and merge candidates based on the diagonal motion partitions. Experimental results show that on average 0.8%-1.0% BD-rate reduction can be achieved by DMP.

Perspective motion is commonly represented in video content that is captured and compressed for various applications including cloud gaming, vehicle and aerial monitoring, etc. Existing approaches based on an eight-parameter homography motion model cannot deal with this efficiently, either due to low prediction accuracy or excessive bit rate overhead. In this paper, we consider the camera motion model and scene structure in such video content and propose a joint global and local homography motion coding approach for video with perspective motion. The camera motion is estimated by a computer vision approach, and camera intrinsic and extrinsic parameters are globally coded at the frame level. The scene is modeled as piece-wise planes, and three plane parameters are coded at the block level. Fast gradient-based approaches are employed to search for the plane parameters for each block region. In this way, improved prediction accuracy and low bit costs are achieved. Experimental results based on the HEVC test model show that up to 9.1% bit rate savings can be achieved (with equal PSNR quality) on test video content with perspective motion. Test sequences for the example applications showed a bit rate savings ranging from 3.7 to 9.1%.

Traditional intra prediction usually utilizes the nearest reference line to generate the predicted block when considering strong spatial correlation. However, this kind of single line-based method does not always work well due to at least two issues. One is the incoherence caused by the signal noise or the texture of other object, where this texture deviates from the inherent texture of the current block. The other reason is that the nearest reference line usually has worse reconstruction quality in block-based video coding. Due to these two issues, this paper proposes an efficient multiple line-based intra prediction scheme to improve coding efficiency. Besides the nearest reference line, further reference lines are also utilized. The further reference lines with relatively higher quality can provide potential better prediction. At the same time, the residue compensation is introduced to calibrate the prediction of boundary regions in a block when we utilize further reference lines. To speed up the encoding process, this paper designs several fast algorithms. Experimental results show that, compared with HM-16.9, the proposed fast search method achieves 2.0% bit saving on average and up to 3.7%, with increasing the encoding time by 112%., Accepted for publication in IEEE Transactions on Circuits and Systems for Video Technology

Discrete Cosine Transform (DCT) has been the commonly used transform for a few decades in image/video coding. However, DCT does not work well on the blocks having anisotropic correlations. In this paper, based on the adaptive dictionary, we propose a new online transform scheme using Orthogonal Matching Pursuit (OMP) for High Efficiency Video Coding (HEVC). For a coding block, we construct its dictionary by exploiting non-local correlations from the reconstructed regions. The OMP algorithm is implemented to obtain the sparse transform coefficients. Experimental results show that the BD-rate savings of the proposed scheme for the sequences with strong edges can be up to 19.9%.

This paper proposes a distributed compressive sensing (CS) scheme for robust image transmission over unknown or time-varying channels with highly correlated images at the decoder. A compressed thumbnail is first transmitted after digital forward error correction (FEC) and modulation to retrieve highly correlated images and generate a side information (SI) at the decoder. The current residual image after subtracting the decompressed thumbnail is then coded and transmitted by CS through a very dense constellation without FEC. The linear representation of the residual signal by CS measurements and rateless sampling makes it able to achieve graceful degradation and bandwidth scalability without channel feedback. Moreover, a transform-domain power allocation is employed before random sampling to protect against channel errors. At the decoder, both the nonlocal correlations within the original image and the correlation with the SI are exploited in CS decoding via a low-rank regulation on similar patches. After CS decoding, a block-wise minimum-mean-square-error (MMSE) reconstruction using the SI is further performed in the spatial domain to enhance the reconstruction quality. Simulations on landmark images and an unknown Gaussian channel show that an up to 10 dB gain is achieved at low channel SNRs compared with the state-of-the-art uncoded image transmission scheme, i.e. SoftCast, when highly correlated images are available at the decoder.

Screen content refers to computer generated content like text, graphics, and animations. In such video, many regions may remain static for a long period after a sudden change. Traditional hierarchical Quantization Parameter (QP) setting may not be able to handle these regions efficiently because the encoder probably needs to refine the quality of these static regions multiple times. It will cost more bits while the quality of the static regions may reach the expected degree which the flat QP setting is able to achieve. This paper proposes using different QP settings for different regions in a picture. Region classification algorithms are developed to determine whether a flat or hierarchical QP setting is used. Experimental results demonstrate that the proposed scheme can achieve an average bitrate reduction of 3.1%, and up to 8.1% bitrate reduction for IBBB coding. The proposed method improves coding efficiency without increasing encoding complexity.

Emerging screen content coding brings great challenges to rate control due to significantly different characteristics of screen content from conventional video, e.g. large motion, frequent scene changes. This paper proposes an efficient rate control scheme for screen content coding by considering the characteristics of screen content. We first classify pictures of a video sequence into different groups by comparing the current picture with its neighbours. Then for each group, we apply different strategies to bit allocation and parameter updating process based on the characteristics of each group. The proposed algorithm can control the bitrate accurately without introducing any additional encoding delay. Experiment results show that the proposed scheme can significantly improve PSNR with a more accurate bitrate compared with the existing rate control scheme.

Over the past few decades, many studies on image and video compression have found various approaches to the exploitation of spatial and temporal local correlations. However, we believe it is imperative to find more efficient methods to progress the development of image and video compression. In this paper, we first study spatial non-local correlation, deducing that there exist strong correlations in non-local regions. However, it is rather difficult to make use of these non-local correlations while simultaneously minimizing overhead. To solve this problem, we propose the signal-dependent transform (SDT), which is derived from decoded non-local blocks that are selected by matching neighboring pixels. Since the encoder and decoder can use the same methods to derive the proposed transform, we can successfully eliminate overhead. Finally, we have implemented the proposed transform into the Key Technology Area (KTA) software to exploit both spatial and temporal non-local correlations. The experimental results show that the coding gain over KTA can be as high as 1.4 dB in intra-frame coding, and up to 1.0 dB in inter-frame coding. We believe we have effectively created an alternate method to improve image and video compression.

While directional adaption is introduced into traditional transforms, different orders of two 1-D transforms will result in different results of one 2-D transform. Based upon an anisotropic image model, this paper analyzes the effect of transform orders in terms of theoretical coding gain. Our results reveal that the transform orders have little effect on the coding gain with full decomposition, good directional modes and good interpolation. However, in practical compression schemes, since high-pass bands are not decomposed fully because of the consideration on complexity, different transform orders have different coding performances, which can be solved by an adaptive transform order. Motivated by our analyzed results, a directional filtering transform (dFT, in order to distinguish from the common usage on DFT) is proposed in this paper to better exploit correlations among samples in H.264 intraframe coding. It provides an evenly distributed set of prediction modes with an adaptive transform order. Both interblock and intrablock correlations are exploited in this scheme. Experimental results in H.264 intraframe coding demonstrate its superiority both objectively and subjectively.

In most existing in-band wavelet video coding schemes, over-complete wavelet transform is used for the motion-compensated temporal filtering (MCTF) of each spatial subband. It can overcome the shift-variance of critical sampling wavelet transform and improve the coding efficiency of the in-band scheme. However, a dilemma exists in the current implementations of in-band MCTF (IBMCTF), which is whether or not to exploit the spatial highpass subbands in motion compensation of the spatial lowpass subband. The absence of the spatial highpass subbands will result in significant quality loss in the reconstructed full-resolution video, whereas the presence of the spatial highpass subbands may bring serious mismatch error in the decoded low-resolution video since the corresponding highpass subbands may be unavailable at the decoder. In this paper, we first analyze the mismatch error propagation in decoding the low-resolution video. Based on our analysis, we then propose a frame-based cross-resolution leaky prediction scheme for IBMCTF. It can make a good tradeoff between alleviating the low-resolution mismatch and improving the full-resolution coding efficiency. Experimental results show that the proposed scheme can dramatically reduce the mismatch error by 0.3-2.5 dB for low resolution, while the performance loss is marginal for high resolution.

In existing pyramid-based spatially scalable coding schemes, such as H.264/MPEG-4 SVC (scalable video coding), video frame at a certain high-resolution layer is mainly predicted either from the same frame at the next lower resolution layer, or from the temporal neighboring frames within the same resolution layer. But these schemes fail to exploit both kinds of correlation simultaneously and therefore cannot remove the redundancies among resolution layers efficiently. This paper extends the idea of spatiotemporal subband transform and proposes a general in-scale motion compensation technique for pyramid-based spatially scalable video coding. Video frame at each high-resolution layer is partitioned into two parts in frequency. Prediction for the lowpass part is derived from the next lower resolution layer, whereas prediction for the highpass part is obtained from neighboring frames within the same resolution layer, to further utilize temporal correlation. In this way, both kinds of correlation are exploited simultaneously and the cross-resolution-layer redundancy can be highly removed. Furthermore, this paper also proposes a macroblock-based adaptive in-scale technique for hybrid spatial and SNR scalability. Experimental results show that the proposed techniques can significantly improve the spatial scalability performance of H.264/MPEG-4 SVC, especially when the bit-rate ratio of lower resolution bit stream to higher resolution bit stream is considerable.

The adaptive directional lifting-based wavelet transform (ADL) locally adapts the filtering directions to the local properties of the image. In this letter, instead of using the conventional interpolation filter for the directional prediction with fractional-pel accuracy, a new two-dimensional nonseparable adaptive interpolation filter is proposed. The adaptive filter is calculated for every fractional-pel direction so as to minimize the energy of the prediction error. The tradeoff between reducing the prediction error and the overhead to code the interpolation filter is discussed. This enables coding gains of up to 0.98 dB, compared to ADL coder, and up to 2.4 dB, compared to the JPEG 2000 for typical test images.

The motion compensated temporal filtering (MCTF) technique, which is extensively used in 3-D wavelet video coding schemes nowadays, leads to signal coupling among various spatial subbands because motion alignment is introduced in the temporal filtering. Using all spatial subbands as a reference enables MCTF to fully take advantage of temporal correlation across frames but inevitably brings drifting problem in supporting spatial scalability. This paper first analyzes the signal coupling phenomenon and then proposes a quantitative model to describe signal propagation across spatial subbands during the MCTF process. The signal propagation is modeled for a single MC step based on the shifting effect of wavelet synthesis filters and then it is extended to multilevel MCTF. This model is called subband coupling aware signal propagation (SCASP) model in this paper. Based on the model, we further propose a subband coupling aware rate allocation scheme as one possible solution to the above dilemma in supporting spatial scalability. To find the optimal rate allocation among all subbands for a specified reconstruction resolution, the SCASP model is used to approximate the reconstruction process and derive the synthesis gain of each subband with regard to that reconstruction. Experimental results have fully demonstrated the advantages of our proposed rate allocation scheme in improving both objective and subjective qualities of reconstructed low-resolution video, especially at middle bit rates and high bit rates.

This paper provides an overview of the Barbell lifting coding scheme that has been adopted as common software by the MPEG ad hoc group on further exploration of wavelet video coding. The core techniques used in this scheme, such as Barbell lifting, layered motion coding, 3D entropy coding and base layer embedding, are discussed. The paper also analyzes and compares the proposed scheme with the oncoming scalable video coding (SVC) standard because the hierarchical temporal prediction technique used in SVC has a close relationship with motion compensated temporal lifting (MCTF) in wavelet coding. The commonalities and differences between these two schemes are exhibited for readers to better understand modern scalable video coding technologies. Several challenges that still exist in scalable video coding, e.g., performance of spatial scalable coding and accurate MC lifting, are also discussed. Two new techniques are presented in this paper although they are not yet integrated into the common software. Finally, experimental results demonstrate the performance of the Barbell-lifting coding scheme and compare it with SVC and another well-known 3D wavelet coding scheme, MC embedded zero block coding (MC-EZBC).

This paper presents a comparison-based test of the objective compression performance of the High Efficiency Video Coding (HEVC) standard, its format range extensions (RExt), and its draft screen content coding extensions (SCC). The current dominant standard, H.264/MPEG-4 AVC, is used as an anchor reference in the comparison. The conditions used for the comparison tests were designed to reflect relevant application scenarios and to enable a fair comparison to the maximum extent feasible – i.e., using comparable quantization settings, reference frame buffering, intra refresh periods, rate-distortion optimization decision processing, etc. It is noted that such PSNR-based objective comparisons generally provide more conservative estimates of HEVC benefit than are found in subjective studies. The experimental results show that, when compared with H.264/MPEG-4 AVC, HEVC version 1 provides a bit rate savings for equal PSNR of about 23% for all-intra coding, 34% for random access coding, and 38% for low-delay coding. This is consistent with prior studies and the general characterization that HEVC can provide about a bit rate savings of about 50% for equal subjective quality for most applications. The HEVC format range extensions provide a similar bit rate savings of about 13–25% for all-intra coding, 28–33% for random access coding, and 32–38% for low-delay coding at different bit rate ranges. For lossy coding of screen content, the HEVC screen content coding extensions achieve a bit rate savings of about 66%, 63%, and 61% for all-intra coding, random access coding, and low-delay coding, respectively. For lossless coding, the corresponding bit rate savings are about 40%, 33%, and 32%, respectively.

Screen content video coding extension of HEVC (SCC) is being developed by Joint Collaborative Team on Video Coding (JCT-VC) of ISO/IEC MPEG and ITU-T VCEG. Screen content usually features a mix of camera captured content and a significant proportion of rendered graphics, text, or animation. These two types of content exhibit distinct characteristics requiring different compression scheme to achieve better coding efficiency. This paper presents an efficient block matching schemes for coding screen content to better capture the spatial and temporal characteristics. The proposed schemes are mainly categorized as a) hash based global region block matching for intra block copy b) selective search based local region block matching for inter frame prediction c) hash based global region block matching for inter frame prediction. In the first part, a hash-based full frame block matching algorithm is designed for intra block copy to handle the repeating patterns and large motions when the reference picture constituted already decoded samples of the current picture. In the second part, a selective local area block matching algorithm is designed for inter motion estimation to handle sharp edges, high spatial frequencies and non-monotonic error surface. In the third part, a hash based full frame block matching algorithm is designed for inter motion estimation to handle repeating patterns and large motions across the temporal reference picture. The proposed schemes are compared against HM-13.0+RExt-6.0, which is the state-of-art screen content coding. The first part provides a luma BD-rate gains of -26.6%, -15.6%, -11.4% for AI, RA and LD TGM configurations. The second part provides a luma BD-rate gains of -10.1%, -12.3% for RA and LD TGM configurations. The third part provides a luma BD-rate gains of -12.2%, -11.5% for RA and LD TGM configurations.

We present a compound image compression scheme based on the dictionary-based Lempel-Ziv-Markov chain algorithm (LZMA), under the framework of High Efficiency Video Coding (HEVC). Through matching strings from the sliding window dictionary, LZMA exploits the characteristics of the repeated patterns over the text and graphics regions of compound images, and represents them compactly. To obtain high compression efficiency even for noisy text and graphics contents, we have modified LZMA to support both lossless and lossy compression. We develop and treat it as a new intramode of HEVC. Experimental results show that the proposed scheme achieves significant coding gains for compound image compression. Thanks to the introduction of the lossy LZMA, the compression performance for noisy compound images is improved for more than 5dB in terms of PSNR in comparison with the lossless LZMA scheme.

This paper presents an in-loop adaptive color-space transform for the HEVC Screen Content Coding extension. In the proposed method, the prediction residual is adaptively converted into a different color space to reduce the cross-component redundancy. After the ACT, the signal is coded following the existing HEVC framework. To keep the complexity as low as possible, fixed color-space transforms that are easily implemented with shift and add operations are utilized. Significant coding gains are achieved by this method in the current HEVC Screen Content Coding reference software with no increase of decoding runtime. The proposed method has been adopted to the HEVC Screen Content Coding extension.

Motion estimation plays an important role in video coding standards, such as H.264/AVC and HEVC. In this paper, we propose a multi-stage hash based motion estimation algorithm for HEVC, which enables hash based motion estimation for natural videos. In the proposed method, the prediction blocks significantly different from the current prediction unit will be eliminated in the motion estimation process. Locality sensitive hashing functions are used to measure the difference between the input block and predicted blocks. The proposed algorithm is implemented into the HM 12.0 software, and the simulation results show that the complexity of motion estimation is significantly reduced with negligible coding performance loss.

Fractional-pel motion compensation is very good at improving video coding efficiency, especially for camera-captured content. But for screen content, which is obtained from a computer desktop, motion vectors with integer-precision may be enough to represent the motion in different pictures. Using fractional-pel motion compensation for such content is a waste of bits. Thus, adaptive motion compensation precision is helpful for improving coding efficiency, especially for screen content coding. Usually, to select suitable motion compensation precision, multi-pass encoding is introduced, which significantly increases the encoding time. This paper presents a fast encoding algorithm for adaptive motion compensation precision used in screen content coding by hash-based block matching. With the proposed method, multi-pass encoding is avoided and most of the benefits brought by adaptive motion compensation precision are preserved. The experimental results show that with the proposed method, up to 7.7% bit saving is obtained without a significant impact on encoding time.

This paper introduces a unified framework of hash-based matching method for screen content coding. Screen content has some different characteristics from camera-captured content, such as large motion and repeating patterns. Hash-based matching is proposed to better explore the correlation in screen content, thus, improving the coding efficiency. The proposed method can handle both intra picture and inter picture block matching with variable block sizes in a unified framework. The proposed framework is also easy to be extended to handle other motion models to further improve the coding efficiency of screen content. We also develop fast encoding algorithms to make full use of the hash results. The experimental results show the proposed algorithm achieves about 12% bit saving while saving more than 25% encoding time. The bit saving is up to 57% and the encoding time saving is up to 60% for the proposed method.

This paper introduces 1-D dictionary mode designed for screen content coding. Two 1-D dictionary modes are designed to improve the coding efficiency for screen content. The first one is called normal dictionary mode, in which a virtual dictionary should be maintained and all the prediction comes from the virtual dictionary. The other one is called reconstruction based dictionary mode, where no virtual dictionary is to be maintained and all the previously reconstructed pixels in the same picture can be used for prediction. Hash based search is designed to find matching for both dictionary modes efficiently. 1-D dictionary mode with variable block sizes are also supported in the proposed scheme. The experimental results show the proposed algorithm achieves about 10% ∼ 18.4% bit saving for different coding structures. The bit saving is up to 60% for the proposed method.

Three-dimensional (3D) wavelet-based scalable video coding provides a viable alternative to standard MC-DCT coding. However, many current 3D wavelet coders experience severe boundary effects across group of pictures (GOP) boundaries. This paper proposes a memory-efficient transform technique via lifting that effectively computes wavelet transforms of a video sequence continuously on the fly, thus eliminating the boundary effects due to limited length of individual GOPs. Coding results show that the proposed scheme completely eliminates the boundary effects and gives superb video playback quality.

This paper proposes a cloud-based distributed image coding scheme (Cloud-DIC) to exploit the strong correlations with external partial-duplicate images in the cloud. It features both high coding efficiency and low encoder complexity, which makes it suitable for photo sharing on mobile devices. To get the side information in the cloud, a thumbnail of the current image is transmitted to retrieve highly correlated images and reconstruct through geometrical registration and adaptive patched-based stitching. The current image is then compressed by a transform-domain syndrome coding, bitplane by bitplane. Once a bitplane is received, the decoded high-quality image is further used to refine the side information in the cloud, which will benefit the coding of following bitplanes and the reconstruction. Experimental results on a landmark image database show that it can largely enhance the coding efficiency both subjectively and objectively with up to 5 dB gains and 58% bits saving over JPEG.

The line-based intra block copy (IntraBC) technique is newly proposed in the High Efficient Video Coding (HEVC) Range Extensions to deal with repeated patterns within a picture for screen content coding. One challenge for line-based IntraBC is the large overhead by transmitting displacement vectors (DV) for each line. In this paper, a DV prediction is proposed to reduce such an overhead. Moreover, a flipping copy based prediction is proposed to better exploit the correlations within screen content. Experimental results show that our scheme can provide a BD-rate reduction of 11.0% compare with the HEVC Range Extension anchor with the encoding time increased by 23% and no decoding time increase.