Search

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for $Y'C_{B}C_{R}$ and of about 6% to 14% for $R'G'B'$ screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for $Y'C_{B}C_{R}$ and 43% to 66% for $R'G'B'$ screen content. [ABSTRACT FROM AUTHOR]

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%. [ABSTRACT FROM AUTHOR]

Efficient compression of omnidirectional video is important for emerging virtual reality applications. To compress this kind of video, each frame is projected to a 2D plane [e.g., equirectangular projection (ERP) map] first, adapting to the input format of existing video coding systems. At the display side, an inverse projection is applied to the reconstructed video to restore signals in spherical domain. Such a projection, however, makes presentation and encoding in different domains. Thus, an encoder agnostic to the projection performs inefficiently. In this paper, we analyze how a projection influences the distortion measurements in different domains. Based on the analysis, we propose a scheme to optimize the encoding process based on signals’ distortion in spherical domain. With the proposed optimization, an average 4.31% (up to 9.67%) luma BD-rate reduction is achieved for ERP in random access configuration. The corresponding bit saving is averagely 10.84% (up to 34.44%) when considering the viewing field being $\pi /2$. The proposed method also benefits other projections and viewport settings, with a marginal complexity increase. [ABSTRACT FROM AUTHOR]

Transform, as one of the most important modules of mainstream video coding systems, seems very stable over the past several decades. However, recent developments indicate that bringing more options for transform can lead to coding efficiency benefits. In this paper, we go further to investigate how the coding efficiency can be improved over the state-of-the-art method by adapting a transform for each block. We present a variable block-sized signal-dependent transforms (SDTs) design based on the High Efficiency Video Coding (HEVC) framework. For a coding block ranged from $4\times4$ to $32\times32$ , we collect a quantity of similar blocks from the reconstructed area and use them to derive the Karhunen–Loève transform. We avoid sending overhead bits to denote the transform by performing the same procedure at the decoder. In this way, the transform for every block is tailored according to its statistics, to be signal-dependent. To make the large block-sized SDTs feasible, we present a fast algorithm for transform derivation. Experimental results show the effectiveness of the SDTs for different block sizes, which leads to up to 23.3% bit-saving. On average, we achieve BD-rate saving of 2.2%, 2.4%, 3.3%, and 7.1% under AI-Main10, RA-Main10, RA-Main10, and LP-Main10 configurations, respectively, compared with the test model HM-12 of HEVC. The proposed scheme has also been adopted into the joint exploration test model for the exploration of potential future video coding standard. [ABSTRACT FROM AUTHOR]

This paper proposes a deep learning method for intra prediction. Different from traditional methods utilizing some fixed rules, we propose using a fully connected network to learn an end-to-end mapping from neighboring reconstructed pixels to the current block. In the proposed method, the network is fed by multiple reference lines. Compared with traditional single line-based methods, more contextual information of the current block is utilized. For this reason, the proposed network has the potential to generate better prediction. In addition, the proposed network has good generalization ability on different bitrate settings. The model trained from a specified bitrate setting also works well on other bitrate settings. Experimental results demonstrate the effectiveness of the proposed method. When compared with high efficiency video coding reference software HM-16.9, our network can achieve an average of 3.4% bitrate saving. In particular, the average result of 4K sequences is 4.5% bitrate saving, where the maximum one is 7.4%. [ABSTRACT FROM AUTHOR]

Screen content coding plays an important role in many applications. Conventional reference picture management (RPM) strategies developed for natural content may not work well for screen content. This is because many regions in screen content remain static for a long time, causing a lot of repetitive contents to stay in the decoded picture buffer. The repetitive contents are not conducive to inter prediction, but still occupy valuable memory. This paper proposes a diversity-based RPM scheme for screen content coding. The concept of diversity is introduced for the reference picture set (RPS) to help formulate the RPM problem. By maximizing the diversity of RPS, more potentially better predictions are provided. Better compression performance can then be achieved. Meanwhile, the proposed scheme is nonnormative and compatible with existing video coding standards, such as High Efficiency Video Coding. The experimental results show that, for low delay screen content coding, the bit saving of the proposed scheme is 4.9% on average and up to 13.7%, without increasing encoding time. [ABSTRACT FROM AUTHOR]

In the latest High Efficiency Video Coding (HEVC) development, i.e., HEVC screen content coding extensions (HEVC-SCC), a hash-based inter-motion search/block matching scheme is adopted in the reference test model, which brings significant coding gains to code screen content. However, the hash table generation itself may take up to half the encoding time and is thus too complex for practical usage. In this paper, we propose a hierarchical hash design and the corresponding block matching scheme to significantly reduce the complexity of hash-based block matching. The hierarchical structure in the proposed scheme allows large block calculation to use the results of small blocks. Thus, we avoid redundant computation among blocks with different sizes, which greatly reduces complexity without compromising coding efficiency. The experimental results show that compared with the hash-based block matching scheme in the HEVC-SCC test model (SCM)-6.0, the proposed scheme reduces about 77% of hash processing time, which leads to 12% and 16% encoding time savings in random access (RA) and low-delay B coding structures. The proposed scheme has been adopted into the latest SCM. A parallel implementation of the proposed hash table generation on graphics processing unit (GPU) is also presented to show the high parallelism of the proposed scheme, which achieves more than 30 frames/s for 1080p sequences and 60 frames/s for 720p sequences. With the fast hash-based block matching integrated into x265 and the hash table generated on GPU, the encoder can achieve 11.8% and 14.0% coding gains on average for RA and low-delay P coding structures, respectively, for real-time encoding. [ABSTRACT FROM PUBLISHER]

Unlike camera-captured video, screen content (SC) often contains a lot of repeating patterns, which makes some blocks used as references much more important than others. However, conventional rate-distortion optimization (RDO) schemes in video coding do not consider the dependence among image blocks, which often leads to a locally optimal parameter selection, especially for SC. In this paper, we present a weighted RDO scheme for SC coding (SCC), in which the repeating characteristics are taken into account when deciding RD tradeoff for each block. For one block, the number being referenced by the current picture and following pictures is estimated and based on the number, we set a proper weight in the RDO process to reflect its importance from a global point of view. To estimate the number being referenced, we propose a hash-based method to approximate the results to avoid the complexity of direct search. Experimental results show that compared with the High Efficiency Video Coding SCC reference software, 10.1%, 14.5%, and 2.2% on average and up to 25.7%, 39.8%, and 4.6% bit saving can be achieved by considering weights provided by our scheme for hierarchical-B, IBBB, and all intra coding structures, respectively. Thanks to our hash-based design, the complexity increase brought by the proposed scheme is marginal. [ABSTRACT FROM PUBLISHER]

A screen content coding (SCC) extension to High Efficiency Video Coding (HEVC) is currently under development by the Joint Collaborative Team on Video Coding, which is a joint effort from the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. The main goal of the HEVC-SCC standardization effort is to enable significantly improved compression performance for videos containing a substantial amount of still or moving rendered graphics, text, and animation rather than, or in addition to, camera-captured content. This paper provides an overview of the technical features and characteristics of the current HEVC-SCC test model and related coding tools, including intra-block copy, palette mode, adaptive color transform, and adaptive motion vector resolution. The performance of the SCC extension is compared against existing standards in terms of bitrate savings at equal distortion. [ABSTRACT FROM PUBLISHER]

The Range Extensions (RExt) of the High Efficiency Video Coding (HEVC) standard have recently been approved by both ITU-T and ISO/IEC. This set of extensions targets video coding applications in areas including content acquisition, postproduction, contribution, distribution, archiving, medical imaging, still imaging, and screen content. In addition to the functionality of HEVC Version 1, RExt provide support for monochrome, 4:2:2, and 4:4:4 chroma sampling formats as well as increased sample bit depths beyond 10 bits per sample. This extended functionality includes new coding tools with a view to provide additional coding efficiency, greater flexibility, and throughput at high bit depths/rates. Improved lossless, near-lossless, and very high bit-rate coding is also a part of the RExt scope. This paper presents the technical aspects of HEVC RExt, including a discussion of RExt profiles, tools, applications, and provides experimental results for a performance comparison with previous relevant coding technology. When compared with the High 4:4:4 Predictive Profile of H.264/Advanced Video Coding (AVC), the corresponding HEVC 4:4:4 RExt profile provides up to $\sim 25$ %, $\sim 32$ %, and $\sim 36$ % average bit-rate reduction at the same PSNR quality level for intra, random access, and low delay configurations, respectively. [ABSTRACT FROM PUBLISHER]

With multimedia flourishing on the Web, it is easy to find similar images for a query, especially landmark images. Traditional image coding, such as JPEG, cannot exploit correlations with external images. Existing vision-based approaches are able to exploit such correlations by reconstructing from local descriptors but cannot ensure the pixel-level fidelity of the reconstruction. In this paper, a cloud-based distributed image coding (Cloud-DIC) scheme is proposed to exploit external correlations for mobile photo uploading. For each input image, a thumbnail is transmitted to retrieve correlated images and reconstruct it in the cloud by geometrical and illumination registrations. Such a reconstruction serves as the side information (SI) in the Cloud-DIC. The image is then compressed by a transform-domain syndrome coding to correct the disparity between the original image and the SI. Once a bitplane is received in the cloud, an iterative refinement process is performed between the final reconstruction and the SI. Moreover, a joint encoder/decoder mode decision at block, frequency, and bitplane levels is proposed to adapt to different correlations. Experimental results on a landmark image database show that the Cloud-DIC can largely enhance the coding efficiency both subjectively and objectively, with up to 5-dB gains and 70% bits saving over JPEG with arithmetic coding, and perform comparably at low bitrates with the intra coding of the High Efficiency Video Coding standard with a much lower encoder complexity. [ABSTRACT FROM AUTHOR]

Although the High Efficiency Video Coding (HEVC) standard significantly improves the coding efficiency of video compression, it is unacceptable even in offline applications to spend several hours compressing 10 s of high-definition video. In this paper, we propose using a multicore central processing unit (CPU) and an off-the-shelf graphics processing unit (GPU) with 3072 streaming processors (SPs) for HEVC fast encoding, so that the speed optimization does not result in loss of coding efficiency. There are two key technical contributions in this paper. First, we propose an algorithm that is both parallel and fast for the GPU, which can utilize 3072 SPs in parallel to estimate the motion vector (MV) of every prediction unit (PU) in every combination of the coding unit (CU) and PU partitions. Furthermore, the proposed GPU algorithm can avoid coding efficiency loss caused by the lack of a MV predictor (MVP). Second, we propose a fast algorithm for the CPU, which can fully utilize the results from the GPU to significantly reduce the number of possible CU and PU partitions without any coding efficiency loss. Our experimental results show that compared with the reference software, we can encode high-resolution video that consumes 1.9% of the CPU time and 1.0% of the GPU time, with only a 1.4% rate increase. [ABSTRACT FROM AUTHOR]

By considering the increasing importance of screen contents, the high efficiency video coding (HEVC) standard includes screen content coding as one of its requirements. In this paper, we demonstrate that enabling frame level block searching in HEVC can significantly improve coding efficiency on screen contents. We propose a hash-based block matching scheme for the intra block copy mode and the motion estimation process, which enables frame level block searching in HEVC without changing the HEVC syntaxes. In the proposed scheme, the blocks sharing the same hash values with the current block are selected as prediction candidates. Then the hash-based block selection is employed to select the best candidates. To achieve the best coding efficiency, the rate distortion optimization is further employed to improve the proposed scheme by balancing the coding cost of motion vectors and prediction difference. Compared with HEVC, the proposed scheme achieves 21% and 37% bitrate saving with all intra and low delay configurations with encoding time reduction . Up to 59% bitrate saving can be achieved on sequences with large motions. [ABSTRACT FROM PUBLISHER]

High Efficiency Video Coding (HEVC) provides superior coding efficiency than previous video coding standards at the cost of increasing encoding complexity. The complexity increase of motion estimation (ME) procedure is rather significant, especially when considering the complicated partitioning structure of HEVC. To fully exploit the coding efficiency brought by HEVC requires a huge amount of computations. In this paper, we analyze the ME structure in HEVC and propose a parallel framework to decouple ME for different partitions on many-core processors. Based on local parallel method (LPM), we first use the directed acyclic graph (DAG)-based order to parallelize coding tree units (CTUs) and adopt improved LPM (ILPM) within each CTU (DAGILPM), which exploits the CTU-level and prediction unit (PU)-level parallelism. Then, we find that there exist completely independent PUs (CIPUs) and partially independent PUs (PIPUs). When the degree of parallelism (DP) is smaller than the maximum DP of DAGILPM, we process the CIPUs and PIPUs, which further increases the DP. The data dependencies and coding efficiency stay the same as LPM. Experiments show that on a 64-core system, compared with serial execution, our proposed scheme achieves more than 30 and 40 times speedup for $1920\times 1080$ and $2560\times 1600$ video sequences, respectively. [ABSTRACT FROM AUTHOR]

Screen content like cartoons, captures of typical computer screens or video with text overlay or news ticker is an important category of video, which needs new techniques beyond the existing video coding techniques. In this paper, we analyze the characteristics of screen content and coding efficiency of HEVC on screen content. We propose a new coding scheme, which adopts a non-transform representation, separating screen content into color component and structure component. Based on the proposed representation, two coding modes are designed for screen content to exploit the directional correlation and non-translational changes in screen video sequences. The proposed scheme is then seamlessly incorporated into the HEVC structure and implemented into HEVC range extension reference software HM9.0. Experimental results show that the proposed scheme achieves up to 52.6% bitrate saving compared with HM9.0. On average, 35.1%, 29.2% and 23.6% bitrate saving are achieved with intra, random-access and low-delay configurations, respectively. The visual quality of the decoded video sequences is also significantly improved by reducing ringing artifacts around sharp edges and reserving the shape of text without blur. [ABSTRACT FROM PUBLISHER]

Motion compensation with multiple reference pictures has been widely used during the development of the emerging High Efficiency Video Coding (HEVC) standard, which greatly helps to improve the coding efficiency. Usually, a heuristic strategy is exploited to use the nearest reconstructed pictures as references. However, such a strategy may not be efficient on all occasions, especially when different content characteristics and coding settings are considered. In this paper, we investigate how to manage reference pictures so as to achieve better rate-distortion performance under the memory constraint of the decoded picture buffer at the decoder. We formulate the reference picture management as an optimization problem and approximate its optimal solution. Moreover, we explore how to adjust quality for each picture according to the reference structure to further improve coding efficiency. For some coding cases, where a complicated encoder optimization is unaffordable, we also develop fast algorithms to get the most benefit from reference picture selection. Among them, one strategy has been adopted by the HEVC software and common test conditions to generate the anchor. Experimental results show that the proposed full search algorithm and fast search algorithms achieve significant bitrate reduction. [ABSTRACT FROM PUBLISHER]

Computers are developing along with a new trend from the dual-core and quad-core processors to ones with tens or even hundreds of cores. Multimedia, as one of the most important applications in computers, has an urgent need to design parallel coding algorithms for compression. Taking intraframe/image coding as a start point, this paper proposes a pure line-by-line coding scheme (LBLC) to meet the need. In LBLC, an input image is processed line by line sequentially, and each line is divided into small fixed-length segments. The compression of all segments from prediction to entropy coding is completely independent and concurrent at many cores. Results on a general-purpose computer show that our scheme can get a 13.9 times speedup with 15 cores at the encoder and a 10.3 times speedup at the decoder. Ideally, such near-linear speeding relation with the number of cores can be kept for more than 100 cores. In addition to the high parallelism, the proposed scheme can perform comparatively or even better than the H.264 high profile above middle bit rates. At near-lossless coding, it outperforms H.264 more than 10 dB. At lossless coding, up to 14% bit-rate reduction is observed compared with H.264 lossless coding at the high 4:4:4 profile. [ABSTRACT FROM PUBLISHER]

Over the past few decades, many studies on image and video compression have found various approaches to the exploitation of spatial and temporal local correlations. However, we believe it is imperative to find more efficient methods to progress the development of image and video compression. In this paper, we first study spatial non-local correlation, deducing that there exist strong correlations in non-local regions. However, it is rather difficult to make use of these non-local correlations while simultaneously minimizing overhead. To solve this problem, we propose the signal-dependent transform (SDT), which is derived from decoded non-local blocks that are selected by matching neighboring pixels. Since the encoder and decoder can use the same methods to derive the proposed transform, we can successfully eliminate overhead. Finally, we have implemented the proposed transform into the Key Technology Area (KTA) software to exploit both spatial and temporal non-local correlations. The experimental results show that the coding gain over KTA can be as high as 1.4 dB in intra-frame coding, and up to 1.0 dB in inter-frame coding. We believe we have effectively created an alternate method to improve image and video compression. [ABSTRACT FROM PUBLISHER]

While directional adaption is introduced into traditional transforms, different orders of two 1-D transforms will result in different results of one 2-D transform. Based upon an anisotropic image model, this paper analyzes the effect of transform orders in terms of theoretical coding gain. Our results reveal that the transform orders have little effect on the coding gain with full decomposition, good directional modes and good interpolation. However, in practical compression schemes, since high-pass bands are not decomposed fully because of the consideration on complexity, different transform orders have different coding performances, which can be solved by an adaptive transform order. Motivated by our analyzed results, a directional filtering transform (dFT, in order to distinguish from the common usage on DFT) is proposed in this paper to better exploit correlations among samples in H.264 intraframe coding. It provides an evenly distributed set of prediction modes with an adaptive transform order. Both interblock and intrablock correlations are exploited in this scheme. Experimental results in H.264 intraframe coding demonstrate its superiority both objectively and subjectively. [ABSTRACT FROM PUBLISHER]