Search

In this paper we combine video compression and modern image processing methods. Iterative filter methods for prediction signals based on classic inpainting methods are introduced and extensive parameter tests are described. In order to construct an alternative prediction filter for video coding, techniques originally employed for inpainting are applied. Thereby, the structures of the underlying prediction were incorporated into the filter construction making it signal adaptive. The resulting optimization problem is solved using the so-called Alternating Direction Method of Multipliers (ADMM). The undertaken novel parameter tests are described and it is shown that they improve the coding efficiency of the tool. The suggested filter is embedded into a software based on HEVC with additional QTBT (Quadtree plus Binary Tree) and MTT (Multi-Type-Tree) block structure. Overall, the proposed filter method obtains average bitrate savings of 1.35% at an average encoder runtime increase of 28% and decoder runtime increase of 38%. UHD test sequences achieve bitrate savings of up to 3.66% for Random Access. [ABSTRACT FROM AUTHOR]

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study. [ABSTRACT FROM AUTHOR]

Recently, there has been a significant progress in the research and development of the high dynamic range (HDR) video technology and the state-of-the-art video pipelines are able to offer a higher bit depth support to capture, store, encode, and display HDR video content. In this paper, we introduce a novel HDR video compression algorithm, which uses a perceptually uniform color opponent space, a novel perceptual transfer function to encode the dynamic range of the scene, and a novel error minimization scheme for accurate chroma reproduction. The proposed algorithm was objectively and subjectively evaluated against four state-of-the-art algorithms. The objective evaluation was conducted across a set of 39 HDR video sequences, using the latest x265 10-bit video codec along with several perceptual and structural quality assessment metrics at 11 different quality levels. Furthermore, a rating-based subjective evaluation ($n=40$) was conducted with six sequences at two different output bitrates. Results suggest that the proposed algorithm exhibits the lowest coding error amongst the five algorithms evaluated. Additionally, the rate–distortion characteristics suggest that the proposed algorithm outperforms the existing state-of-the-art at bitrates ≥ 0.4 bits/pixel. [ABSTRACT FROM AUTHOR]

We investigate video classification via a two-stream convolutional neural network (CNN) design that directly ingests information extracted from compressed video bitstreams. Our approach begins with the observation that all modern video codecs divide the input frames into macroblocks (MBs). We demonstrate that selective access to MB motion vector (MV) information within compressed video bitstreams can also provide for selective, motion-adaptive, MB pixel decoding (a.k.a., MB texture decoding). This in turn allows for the derivation of spatio-temporal video activity regions at extremely high speed in comparison to conventional full-frame decoding followed by optical flow estimation. In order to evaluate the accuracy of a video classification framework based on such activity data, we independently train two CNN architectures on MB texture and MV correspondences and then fuse their scores to derive the final classification of each test video. Evaluation on two standard data sets shows that the proposed approach is competitive with the best two-stream video classification approaches found in the literature. At the same time: 1) a CPU-based realization of our MV extraction is over 977 times faster than GPU-based optical flow methods; 2) selective decoding is up to 12 times faster than full-frame decoding; and 3) our proposed spatial and temporal CNNs perform inference at 5 to 49 times lower cloud computing cost than the fastest methods from the literature. [ABSTRACT FROM AUTHOR]

A residue number system (RNS) has an inherent parallel structure that can be utilized for improving computer hardware systems. An RNS represents large integer numbers as a smaller integer set, or residues of a modulo set, without carry propagation between them. Hence mathematical operations, such as addition or subtraction, can be performed on the residues independently. This paper proposes an RNS implementation of motion estimation for the latest video coding standard known as high-efficiency video coding (HEVC) or H.265. Since motion estimation is the most computationally intensive task in video coding, several simplified algorithms are proposed for mitigating the problem, but the majority of them result in a worsening peak signal-to-noise ratio (PSNR) or bit-rate performance, or sometimes both. This paper also proposes a modified algorithm based on a test-zone (TZ) search algorithm, a widely used fast-search algorithm with good rate-distortion performance, suitable for hardware implementation for encoding ultra-high-definition videos in real time. The results show that worst-case PSNR degradation and bit-rate increases compared with the TZ search in the HEVC reference software implementation are negligible, and the hardware gate count is less than for many other designs in the literature. [ABSTRACT FROM AUTHOR]

A video compression framework based on spatio-temporal resolution adaptation (ViSTRA) is proposed, which dynamically resamples the input video spatially and temporally during encoding, based on a quantisation-resolution decision, and reconstructs the full resolution video at the decoder. Temporal upsampling is performed using frame repetition, whereas a convolutional neural network super-resolution model is employed for spatial resolution upsampling. ViSTRA has been integrated into the high efficiency video coding reference software (HM 16.14). Experimental results verified via an international challenge show significant improvements, with BD-rate gains of 15% based on PSNR and an average MOS difference of 0.5 based on subjective visual quality tests. [ABSTRACT FROM AUTHOR]

Recently, multiple ($k, n$) region-incrementing VCSs (RIVCSs) have been proposed that can gradually reconstruct secrets in a single image. In ($k, n$)-RIVCS, the secret image is subdivided into multiple regions in such a way that any $t$ shadow images, where $k \le t \le n$ , can be used to reveal the secret in the ($t-k+1$)th region. This region-incrementing property provides progressive decoding. However, the regions in previous RIVCSs are disjointed: no two regions have overlapping areas. In this paper, we discuss a ($k, n$) region-in-region progressive VCS (RPVCS). Our ($k, n$)-RPVCS has ($n-k+1$) secrecy-level regions, and each region can be located within any previous region. This new type of region allocation not only provides more areas in which to hide the secret compared with the non-overlapping regions in RIVCS but also provides different presentation methods in progressive decoding. [ABSTRACT FROM AUTHOR]

Various commercial video surveillance systems have become popular in recent years. The increasing demand for video surveillance systems which support high resolutions and operate with battery power makes it important to reduce the power consumption and to prolong battery lifetimes. H.264/AVC video coding standard and HEVC standard are widely used in many video surveillance systems due to their high compression efficiency. However, the high computation complexity of video coding standards requires a considerable amount of power consumption. This paper proposes a novel low-power surveillance video coding system that reduces both the internal power consumption inside the video encoder chip and the external power consumption by the external memory outside the chip. The proposed algorithm attempts to segment the input video into the foreground and background (BG) according to integer motion estimation (IME) information and adaptively controls the complexity of video compression modules according to the segmentation results. In order to reduce the external power consumption further, adaptive frame memory compression (FMC) using the segmentation results from the BG subtraction is proposed. The adaptive selection of the compression ratios and processing units for the FMC significantly improves the compression efficiency of the FMC. The proposed system reduces power consumption by up to 61.6% compared to a conventional video surveillance system. [ABSTRACT FROM PUBLISHER]

To save manufacturing cost, most color digital video cameras employ a single-sensor technology with a red–green–blue (RGB) color filter array (CFA) to capture real-world scenes. Due to only one primary color measured at each pixel location, the captured videos are usually referred to as the mosaic videos. For the purposes of economical storage and transmission, it is very important to achieve a good tradeoff between the quality and bitrate when compressing mosaic videos with different RGB-CFA structures. In this paper, based on mathematical optimization technique, a novel chroma subsampling strategy is presented for compressing mosaic videos with arbitrary RGB-CFA structures in H.264/AVC and High Efficiency Video Coding (HEVC). For each $2 \times 2$ YUV block to be subsampled with 4:2:0 format, the proposed strategy determines the proper sampled $U$ and $V$ components by minimizing, prior to compression, the quality distortion between the original colocated mosaic block and the mosaic block conversed from the current subsampled YUV block. Through the mathematical optimization formulated in the proposed strategy, the significance of the sampled $U$ and $V$ components for reconstructing R, G, and B pixels can be simultaneously taken into consideration. The experimental results demonstrate that the proposed chroma subsampling strategy has the best quality and bitrate tradeoff at a similar execution time requirement for compressing mosaic videos with arbitrary RGB-CFA structures in H.264/AVC and HEVC compared with the state-of-the-art ones by Chen et al. and Yang et al. as well as the three commonly used ones. [ABSTRACT FROM AUTHOR]

This paper provides an overview of Scalable High efficiency Video Coding (SHVC), the scalable extensions of the High Efficiency Video Coding (HEVC) standard, published in the second version of HEVC. In addition to the temporal scalability already provided by the first version of HEVC, SHVC further provides spatial, signal-to-noise ratio, bit depth, and color gamut scalability functionalities, as well as combinations of any of these. The SHVC architecture design enables SHVC implementations to be built using multiple repurposed single-layer HEVC codec cores, with the addition of interlayer reference picture processing modules. The general multilayer high-level syntax design common to all multilayer HEVC extensions, including SHVC, MV-HEVC, and 3D HEVC, is described. The interlayer reference picture processing modules, including texture and motion resampling and color mapping, are also described. Performance comparisons are provided for SHVC versus simulcast HEVC and versus the scalable video coding extension to H.264/advanced video coding. [ABSTRACT FROM PUBLISHER]

Depth image compression is important for compact representation of 3D visual data in texture-plus-depth format, where texture and depth maps from one or more viewpoints are encoded and transmitted. A decoder can then synthesize a freely chosen virtual view via depth-image-based rendering using nearby coded texture and depth maps as reference. Further, depth information can be used in other image processing applications beyond view synthesis, such as object identification, segmentation, and so on. In this paper, we leverage on the observation that neighboring pixels of similar depth have similar motion to efficiently encode depth video. Specifically, we divide a depth block containing two zones of distinct values (e.g., foreground and background) into two arbitrarily shaped regions (sub-blocks) along the dividing boundary before performing separate motion prediction (MP). While such arbitrarily shaped sub-block MP can lead to very small prediction residuals (resulting in few bits required for residual coding), it incurs an overhead to transmit the dividing boundaries for sub-block identification at decoder. To minimize this overhead, we first devise a scheme called arithmetic edge coding (AEC) to efficiently code boundaries that divide blocks into sub-blocks. Specifically, we propose to incorporate the boundary geometrical correlation in an adaptive arithmetic coder in the form of a statistical model. Then, we propose two optimization procedures to further improve the edge coding performance of AEC for a given depth image. The first procedure operates within a code block, and allows lossy compression of the detected block boundary to lower the cost of AEC, with an option to augment boundary depth pixel values matching the new boundary, given the augmented pixels do not adversely affect synthesized view distortion. The second procedure operates across code blocks, and systematically identifies blocks along an object contour that should be coded using sub-block MP via a rate-distortion optimized trellis. Experimental results show an average overall bitrate reduction of up to 33% over classical H.264/AVC. [ABSTRACT FROM AUTHOR]

Saliency detection is widely used to extract regions of interest in images for various image processing applications. Recently, many saliency detection models have been proposed for video in uncompressed (pixel) domain. However, video over Internet is always stored in compressed domains, such as MPEG2, H.264, and MPEG4 Visual. In this paper, we propose a novel video saliency detection model based on feature contrast in compressed domain. Four types of features including luminance, color, texture, and motion are extracted from the discrete cosine transform coefficients and motion vectors in video bitstream. The static saliency map of unpredicted frames (I frames) is calculated on the basis of luminance, color, and texture features, while the motion saliency map of predicted frames (P and B frames) is computed by motion feature. A new fusion method is designed to combine the static saliency and motion saliency maps to get the final saliency map for each video frame. Due to the directly derived features in compressed domain, the proposed model can predict the salient regions efficiently for video frames. Experimental results on a public database show superior performance of the proposed video saliency detection model in compressed domain. [ABSTRACT FROM AUTHOR]

Among the existing block partitioning schemes, the pattern-based video coding (PVC) has already established its superiority at low bit-rate. Its innovative segmentation process with regular-shaped pattern templates is very fast as it avoids handling the exact shape of the moving objects. It also judiciously encodes the pattern-uncovered background segments capturing high level of interblock temporal redundancy without any motion compensation, which is favoured by the rate-distortion optimizer at low bit-rates. The existing PVC technique, however, uses a number of content-sensitive thresholds and thus setting them to any prede- fined values risks ignoring some of the macroblocks that would otherwise be encoded with patterns. Furthermore, occluded background can potentially degrade the performance of this technique. In this paper, a robust PVC scheme is proposed by removing all the content-sensitive thresholds, introducing a new similarity metric, considering multiple top-ranked patterns by the rate-distortion optimizer, and refining the Lagrangian multiplier of the H.264 standard for efficient embedding. A novel pattern-based residual encoding approach is also integrated to address the occlusion issue. Once embedded into the H.264 Baseline profile, the proposed PVC scheme improves the image quality perceptually significantly by at least 0.5 dB in low bit-rate video coding applications. A similar trend is observed for moderate to high bit-rate applications when the proposed scheme replaces the bi-directional predictive mode in the H.264 High profile. [ABSTRACT FROM AUTHOR]

This paper presents a method to reduce the computation and memory access for variable block size motion estimation (ME) using pixel truncation. Previous work has focused on implementing pixel truncation using a fixed-block- size (16 x 16 pixels) ME. However, pixel truncation fails to give satisfactory results for smaller block partitions. In this paper, we analyze the effect of truncating pixels for smaller block partitions and propose a method to improve the frame prediction. Our method is able to reduce the total computation and memory access compared to conventional full-search method without significantly degrading picture quality. With unique data arrangement, the proposed architectures are able to save up to 53% energy compared to the conventional full-search architecture. This makes such architectures attractive for H.264 application in future mobile devices. [ABSTRACT FROM AUTHOR]

This paper presents new fast full search motion estimation (ME) algorithms using discrete cosine transform (DCT) coefficients as a gradient magnitude. The proposed algorithms use intra DCT coefficients to arrange the matching scan order. The proposed algorithm 1 uses some of DCT coefficients based on the relationship between gradient magnitude and DCT coefficients, and the proposed algorithm 2 uses all of DCT alternating current (AC) coefficients in order to predict the fluctuation of a block. The proposed algorithms significantly reduce the computational complexity of the conventional algorithm while maintaining the visual quality of video. [ABSTRACT FROM AUTHOR]

Following recent commercial availability of autostereoscopic 3D displays that allow 3D visual data to be viewed without the use of special headgear or glasses, it is anticipated that the applications of 3D video will increase rapidly in near future. In this paper we propose a joint source channel coding scheme for depth image-based rendering based 3D video coding. We considered different source and channel coding rates to find the optimum coding performance under a given channel bit rate for a WiMAX based communication channel. When the optimum bit allocation combination for color and depth image sequences are found, different protection levels have been considered for coding both image sequences. Finally, an optimum protection levels are proposed for the best video quality. [ABSTRACT FROM AUTHOR]

Subpixel motion estimation plays an important role in compression efficiency within modern video codecs such as MPEG2, MPEG4, and H.264. Subpixel motion estimation is implemented within these standards using interpolated values at 1/2 or 1/4 pixel accuracy. Such interpolation gives a good reduction in residual energy for each predicted macroblock and, therefore, improves compression. However, this leads to a significant increase in computational complexity at the encoder. This is especially true for H.264 where the cost of an exhaustive set of macroblock segmentations need to be estimated in order to obtain an optimal mode for prediction. This paper presents a novel interpolation-free scheme for subpixel motion estimation using the result of the full pixel sum of absolute difference distribution of each motion compensated block applied to an H.264 encoder. This system produces reduced complexity motion estimation with a controllable tradeoff between compression performance and encoder speed. These methods facilitate the generation of a real time software H.264 encoder. [ABSTRACT FROM AUTHOR]