Search

The rate-distortion optimized quantization (RDOQ) provides an excellent trade-off between rate and distortion in High Efficiency Video Coding (HEVC), leading to inspiring improvement in terms of rate-distortion performance. However, its heavy use imposes high complexity on the encoder in real-world video compression applications. In this paper, we provide a comprehensive review on low complexity quantization techniques in HEVC, including both fast RDOQ and all-zero block detection. In particular, the fast RDOQ relies on rate and distortion models for rate-distortion cost estimation, such that the most appropriate quantized coefficient is selected in a low complexity way. All-zero block detection infers the all-zero block by skipping transform and quantization, in an effort to further reduce the complexity. The relationship between the two techniques is also discussed, and moreover, we also envision the future design of low complexity quantization in the upcoming Versatile Video Coding (VVC) standard.

In recent years, supervised deep learning methods have shown a great promise in dense depth estimation. However, massive high-quality training data are expensive and impractical to acquire. Alternatively, self-supervised learning-based depth estimators can learn the latent transformation from monocular or binocular video sequences by minimizing the photometric warp error between consecutive frames, but they suffer from the scale ambiguity problem or have difficulty in estimating precise pose changes between frames. In this paper, we propose a joint self-supervised deep learning pipeline for depth and ego-motion estimation by employing the advantages of adversarial learning and joint optimization with spatial-temporal geometrical constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with an absolute scale and a pre-trained pose network serves as a good starting point for direct visual odometry (DVO). DVO optimization based on spatial geometric constraints can result in a fine-grained ego-motion estimation with the additional backpropagation signals provided to the depth estimation network. Finally, the spatial and temporal domain-based reconstructed views are concatenated, and the iterative coupling optimization process is implemented in combination with the adversarial learning for accurate depth and precise ego-motion estimation. The experimental results show superior performance compared with state-of-the-art methods for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

In recent years, large amount of ground truth data is typically required to feed into the supervised depth estimation models to produce satisfactory performance, while it is usually costly and impracticable to acquire depth ground truth. In this paper, we propose a self-supervised joint deep learning pipeline for depth and pose estimation of monocular video sequences, which uses the cycle generation adversarial network structure to extend the existing reconstruction loss function based on photometric consistency. The generation function of the algorithm learns to synthesize the adjacent image to predict the depth map and the relative target pose, and the discriminant function learns the dispersion of the monocular images to correctly classify the realism of the composite image. At the same time, a reconstruction loss function based on pose consistency is used to assist the generator function in training. Extensive experimental results on the KITTI dataset show superior performance of the proposed method.

Depth compression plays an important role in 3D video coding with the typical texture-plus-depth representation. In this paper, to reduce the encoding complexity of depth map, we propose a low complexity adaptive View Synthesis Optimization scheme for the 3D extension of high efficiency video coding (3D-HEVC) standard. More specifically, we distinguish the coding tree units (CTUs) based on the influence of depth map compression on the quality of rendered synthesized view, and classify them into two categories, including synthesized view distortion change (SVDC) based and view synthesis distortion estimation (VSDE) based CTUs. In this manner, we can dynamically distinguish the CTUs and apply different rate-distortion optimization strategies. Moreover, for VSDE based CTUs, a new distortion model is proposed to infer the distortion of the synthesized view based on the depth distortion and texture characteristics. As such, we can achieve a good trade-off between the rate-distortion performance and computational complexity for depth map coding. Experimental results also confirm that the proposed scheme is effective in reducing the encoding complexity with ignorable rate-distortion performance loss compared with the state-of-the-art scheme in the 3D-HEVC platform.

In this paper, we overview the recent work on entropy of primitive (EoP), including its concept, design, extension, and mathematical analysis in evaluating the visual information of natural images. The design philosophy of EoP is establishing an entropy model that quantifies the visual information based on patch-level sparse representation, due to the close relationship between sparse representation and the hierarchical cognitive process of human perception. Furthermore, based on the concept and definition of EoP, we also demonstrate several applications, including just noticeable difference estimation and visual quality assessment. The future research directions of visual information evaluation are also envisioned, where we can perceive both promises and challenges.

We propose an inter-view dependency-based rate control (RC) algorithm for 3D extension of high efficiency video coding (HEVC). First, considering the rate-distortion (R-D) dependency between the synthesized views and the input views (including texture videos and depth maps), a synthesized view distortion model is derived. Second, a novel distortion model for dependent views is proposed by investigating the inter-view dependency between the base and dependent views. Based on these two distortion models, a joint optimal bit allocation strategy (including texture/depth level, view level, and frame level) is developed to allocate target bits for both texture videos and depth maps of different views. Furthermore, an effective initial quantization parameter decision scheme considering the characteristics of the input video content is presented. Extensive experimental results exhibit that the proposed scheme achieves a higher rate control accuracy and provides a better R-D performance than the state-of-the-art RC algorithms.

In this paper, we propose a low complexity Hybrid View Synthesis Optimization (HVSO) scheme for 3D extension of high efficiency video coding (3D-HEVC). In particular, based on the analyses regarding the impact of depth map compression on the quality of synthesized view rendering, we locally classify the coding tree units (CTUs) of depth map into two categories, including synthesized view distortion change (SVDC) based and view synthesis distortion estimation (VSDE) based CTUs. This allows us to dynamically distinguish the rate-distortion optimization process of these CTUs and obtain the synthesized view distortion of the VSDE based ones in a high efficiency way. More specifically, a new interpolation-based distortion estimation algorithm dedicated for VSDE based CTUs is proposed such that the actual view rendering process can be successfully avoided. In this manner, a good trade-off between the rate-distortion performance and computational complexity in depth map compression is achieved. Experimental results show that with our proposed HVSO method, 75% of the computational complexity on average can be saved for the view synthesis optimization process with only 1.6% BD-rate loss compared with the state-of-the-art scheme in the 3D-HEVC platform.

In this paper, we propose a new frame level rate control algorithm for the high efficiency video coding (HEVC) based 3D video (3DV) compression standard. In the proposed scheme, a new initial quantization parameter (QP) decision scheme is provided, and the bit allocation for each view is investigated to smooth the bitrate fluctuation and reach accurate rate control. Meanwhile, a simplified complexity estimation method for the extended view is introduced to reduce the computational complexity while improves the coding performance. The experimental results on 3DV test sequences demonstrate that the proposed algorithm can achieve better R-D performance and more accurate rate control compared to the benchmark algorithms in HTM10.0. The maximum performance improvement can be up to 12.4% and the average BD-rate gain for each view is 5.2%, 6.5% and 6.6% respectively.