Search

The constant increase in multimedia Internet traffic in the form of video streaming requires new solutions for efficient video coding to save bandwidth and network resources. HTTP adaptive streaming (HAS), the most widely used solution for video streaming, allows the client to adaptively select the bitrate according to the transmission conditions. For this purpose, multiple presentations of the same video content are generated on the video server, which contains video sequences encoded at different bitrates with resolution adjustment to achieve the best Quality of Experience (QoE). This set of bitrate–resolution pairs is called a bitrate ladder. In addition to the traditional one-size-fits-all scheme for the bitrate ladder, context-aware solutions have recently been proposed that enable optimum bitrate–resolution pairs for video sequences of different complexity. However, these solutions use only spatial resolution for optimization, while the selection of the optimal combination of spatial and temporal resolution for a given bitrate has not been sufficiently investigated. This paper proposes bit-ladder optimization considering spatiotemporal features of video sequences and usage of optimal spatial and temporal resolution related to video content complexity. Optimization along two dimensions of resolution significantly increases the complexity of the problem and the approach of intensive encoding for all spatial and temporal resolutions in a wide range of bitrates, for each video sequence, is not feasible in real time. In order to reduce the level of complexity, we propose a data augmentation using a neural network (NN)-based model. To train the NN model, we used seven video sequences of different content complexity, encoded with the HEVC encoder at five different spatial resolutions (SR) up to 4K. Also, all video sequences were encoded using four frame rates up to 120 fps, presenting different temporal resolutions (TR). The Structural Similarity Index Measure (SSIM) is used as an objective video quality metric. After data augmentation, we propose NN models that estimate optimal TR and bitrate values as switching points to a higher SR. These results can be further used as input parameters for the bitrate ladder construction for video sequences of a certain complexity. [ABSTRACT FROM AUTHOR]

High Efficiency Video Coding (HEVC) offers superior compression rates, but its adoption introduces increased coding complexity due to its reliance on a recursive quad-tree for partitioning frames into varying block sizes. This quad-tree process is a central feature in upcoming video coding standards. Our paper presents a novel framework, SVG-CNN, which integrates three shallow Convolutional Neural Networks (CNNs) inspired by VGGNet. Each CNN is specifically designed for individual quad-tree levels to predict the Code Unit (CU) partition in HEVC, leading to reduced intra-frame coding time. SVG-CNN has an inherent capability for early terminations, leveraging sequential CNN feeding based on quad-tree level probabilities. This provides a mechanism to halt processes when further refinement is seemed unlikely. Enhancing the model's efficacy, we have crafted three specialized datasets, each focusing on distinct quad-tree levels and quantization parameter (QP) contexts. This allows each CNN within our framework to undergo targeted training, establishing a cutting-edge training methodology. Our study shows that performance, in terms of accuracy and F1 metrics, is highly dependent on QP settings, with lower QPs yielding better results, and higher QPs diminishing performance due to potential loss of critical features. To enhance our model, we tackled hyperparameter selection and CU split threshold determination for HEVC prediction. We utilized Grid Search Cross-Validation for the former and assessed multiple thresholds across selected videos for the latter. The model has a moderate complexity with over 328,000 parameters across 18 layers, which ensures memory efficiency. It boasts a swift prediction time of 0.05 ms and reduces HEVC encoding time by 61.64%, while slightly improving the bitrate-distortion performance by -0.24% BDBR, indicating better compression without notable PSNR loss. Significantly, our approach outperforms other CNN-based quad-tree partitioning methods that reduce HEVC coding complexity but sacrifice compression performance. [ABSTRACT FROM AUTHOR]

We study the feasibility of video transmission over optical fibre to optimise bandwidth with the implementation of HEVC codec features. We use simulation (Matlab and the OptiSystem software). Different values of the CRF are used to evaluate its impact on the visual quality and the size of the encoded file, as well as its influence on the video transmission performance. The simulation results show that by adjusting the CRF, the encoders can optimise the compression of the video data to reduce the file size while preserving an acceptable visual quality. This makes it possible to adapt the transmission to the bandwidth constraints of the optical fibre, by choosing higher CRF values to reduce the size of the files and save bandwidth, or lower values to maintain optimal quality when the bandwidth is sufficient. In addition, from the optical fibre point of view, the dispersion weakens and the eye opens, and it is observed that the length of the fibres is inversely proportional to the signal transmission quality. Thus, the judicious use of different CRF values can contribute to efficient and high-quality video transmission via optical fibre.

As most of the videos are destined for human perception, many techniques have been designed to improve video coding based on how the human visual system perceives video quality. In this paper, we propose the use of two perceptual coding techniques, namely contrast masking and texture masking, jointly operating under the High Efficiency Video Coding (HEVC) standard. These techniques aim to improve the subjective quality of the reconstructed video at the same bit rate. For contrast masking, we propose the use of a dedicated weighting matrix for each block size (from 4 × 4 up to 32 × 32 ), unlike the HEVC standard, which only defines an 8 × 8 weighting matrix which it is upscaled to build the 16 × 16 and 32 × 32 weighting matrices (a 4 × 4 weighting matrix is not supported). Our approach achieves average Bjøntegaard Delta-Rate (BD-rate) gains of between 2.5 % and 4.48 % , depending on the perceptual metric and coding mode used. On the other hand, we propose a novel texture masking scheme based on the classification of each coding unit to provide an over-quantization depending on the coding unit texture level. Thus, for each coding unit, its mean directional variance features are computed to feed a support vector machine model that properly predicts the texture type (plane, edge, or texture). According to this classification, the block's energy, the type of coding unit, and its size, an over-quantization value is computed as a QP offset (DQP) to be applied to this coding unit. By applying both techniques in the HEVC reference software, an overall average of 5.79 % BD-rate gain is achieved proving their complementarity. [ABSTRACT FROM AUTHOR]

Existing cost allocation methods for adaptive video steganography mainly focus on specific transform coefficients, resulting in lower capacity. Moreover, distortion drift is a significant challenge for steganography in HEVC videos. Therefore, this paper proposed a cost allocation method that combined the intra-frame and inter-frame processes of HEVC video coding to achieve high-capacity, low-distortion transmission in high-performance adaptive video steganography. Firstly, the method investigated the discrete sine transform features in HEVC video coding, analyzing the error propagation patterns of these coefficients under disturbance. During the embedding processed, it conducted a detailed analysis on the intra-block distortion, interblock distortion, and inter-frame distortion caused by modifying transform coefficients. The algorithm also took into account the differentiation in inter-block distortion resulting from steganography in different blocks, leading to the classification of blocks. This paper maximized the utilization of all non-zero transform coefficients, allocating distinct distortion costs for various carrier coefficients. The covert information was then embedded into frames that minimally impacted video quality. Experimental results indicate that, compared to existing HEVC video coefficient domain steganography methods, the proposed algorithm demonstrates advantages in terms of video bitrate, video quality, and embedding capacity. [ABSTRACT FROM AUTHOR]

Digital video coding aims to reduce the bitrate and keep the integrity of visual presentation. High-Efficiency Video Coding (HEVC) can effectively compress video content to be suitable for delivery over various networks and platforms. Finding the optimal coding configuration is challenging as the compression performance highly depends on the complexity of the encoded video sequence. This paper evaluates the effects of motion content on coding performance and suggests an adaptive encoding scheme based on the motion content of encoded video. To evaluate the effects of motion content on the compression performance of HEVC, we tested three coding configurations with different Group of Pictures (GOP) structures and intra refresh mechanisms. Namely, open GOP IPPP, open GOP Periodic-I, and closed GOP periodic-IDR coding structures were tested using several test sequences with a range of resolutions and motion activity. All sequences were first tested to check their motion activity. The rate–distortion curves were produced for all the test sequences and coding configurations. Our results show that the performance of IPPP coding configuration is significantly better (up to 4 dB) than periodic-I and periodic-IDR configurations for sequences with low motion activity. For test sequences with intermediate motion activity, IPPP configuration can still achieve a reasonable quality improvement over periodic-I and periodic-IDR configurations. However, for test sequences with high motion activity, IPPP configuration has a very small performance advantage over periodic-I and periodic-IDR configurations. Our results indicate the importance of selecting the appropriate coding structure according to the motion activity of the video being encoded. [ABSTRACT FROM AUTHOR]

A codec is the most important element of any streaming application like Netflix or YouTube, but its complexity is increasing with its advancement. The codec, High Efficiency Video Coding (HEVC), selects the intra mode out of a total of 35 intra modes using the brute-force method. Therefore, the Spiral Optimization Algorithm (SOA) is utilized in this article to overcome the aforementioned limitation of HEVC. Firstly, the elements of SOA are efficiently mapped to the elements of HEVC. Secondly, the current best intra mode becomes the center of the spiral of SOA. Thirdly, the current best intra mode is considered optimal if the spiral completes three spiral-loops and the current intra mode still remains the best. Otherwise, the new best intra modes becomes the center of the spiral. Results show that SOA based fast intra-mode selection reduces the time complexity of HEVC by 34.09% while BD-BR (Bjontegaard Delta - Bit Rate) overhead is only 1.56%. [ABSTRACT FROM AUTHOR]

Motion estimation involves determining the direction of an object's movement, which is crucial in video coding during the transmission process. The motion vector can indicate the shift point between the currently processed frame and the frame used as a processing reference. The sum-of-absolute-difference (SAD) block-matching algorithm relies heavily on estimating object movement. In this research, we propose the integration of three-step search (TSS) and full search (FS) methods in multi-scalable video transmission using high-efficiency video coding (HEVC), applying three scalabilities--spatial, signal-to-noise ratio (SNR), and temporal. This integration aims to increase efficiency and improve quality by employing a block-matching algorithm. With this design, we evaluate the performance of the TSS and FS methods in multi-scalable video coding, obtaining the video frame quality with peak SNR (PSNR) and bit rate efficiency. From the results of experiments using video tests in Standard Definition (SD), Common Intermediate Format (CIF), and High- Definition (HD) formats, the FS algorithm has a total Bjontegaard delta (BD)-PSNR value of 0 dB and an efficiency of 62.4%, while the TSS achieves a total BD-PSNR value of 0.8 dB and an efficiency of 23.6%. Meanwhile, the optimal PSNR and bit rate for the multi-scalability average were found with the FS algorithm enhancement layer and the TSS algorithm enhancement layer. [ABSTRACT FROM AUTHOR]

In recent years, the explosive growth of video data has posed a challenge to the performance of traditional video coding frameworks. The industry is thus faced with the pressing issue of how to transmit high-quality videos under limited bandwidth conditions. To address this challenge, a super-resolution reconstruction video coding scheme has been proposed, which combines traditional coding frameworks with deep learning-based super-resolution technology. By reducing the data coding volume through pre-coding downsampling and reconstructing videos through post-decoding upsampling, this scheme shows great potential in solving the aforementioned problems. However, previous super-resolution reconstruction video coding schemes have failed to effectively utilize the inter-frame correlation of video sequences, which limits the coding efficiency of the scheme. To overcome this limitation, this paper proposes an upsampling reconstruction network based on inter-frame information exploration. Experimental results show that, compared with the HEVC standard, the proposed scheme achieves a reduction of 11.05%, 11.3%, and 8.84% in the BD-BR index under the All-Intra, Low Delay P, and Random Access coding configurations, respectively, demonstrating higher coding efficiency than the previous human-designed scheme. [ABSTRACT FROM AUTHOR]

With the surge in demand for high-quality video content over various platforms, efficient video compression techniques have become indispensable. High-Efficiency Video Coding (HEVC) has been a cornerstone, yet further enhancements are essential for optimal compression. Despite HEVC’s advancements, achieving optimal compression while maintaining video quality remains challenging. Additionally, existing methods often overlook the computational complexity, hindering real-time applications. We propose a novel approach integrating HEVC with Non-Linear Convolutional MobileNet (NLCM) for enhanced compression efficiency. Our method employs a rate-distortion optimization framework, leveraging the capabilities of both HEVC and NLCM to achieve superior compression performance. NLCM provides adaptive filtering, enhancing spatial and temporal correlations, while HEVC ensures high compression efficiency. Through experimentation on standard video datasets, our method demonstrates significant improvements over existing techniques. Compared to HEVC alone, our approach achieves up to 30% reduction in bitrate at equivalent perceptual quality levels. Moreover, computational complexity is reduced by 15%, enabling realtime applications without compromising performance. The proposed method exhibits competitive results across various resolutions and frame rates, making it versatile for diverse video compression scenarios. [ABSTRACT FROM AUTHOR]

With the High Efficiency Video Coding (HEVC) becoming a popular video coding standard in the world, combing the HEVC standard with data hiding methods is a complex task. In this paper, we propose a video steganography method based on intra luma and chroma prediction modes for HEVC. Concerning the characteristics of the quadtree partition and the correlation between the intra luma and chroma prediction modes, we modify the intra chroma prediction modes to embed secret messages. In order to improve the security and embedding efficiency of the method, we use the Logistic chaotic sequence to preprocess the secret messages, and then use matrix coding to select appropriate parameters to embed the messages. The experimental results demonstrate that the proposed method maintains a significant embedding capacity across various encoding structures. Furthermore, it achieves an average ΔPSNR of 0.02 dB, indicating excellent concealment performance while ensuring minimal impact on image quality. [ABSTRACT FROM AUTHOR]

As a practical technique in mainstream video coding applications, rate control dominates important to ensure compression quality with limited bitrates constraints. However, most rate control methods mainly focus on objective quality while ignoring the perceptual quality improvement for human eyes. In this paper, we propose a two-stage rate control algorithm to optimize the perceptual quality at the frame encoding stage and the coding tree unit (CTU) encoding stage for high efficiency video coding (HEVC), respectively. Firstly, for the frame encoding stage, with inter-frame distortion dependency consideration, a frame-level rate control method is presented by adjusting the frame-level Lagrange multiplier adaptively with a preprocessing method. Secondly, for the CTU encoding stage, we propose a saliency-based CTU-level perceptual quality rate control algorithm, which employs CTU-level saliency weight to adjust the perceptual rate-distortion (R-D) model. We conduct the CTU-level rate control by an optimized Lagrange multiplier and quantization parameter (QP) to achieve perceptual quality optimization. Extensive experimental results reveal that, compared with state-of-the-art rate control methods on HEVC, our algorithm achieves significant perceptual coding performance with improved subjective visual quality. [ABSTRACT FROM AUTHOR]

The High-Efficiency Video Coding (HEVC/H.265) standard was created by a group called the Joint Collaborative Team on Video Coding (JCT-VC) and was released in 2013.In HEVC Rate-Distortion Optimization (RDO) is employed for partitioning the coding tree units into coding units recursively due to this the computational complexity increases. However, changing from macroblocks to coding tree units the compression efficiency increases and increases the encoding of the video sequence. This paper presents a Support Vector Machine (SVM)-based method for finding the fastest coding unit division in intra-prediction HEVC without compromising compression efficiency. All partitions of CTU are assessed using five characteristics: Root Mean Square Error (RMSE), Sub CU Complexity Difference (SCCD), Standard Deviation (SD), Directional Complexity (DC), and Quantization Parameter (QP) to optimize the intra-prediction of HEVC in all intra-configurations. The simulation results reveal combining directional complexity and standard deviation yields a more accurate classification. SVM was used to separate split-unsplit data, and the conventional rate-distortion optimisation technique was applied to separate samples that were difficult to separate. The results show that the encoding process has been completed 67.44% faster, albeit with a marginal increase in bit rate. [ABSTRACT FROM AUTHOR]

High-Efficiency Video Coding (HEVC) has become popular according to its excellent coding performance, in particular in the case of high-resolution video applications. However, the significant gain in performance is accompanied by a higher encoding complexity compared to the H.264/AVC standard. The motion estimation (ME) is the most time-consuming part that removes temporal redundancy. To reduce the motion estimation complexity, many fast algorithms have been developed in order to minimize search positions and speed up computation but they do not take into account how they can be effectively implemented by hardware. This paper presents a design for the fast ME algorithm with a variable block size of HEVC standard which is the "TZ search". The design is described in VHDL language and synthesized to Altera Stratix III FPGA. The hardware architecture throughput reaches a processing rate up to 78 million pixels per second at 100 MHz. For the validation proposed design, an IP core is presented using the embedded video system on a programmable chip (SoPC). Finally, compared to other designs existing in the literature, the proposed architecture shows more efficiency in terms of hardware cost and improved performance. This design can be used in ultra-high-definition real-time TV coding (UHD) applications. [ABSTRACT FROM AUTHOR]

Enhancing video quality is a vital area of research that has piqued the interest of academia and industry alike. The High-Efficiency Video Coding (HEVC) compressed videos suffer from significant quality loss, especially at low bit rates. As a result, the decoder must improve the image quality of HEVC videos. Deep Learning with Super Interpolation based Super Resolution (DLSI-SR) and Laplacian Filter (LF) method is proposed to eliminate artifacts. A Multi-Layered Deep Convolutional Neural Network (ML-DCNN) with LF has been introduced to reduce the artifacts quickly in compressed video and increase the subjective video quality. The quality of Low-Resolution (LR) frames can be effectively enhanced using Super Interpolation and Laplacian filter. The Edge orientation analysis is performed on various external training video frames during the training stage. The LR video frame is up-sampled and interpolated using the Super Interpolation algorithm during the upscaling phase. The interpolated frame is then submitted to edge detection using a Canny Edge detection algorithm for frame smoothing. Using local Laplacian filter, the reconstructed high quality video frame is sharpened with an edge preservation approach. The simulation results of the proposed DLSI-SR method shows higher performance in terms of PSNR and SSIM with less processing time in comparison with the existing approaches. The experimental results show that the proposed approach achieves a maximum PSNR of 42.5 dB (enhanced by 2.56 dB) and produced an SSIM improvement of 0.948 (enhanced by +0.022) as compared to the existing approaches. [ABSTRACT FROM AUTHOR]

In HTTP Adaptive Streaming (HAS), videos are encoded at multiple bitrates and spatial resolutions (i.e., representations) to adapt to the heterogeneity of network conditions, device attributes, and end-user preferences. Encoding the same video segment at multiple representations increases costs for content providers. State-of-the-art multi-encoding schemes improve the encoding process by utilizing encoder analysis information from already encoded representation(s) to reduce the encoding time of the remaining representations. These schemes typically use the highest bitrate representation as the reference to accelerate the encoding of the remaining representations. Nowadays, most streaming services utilize cloud-based encoding techniques, enabling a fully parallel encoding process to reduce the overall encoding time. The highest bitrate representation has a higher encoding time than the other representations. Thus, utilizing it as the reference encoding is unfavorable in a parallel encoding setup as the overall encoding time is bound by its encoding time. This article provides a comprehensive study of various multi-rate and multi-encoding schemes in both serial and parallel encoding scenarios. Furthermore, it introduces novel heuristics to limit the Rate Distortion Optimization (RDO) process across various representations. Based on these heuristics, three multi-encoding schemes are proposed, which rely on encoder analysis sharing across different representations: (i) optimized for the highest compression efficiency, (ii) optimized for the best compression efficiency-encoding time savings tradeoff, and (iii) optimized for the best encoding time savings. Experimental results demonstrate that the proposed multi-encoding schemes (i), (ii), and (iii) reduce the overall serial encoding time by 34.71%, 45.27%, and 68.76% with a 2.3%, 3.1%, and 4.5% bitrate increase to maintain the same VMAF, respectively, compared to stand-alone encodings. The overall parallel encoding time is reduced by 22.03%, 20.72%, and 76.82% compared to stand-alone encodings for schemes (i), (ii), and (iii), respectively. [ABSTRACT FROM AUTHOR]

High-efficiency video coding (HEVC) outperforms H.264 in coding efficiency. However, the rate–distortion optimization (RDO) process in coding tree unit (CTU) partitioning requires an exhaustive exploration of all possible quad-tree partitions, resulting in high encoding complexity. To simplify this process, this paper proposed a convolution neural network (CNN) based optimization algorithm combined with a hybrid attention mechanism module. Firstly, we designed a CNN compatible with the current coding unit (CU) size to accurately predict the CU partitions. In addition, we also designed a convolution block to enhance the information interaction between CU blocks. Then, we introduced the convolution block attention module (CBAM) into CNN, called CNN-CBAM. This module concentrates on important regions in the image and attends to the target object correctly. Finally, we integrated the CNN-CBAM into the HEVC coding framework for CU partition prediction in advance. The proposed network was trained, validated, and tested using a large scale dataset covering various scenes and objects, which provides extensive samples for intra-frame CU partition prediction in HEVC. The experimental findings demonstrate that our scheme can reduce the coding time by up to 64.05% on average compared to a traditional HM16.5 encoder, with only 0.09 dB degradation in BD-PSNR and a 1.94% increase in BD-BR. [ABSTRACT FROM AUTHOR]

The advancements in the domain of video coding technologies are tremendously fluctuating in recent years. As the public got acquainted with the creation and availability of videos through internet boom and video acquisition devices including mobile phones, camera etc., the necessity of video compression become crucial. The resolution variance (4 K, 2 K etc.), framerate, display is some of the features that glorifies the importance of compression. Improving compression ratio with better efficiency and quality was the focus and it has many stumbling blocks to achieve it. The era of artificial intelligence, neural network, and especially deep learning provided light in the path of video processing area, particularly in compression. The paper mainly focuses on a precise, organized, meticulous review of the impact of deep learning on video compression. The content adaptivity quality of deep learning marks its importance in video compression to traditional signal processing. The development of intelligent and self-trained steps in video compression with deep learning is reviewed in detail. The relevant and noteworthy work that arose in each step of compression is inculcated in this paper. A detailed survey in the development of intra- prediction, inter-prediction, in-loop filtering, quantization, and entropy coding in hand with deep learning techniques are pointed along with envisages ideas in each field. The future scope of enhancement in various stages of compression and relevant research scope to explore with Deep Learning is emphasized. [ABSTRACT FROM AUTHOR]

It is well-known that each new video coding standard significantly increases in computational complexity with respect to previous standards, and this is particularly true for the HEVC and VVC video coding standards. The development of techniques for reducing the required complexity without affecting the rate/distortion (R/D) performance is therefore always a topic of intense research interest. In this paper, we propose a combination of two powerful techniques, deep learning and parallel computing, to significantly reduce the complexity of the HEVC encoding engine. Our experimental results show that a combination of deep learning to reduce the CTU partitioning complexity with parallel strategies based on frame partitioning is able to achieve speedups of up to 26 × when 16 threads are used. The R/D penalty in terms of the BD-BR metric depends on the video content, the compression rate and the number of OpenMP threads, and was consistently between 0.35 and 10% for the video sequence test set used in our experiments [ABSTRACT FROM AUTHOR]

Nowadays, smartphones are becoming the predominant source of video content that is being widely shared through various platforms of social media. According to recent studies, 78% of iPhones and 57% of android smartphones support hardware-accelerated HEVC decoding. Growing usage of videos on social media applications poses the challenge of distinguishing between authentic and manipulated content. In this paper, we have proposed the development of HEVC based Tampered Video Dataset (HTVD) consisting of diverse scenarios of authentic and forged videos for more comprehensive testing capabilities. The dataset will provide the researchers with a benchmark with a varied range of realistic and smartly tampered videos for validation and comparison of their forensic investigation techniques. The HTVD dataset is developed from videos captured under scenarios of indoor, outdoor, and surveillance shots. It includes 60 original videos, 966 tampered videos, their corresponding ground truth information, and masks. All the original videos are captured with HEVC supported smartphones. Various types of inter-frame forgeries such as frame insertion, frame deletion and frame duplication, and object-based intra-frame forgeries such as cloning, splicing, and inpainting are incorporated to create a diversified database of forged videos. Further, the tampered videos are provided with variations based on the encoding parameters of the codec namely, GOP size, CRF and frame types resulting in a total of 8,694 forged videos. To perform video tampering, the forger needs to perform recompression. However, videos may also undergo recompression while transferring over the internet or exchanged through social media applications. Thus, recompression doesn't always mean that forgery has been performed. The proposed HTVD dataset provides a video dataset for experimenting with both circumstances. This video dataset is publicly available on https://drive.google.com/drive/folders/143NEyVjcHNVDDZVzIZm6nTEUk5-ezWb5?usp=sharing. [ABSTRACT FROM AUTHOR]

As the necessity of supporting ever-increasing demands in video resolution leads to new video coding standards, the challenge of harnessing their computational overhead becomes important. Such overhead stems not only from the increased image data due to higher resolutions but also from the coding techniques per se that are introduced by each standard to improve compression. All modern standards in the field of video coding offer high compression efficiency, but this is achieved by increasing the computational complexity of the encoding part. Ultra-High-Definition (UHD) videos, bring new encoding implementation schemes that are being recommended for CPU and GPU parallelization. Therefore, several works are published to achieve better performance and reduce encoding complexity. Following this idea, we proposed and evaluated a hybrid encoding scheme that utilizes the constant growth of the CPU power with the massive GPU popularity in parallel. Taking advantage of the encoding schemes from the leading video coding standards, such as High-Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC), which support parallel processing thru Wavefront or Tiling, in our work, we combined both of them at the same time as a whole, and in addition, we introduced a GPU-friendly fast search algorithm that is highly parallel and alternative to the default non-parallel TZ-Search. Through an experimental evaluation with common test sequences, the proposed GPU Fast Motion Estimation with our previous Wavefront per Tile Parallelism (WTP) was shown to provide valid trade-off between speedup and video coding efficiency, effectively combining the best of two worlds, i.e., WTP using CPUs and parallel Motion Estimation with GPUs. [ABSTRACT FROM AUTHOR]

Among a series of video coding standards jointly developed by ITU-T, VCEG, and MPEG, high-efficiency video coding (HEVC) is one of the most widely used video coding standards today. Therefore, it is still necessary to further reduce the coding complexity of HEVC. In the HEVC standard, a flexible partitioning procedure entitled "quad-tree partition" is proposed to significantly improve the coding efficiency, which, however, leads to high coding complexity. To reduce the coding complexity of the intra-frame prediction, this paper proposes a scheme based on a densely connected convolution neural network (D-CNN) to predict the partition of coding units (CUs). Firstly, a densely connected block was designed to improve the efficiency of the CU partition by fully extracting the pixel features of CTU. Then, efficient channel attention (ECA) and adaptive convolution kernel size were applied to a fast CU partition for the first time to capture the information of the D-CNN convolution channels. Finally, a threshold optimization strategy was formulated to select the best threshold for each depth to further balance the computation complexity of video coding and the performance of RD. The experimental results show that the proposed method reduces the encoding time of HEVC by 60.14%, with a negligible reduction in RD performance, which is better than the existing fast partitioning methods. [ABSTRACT FROM AUTHOR]

High efficiency video coding (HEVC) has dramatically enhanced coding efficiency compared to the previous video coding standard, H.264/AVC. However, the existing rate control updates its parameters according to a fixed initialization, which can cause errors in the prediction of bit allocation to each coding tree unit (CTU) in frames. This paper proposes a learning-based mapping method between rate control parameters and video contents to achieve an accurate target bit rate and good video quality. The proposed framework contains two main structural codings, including spatial and temporal coding. We initiate an effective learning-based particle swarm optimization for spatial and temporal coding to determine the optimal parameters at the CTU level. For temporal coding at the picture level, we introduce semantic residual information into the parameter updating process to regulate the bit correctly on the actual picture. Experimental results indicate that the proposed algorithm is effective for HEVC and outperforms the state-of-the-art rate control in the HEVC reference software (HM-16.10) by 0.19 dB on average and up to 0.41 dB for low-delay P coding structure. [ABSTRACT FROM AUTHOR]

Researchers have, in recent times, achieved excellent compression efficiency by implementing a more complicated compression algorithm due to the rapid development of video compression. As a result, the next model of video compression, High-Efficiency Video Coding (HEVC), provides high-quality video output while requiring less bandwidth. However, implementing the intra-prediction technique in HEVC requires significant processing complexity. This research provides a completely pipelined hardware architecture solution capable of real-time compression to minimize computing complexity. All prediction unit sizes of 4 × 4 , 8 × 8 , 16 × 16 , and 32 × 32 , and all planar, angular, and DC modes are supported by the proposed solution. The synthesis results mapped to Xilinx Virtex 7 reveal that our solution can do real-time output with 210 frames per second (FPS) at 1920 × 1080 resolution, called Full High Definition (FHD), or 52 FPS at 3840 × 2160 resolution, called 4K, while operating at 232 Mhz maximum frequency. [ABSTRACT FROM AUTHOR]

Visual sensor networks (VSNs) have numerous applications in fields such as wildlife observation, object recognition, and smart homes. However, visual sensors generate vastly more data than scalar sensors. Storing and transmitting these data is challenging. High-efficiency video coding (HEVC/H.265) is a widely used video compression standard. Compare to H.264/AVC, HEVC reduces approximately 50% of the bit rate at the same video quality, which can compress the visual data with a high compression ratio but results in high computational complexity. In this study, we propose a hardware-friendly and high-efficiency H.265/HEVC accelerating algorithm to overcome this complexity for visual sensor networks. The proposed method leverages texture direction and complexity to skip redundant processing in CU partition and accelerate intra prediction for intra-frame encoding. Experimental results revealed that the proposed method could reduce encoding time by 45.33% and increase the Bjontegaard delta bit rate (BDBR) by only 1.07% as compared to HM16.22 under all-intra configuration. Moreover, the proposed method reduced the encoding time for six visual sensor video sequences by 53.72%. These results confirm that the proposed method achieves high efficiency and a favorable balance between the BDBR and encoding time reduction. [ABSTRACT FROM AUTHOR]

With video streaming making up 80% of the global internet bandwidth, the need to deliver high-quality video at low bitrate, combined with the high complexity of modern codecs, has led to the idea of a per-clip optimisation approach in transcoding. In this paper, we revisit the Lagrangian multiplier parameter, which is at the core of rate-distortion optimisation. Currently, video encoders use prediction models to set this parameter but these models are agnostic to the video at hand. We explore the gains that could be achieved using a per-clip direct-search optimisation of the Lagrangian multiplier parameter. We evaluate this optimisation framework on a much larger corpus of videos than that has been attempted by previous research. Our results show that per-clip optimisation of the Lagrangian multiplier leads to BD-Rate average improvements of 1.87% for x265 across a 10 k clip corpus of modern videos, and up to 25% in a single clip. Average improvements of 0.69% are reported for libaom-av1 on a subset of 100 clips. However, we show that a per-clip, per-frame-type optimisation of λ for libaom-av1 can increase these average gains to 2.5% and up to 14.9% on a single clip. Our optimisation scheme requires about 50–250 additional encodes per-clip but we show that significant speed-up can be made using proxy videos in the optimisation. These computational gains (of up to ×200) incur a slight loss to BD-Rate improvement because the optimisation is conducted at lower resolutions. Overall, this paper highlights the value of re-examining the estimation of the Lagrangian multiplier in modern codecs as there are significant gains still available without changing the tools used in the standards.

The standard HEVC codec and its extension for coding multiview videos, known as MV-HEVC, have proven to deliver improved visual quality compared to its predecessor, H.264/MPEG-4 AVC's multiview extension, H.264-MVC, for the same frame resolution with up to 50% bitrate savings. MV-HEVC's framework is similar to that of H.264-MVC, which uses a multi-layer coding approach. Hence, MV-HEVC would require all frames from other reference layers decoded prior to decoding a new layer. Thus, the multi-layer coding architecture would be a bottleneck when it comes to quicker frame streaming across different views. In this paper, an HEVC-based Frame Interleaved Stereo/Multiview Video Codec (HEVC-FISMVC) that uses a single layer encoding approach to encode stereo and multiview video sequences is presented. The frames of stereo or multiview video sequences are interleaved in such a way that encoding the resulting monoscopic video stream would maximize the exploitation of temporal, inter-view, and cross-view correlations and thus improving the overall coding efficiency. The coding performance of the proposed HEVC-FISMVC codec is assessed and compared with that of the standard MV-HEVC's performance for three standard multi-view video sequences, namely: "Poznan_Street", "Kendo" and "Newspaper1". Experimental results show that the proposed codec provides more substantial coding gains than the anchor MV-HEVC for coding both stereo and multi-view video sequences. [ABSTRACT FROM AUTHOR]

High-efficiency video coding (HEVC) effectively reduces the amount of video data while unavoidably introducing compression noise. The in-loop filter can enhance the reconstructed frames at the encoder to prevent compression noise from transmitting to the subsequent frames. The existing incorporated in-loop filters for HEVC do not fully combine the advantages of CNN and video coding priors. Moreover, obtaining the accuracy noise level is essential for adaptively in-loop filtering. To further enhance the compressed video quality in the encoder, we propose a nonlocal in-loop filter for HEVC using a CNN-based compression noise estimation network(CNEN). In the noise estimation part, we adopt a classification network to estimate the noise of compressed HEVC videos according to video content characteristics. In the denoising part, we propose a spatial-temporal nonlocal low-rank(STNLLR) prior by simultaneously exploiting the nonlocal self-similarity of video in spatial and temporal domains. We also propose an adaptive narrow quantization constraint (ANQC) prior by limiting the reconstructed pixel values adaptively according to the quantization parameters(QPs). The experimental results show that CNEN outperforms the existing compression noise estimation methods. Furthermore, our in-loop filter can improve the quality of the HEVC reconstructed frames under AI, LDP, and RA configurations, achieving an average reduction of 4.17%, 10.46%, and 6.10% in the Bjøntegaard Delta Bit Rate (BDBR), respectively. [ABSTRACT FROM AUTHOR]

We propose a new Generative Adversarial Network for Compressed Video frame quality Enhancement (CVEGAN). The CVEGAN generator benefits from the use of a novel Mul2Res block (with multiple levels of residual learning branches), an enhanced residual non-local block (ERNB) and an enhanced convolutional block attention module (ECBAM). The ERNB has also been employed in the discriminator to improve the representational capability. The training strategy has also been re-designed specifically for video compression applications, to employ a relativistic sphere GAN (ReSphereGAN) training methodology together with new perceptual loss functions. The proposed network has been fully evaluated in the context of two typical video compression enhancement tools: post-processing (PP) and spatial resolution adaptation (SRA). CVEGAN has been fully integrated into the MPEG HEVC and VVC video coding test models (HM 16.20 and VTM 7.0) and experimental results demonstrate significant coding gains (up to 28% for PP and 38% for SRA compared to the anchor) over existing state-of-the-art architectures for both coding tools across multiple datasets based on the HM 16.20. The respective gains for VTM 7.0 are up to 8.0% for PP and up to 20.3% for SRA. • Presenting a novel block structure, Mul2Res which is the first use of a nested residual learning structure with various kernel sizes. • Employing enhanced residual non-local blocks and enhanced convolutional block attention modules to improve the representational capability of the network. • Designing a new GAN training methodology, Relativistic SphereGAN to achieve better training performance. • Proposing a novel perceptual loss function to further optimise video quality during training. [ABSTRACT FROM AUTHOR]

With the development of video technology, a large amount of video data generated from video conferences, sports events, live broadcasts and network classes flows into our daily lives. However, ultra-high-definition video transmission is still a challenge due to the limited network bandwidth and instability, which further affects the quality of video service closely linked with consumer electronic video display. To address this challenge, we propose a deep-learned perceptual quality control approach, which can significantly improve the video quality and visual experience at the same bandwidth. The proposed scheme mainly involves saliency region extraction, perceptual-based bits allocation, and video enhancement. Firstly, we exploit a multi-scale deep convolutional network module to predict the static saliency map that semantically highlights the salient regions. Secondly, we develop a recurrent neural network model to extract the dynamic saliency regions. Finally, a three-level rate allocation scheme is developed based on the resulted saliency guidance, which is more reasonable by taking into account the visual characteristics of human eyes. Experimental results on a large dataset show that our method achieves an average gain of 1.5dB on the salient regions without introducing an extra bandwidth burden, which significantly improves the visual experience and paves the way to intelligent video communication. [ABSTRACT FROM AUTHOR]

This article presents a method for transparent watermarking of high-capacity watermarked video under H.265/HEVC (High-Efficiency Video Coding) compression conditions while maintaining high-quality encoded image. The aim of this paper is to present a method for watermark embedding using neural networks under conditions of subjecting video to lossy compression of the HEVC codec using the YUV420p color model chrominance channel for watermarking. This paper presents a method for training a deep neural network to embed a watermark when a compression channel is present. The discussed method is characterized by high accuracy of the video with an embedded watermark compared to the original. The PSNR (peak signal-to-noise ratio) values obtained are over 44 dB. The watermark capacity is 96 bits for an image with a resolution of 128 × 128. The method enables the complete recovery of a watermark from a single video frame compressed by the HEVC codec within the range of compression values defined by the CRF (constant rate factor) up to 22. [ABSTRACT FROM AUTHOR]

Video standards are crucial for exchanging video content, enabling a myriad of services and supporting a wide variety of devices ranging from personal devices to clouds and IoT. One of the core requirements in video standards is the rate control that regulates the bit allocation and picture quality. This paper presents an overview of rate control techniques in the HEVC video coding standard. While providing an insight into the rate control mechanism specific to HEVC, it describes the basic operating principle of rate control algorithms, including their essential parameter, outputs, and performance measures. We review rate control in past coding standards and bring out the basic features of HEVC that drive the need for new rate control algorithms. Alongside, we delineate the Rate-Distortion model-based taxonomy of various algorithms, including their classification criteria. The paper gives out another classification of the rate control algorithms based on their basic principle and mechanisms. The article also explains the scalable extension of HEVC, namely SHVC, while highlighting some of the possible SHVC rate control design challenges. Finally, we present some of the unresolved research issues in HEVC rate control and outline possible future research directions. [ABSTRACT FROM AUTHOR]

Ultra High-Definition (UHD) and Virtual Reality (VR) video streaming over 5G networks are emerging, in which High-Efficiency Video Coding (HEVC) is used as source coding to compress videos more efficiently and polar code is used as channel coding to transmit bitstream reliably over an error-prone channel. In this article, a novel Joint Source-Channel Decoding (JSCD) of polar codes for HEVC-based video streaming is presented to improve the streaming reliability and visual quality. Firstly, a Kernel Density Estimation (KDE) fitting approach is proposed to estimate the positions of error channel decoded bits. Secondly, a modified polar decoder called R-SCFlip is designed to improve the channel decoding accuracy. Finally, to combine the KDE estimator and the R-SCFlip decoder together, the JSCD scheme is implemented in an iterative process. Extensive experimental results reveal that, compared to the conventional methods without JSCD, the error data-frame correction ratios are increased. Averagely, 1.07% and 1.11% Frame Error Ratio (FER) improvements have been achieved for Additive White Gaussian Noise (AWGN) and Rayleigh fading channels, respectively. Meanwhile, the qualities of the recovered videos are significantly improved. For the 2D videos, the average Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM) gains reach 14% and 34%, respectively. For the 360° videos, the average improvements in terms of Weighted-to-Spherically-uniform PSNR (WS-PSNR) and Voronoi-based Video Multimethod Assessment Fusion (VI-VMAF) reach 21% and 7%, respectively. [ABSTRACT FROM AUTHOR]

High Efficiency Video Coding (HEVC) is considered the last standard for video compression with about 50% additional compression compared to the previous standard i.e. H264/AVC, while maintaining image quality. The significant increase in performance of this standard has been achieved with high computational complexity. In this standard, the intra prediction unit is one of the parts that although it greatly improves performance, it significantly increases its computational complexity due to the increase in the number of prediction modes. In this paper, a method has been proposed to reduce the number of intra prediction modes in HEVC by which the computational complexity of compression at this stage can be reduced as much as possible. The proposed method determines the predominant mode of 4 × 4 blocks, and the details in each larger block, and accordingly, by applying the appropriate filters and selecting the most likely mode, the number of candidate modes to select the best mode is reduced. The simulation results showed that on average the proposed method can reduce the compression time by 45% while increasing the bit rate by 0.69%. [ABSTRACT FROM AUTHOR]

High Efficiency Video Coding (HEVC) achieves a significant improvement in compression efficiency at the cost of extremely high computational complexity. Therefore, large-scale and wide deployment applications, especially mobile real-time video applications under low-latency and power-constrained conditions, are more challenging. In order to solve the above problems, a fast decision method for intra-coding unit size based on a new fuzzy support vector machine classifier is proposed in this paper. The relationship between the depth levels of coding units is accurately expressed by defining the cost evaluation criteria of texture and non-texture rate-distortion cost. The fuzzy support vector machine is improved by using the information entropy measure to solve the negative impact of data noise and the outliers problem. The proposed method includes three stages: the optimal coded depth level "0" early decision, coding unit depth early skip, and optimal coding unit early terminate. In order to further improve the rate-distortion complexity optimization performance, more feature vectors are introduced, including features such as space complexity, the relationship between coding unit depths, and rate-distortion cost. The experimental results showed that, compared with the HEVC reference test model HM16.5, the proposed algorithm can reduce the encoding time of various test video sequences by more than 53.24% on average, while the Bjontegaard Delta Bit Rate (BDBR) only increases by 0.82%. In addition, the proposed algorithm is better than the existing algorithms in terms of comprehensively reducing the computational complexity and maintaining the rate-distortion performance. [ABSTRACT FROM AUTHOR]

From the beginnings of ITU-T H.261 to H.265 (HEVC), each new video coding standard has aimed at halving the bitrate at the same perceptual quality by redundancy and irrelevancy reduction. Each improvement has been explained by comparably small changes in the video coding toolset. This contribution aims at starting the Quality of Experience (QoE) analysis of the accumulated improvements over the last thirty years. Based on an overview of the changes in the coding tools, we analyze the changes in the quantized residual information. Visual comparison and statistical measures are performed and some interpretations are provided towards explaining how irrelevancy reduction may have led to such a huge reduction in bitrate. The interpretation of the results in terms of QoE paves the way towards an understanding of the coding tools in terms of visual quality. It may help in understanding how the irrelevancy reduction has been improved over the decades. Understanding how the differences of the residuals relate to known or yet unknown properties of the human visual system, may enable a closer collaboration between perception research and video compression research. [ABSTRACT FROM AUTHOR]

Video is extensively used in everyday applications and services by heterogeneous users. High-definition video information corresponds to high-data-rate representation. Video coding scalability has an effect on bitrate requirement and peak signal-to-noise ratio (PSNR). However, the spatial information (SI) and temporal information (TI), that are inherent properties of the video, need to be maintained for the full-high-definition videos while performing scalably encoding using high-efficiency video coding (HEVC) standard. In this paper, we have established an empirical relationship of SI, TI, PSNR, and bitrate with quantization parameter (QP). Based on this study, parametric models have been derived that achieve correlation accuracy of at least 99.18 % . An optimization algorithm has been developed to find the best QP to achieve the highest PSNR and the least bitrate while maintaining SI and TI during the scalable transcoding of the video. Furthermore, the algorithmic architecture calculates the optimized QP that meets the heterogeneous user-end specification of minimum PSNR and maximum bitrate. The proposed architecture consumes low power and energy of 147 microwatts and 0.186 picojoules, respectively, and therefore, it can be integrated in an HEVC encoder for portable HEVC-compliant consumer electronic products. [ABSTRACT FROM AUTHOR]

This paper presents a method of high-capacity and transparent watermarking based on the usage of deep neural networks with the adjustable subsquares properties algorithm to encode the data of a watermark in high-quality video using the H.265/HEVC (High-Efficiency Video Coding) codec. The aim of the article is to present a method of embedding a watermark in a video with HEVC codec compression by making changes in a video in a way that is not noticeable to the naked eye. The method presented here is characterised by focusing on ensuring the accuracy of the original image in relation to the watermarked image, providing the transparency of the embedded watermark, while ensuring its survival after compression by the HEVC codec. The article includes a presentation of the practical results of watermark embedding with a built-in variation mechanism of its capacity and resistance, thanks to the adjustable subsquares properties algorithm. The obtained PSNR (peak signal-to-noise ratio) results are at the level of 40 dB or better. There is the possibility of the complete recovery of a watermark from a single frame compressed in the CRF (constant rate factor) range of up to 16, resulting in a BER (bit error rate) equal to 0 for the received watermark. [ABSTRACT FROM AUTHOR]

IP traffic has increased significantly in recent years, and it is expected that this progress will continue. Recent studies report that the viewing of online video content accounts for a share of 1% of the global greenhouse gas emissions. To reduce the data traffic of video streaming, the new standard Versatile Video Coding (VVC) has been finalized in 2020. In this paper, the energy efficiency of two different VVC decoders is analyzed in detail. Furthermore, we propose a design space exploration that uses an algorithm based on a greedy strategy to derive coding tool profiles that optimize the energy demand of the decoder. We show that the algorithm derives optimal coding tool profiles for a subset of coding tools. Additionally, we propose profiles that reduce the energy demand of VVC decoders and provide energy savings of more than 50% for sequences with 4K resolution. Thereby, we will also show that the proposed profiles can have a lower decoding energy demand than comparable HEVC-encoded bit streams while also having a significantly lower bit rate. [ABSTRACT FROM AUTHOR]

Spherical video coding is critical to the success of many virtual reality and related applications. This paper focuses on an important class of spherical videos whose dynamics involve camera motion. A common approach to spherical video coding is to project from the sphere onto a plane (or planes), where a standard video coder is applied. The projection induces warping resulting in complex non-linear motion in the projected domain that severely comprises the performance of motion models in standard coders. To overcome this shortcoming, we propose a new motion model that captures the motion field on the sphere, and capitalizes on insights into the perceived motion on the sphere due to camera translation. Specifically, surrounding static points are perceived as moving along their respective geodesics, which all intersect at the points where the camera velocity vector intersects the sphere. We analyze the rate of translation along geodesics and its dependence on the elevation of a pixel on the sphere with respect to the camera velocity vector. The analysis leads to a motion vector modulation scheme that perfectly captures the perceived motion of each pixel. Complementary to the new motion model, we propose a search grid tailored to capture expected geodesic motion on the sphere for effective motion estimation. The proposed method yields significant bit-rate savings over employing standard HEVC after projection, which validates its efficacy. [ABSTRACT FROM AUTHOR]

• Lightfield representation, coding and streaming schemes using a stack of focal images. • Layer pattern optimization from focal stack for efficient streaming on tensor display. • Hybrid Tucker Tensor Sketch Scheme with Vector Quantization for low-rank modeling. • A single integrated, adaptable and flexible system to realize a range of bitrates. Computational multi-view displays involving light fields are a fast emerging choice for 3D presentation of real-world scenes. Tensor autostereoscopic glasses-free displays use just few light attenuating layers in front of a backlight to output high quality light field. We propose three novel schemes, Focal Stack - Hybrid Tucker-TensorSketch Vector Quantization (FS-HTTSVQ), Focal Stack - Tucker-TensorSketch (FS-TTS), and Focal Stack - Tucker Alternating Least-Squares (FS-TALS), for efficient representation, streaming and coding of light fields using a stack of differently focused images. Working with a focal stack instead of the entire light field majorly reduces the data acquisition cost as well as the computation and processing cost. Extensive experiments with real world light field focal stacks demonstrate that proposed novel one-pass Tucker decomposition using TensorSketch with hybrid vector quantization in FS-HTTSVQ, compactly represents the approximated focal stack in codebook form for better transmission and streaming. Encoding with High Efficiency Video Coding (HEVC) eliminates all intrinsic redundancies present in the approximated focal stack. Resultant low-rank approximated and coded focal stack is then employed to analytically optimize layer patterns for the tensor display. The complete end-to-end light field processing pipelines flexibly work for multiple bitrates and are adaptable for a variety of multi-view autostereoscopic platforms. Our schemes exhibit note-worthy performances on focal stacks compared to direct encoding of an entire light field using a standard codec like HEVC. [ABSTRACT FROM AUTHOR]

With the rapid growth of video data traffic on the Internet and the development of new types of video transmission systems, the need for ad hoc video encoders has also increased. One such case involves Unmanned Aerial Vehicles (UAVs), widely known as drones, which are used in drone races, search and rescue efforts, capturing panoramic views, and so on. In this paper, we provide an efficiency comparison of the two most popular video encoders—H.264 and H.265—in a drone piloting system using first-person view (FPV). In this system, a drone is used to capture video, which is then transmitted to FPV goggles in real time. We examine the compression efficiency of 4K drone footage by varying parameters such as Group of Pictures (GOP) size, Quantization Parameter (QP), and target bitrate. The quality of the compressed footage is determined using four objective video quality measures: PSNR, SSIM, VMAF, and BRISQUE. Apart from video quality, encoding time and encoding energy consumption are also compared. The research was performed using numerous nodes on a supercomputer. [ABSTRACT FROM AUTHOR]

HEVC has made significant progress in coding efficiency. Compared with the previous generation H.264/AVC video coding standard, HEVC can save 50% of code rate under the same visual quality. However, this drop in bit rate is accompanied by an increase in computational complexity, which poses a great challenge to the application and development of HEVC. After analyzing the great correlation between the partitioning mode of coding unit in the encoder and several parameters, the partitioning problem of the CU is modeled as a classification problem in data mining, and it is decided to use a decision tree to solve this classification problem. We implement the decision tree on the official software HM16.2 and test the algorithm on the test set. Experiments show that fast algorithm based on decision tree can improve the encoding speed effectively with little influence on the encoding performance. [ABSTRACT FROM AUTHOR]

In-loop filtering is used in video coding to process the reconstructed frame in order to remove blocking artifacts. With the development of convolutional neural networks (CNNs), CNNs have been explored for in-loop filtering considering it can be treated as an image de-noising task. However, in addition to being a distorted image, the reconstructed frame is also obtained by a fixed line of block based encoding operations in video coding. It carries coding-unit based coding distortion of some similar characteristics. Therefore, in this paper, we address the filtering problem from two aspects, (i) global appearance restoration for disrupted texture and (ii) local coding distortion restoration caused by fixed pipeline of coding. Accordingly, a three-stream global appearance and local coding distortion based fusion network is developed with a high-level global feature stream, a high-level local feature stream and a low-level local feature stream. Ablation study is conducted to validate the necessity of different features, demonstrating that the global features and local features can complement each other in filtering and achieve better performance when combined. To the best of our knowledge, we are the first one that clearly characterizes the video filtering process from the above global appearance and local coding distortion restoration aspects with experimental verification, providing a clear pathway to developing filter techniques. Experimental results demonstrate that the proposed method significantly outperforms the existing single-frame based methods and achieves 13.5%, 11.3%, 11.7% BD-Rate saving on average for AI, LDP and RA configurations, respectively, compared with the HEVC reference software. [ABSTRACT FROM AUTHOR]

Video coding algorithms attempt to minimize the significant commonality that exists within a video sequence. Each new video coding standard contains tools that can perform this task more efficiently compared to its predecessors. Modern video coding systems are block-based wherein commonality modeling is carried out only from the perspective of the block that need be coded next. In this work, we argue for a commonality modeling approach that can provide a seamless blending between global and local homogeneity information. For this purpose, at first the frame that need be coded, is recursively partitioned into rectangular regions based on the homogeneity information of the entire frame. After that each obtained rectangular region’s feature descriptor is taken to be the average value of all the pixels’ intensities encompassing the region. In this way, the proposed approach generates a coarse representation of the current frame by minimizing both global and local commonality. This coarse frame is computationally simple and has a compact representation. It attempts to preserve important structural properties of the current frame which can be viewed subjectively as well as from improved rate-distortion performance of a reference scalable HEVC coder that employs the coarse frame as a reference frame for encoding the current frame. [ABSTRACT FROM AUTHOR]

Segmentation of foreground object is pivotal in identifying the other finer details about objects in a scene. Our work serves to investigate ways to not only segment but encode the foreground objects in the HEVC encoder with minimal change in original HEVC encoder performance in terms of bitrate increase and encoding time. We achieve this by reutilizing the intermediate residual data at the encoder to segment the foreground activity in each frame and finally encoding the same in the final compressed bitstream using specific provisions of HEVC high level bitstream syntax. The method operates entirely in the HEVC encoder loop, taking residual data of the frame, it divides the entire frame into 8 × 8 target patches and applies the proposed algorithm Median of Discrete Variance (MoDV) to classify the target block of each frame of the video sequence as foreground or background. The foreground information in each frame is then encoded into the compressed bitstream by harnessing our proposed format of supplemental enhancement information (SEI) Network Abstraction Layer (NAL) units to tag the location of the foreground activity. Along with the segmentation accuracy, change in encoder performance is also measured to judge the various trade-offs. We conclude by testing its efficacy on a variety of videos with difference frame resolutions and background conditions. [ABSTRACT FROM AUTHOR]

The fast development of video technology and hardware has led to a great amount of video applications in the field of industry, such as the video conference, video surveillance and live video streaming. Most of the applications are facing with the problem of limited network bandwidth while try to keep high video resolution. Perceptual video compression addresses the problem by introducing saliency information to reduce the perceptual redundancy, which retains more information in salient region while compresses the non-salient region as much as possible. In this paper, in order to combine the multi-scale information in video saliency, an advanced video saliency detection model SALDPC is proposed which is built based on deformable pyramid convolution decoder and multi-scale temporal recurrence. In order to better guide video coding through saliency, based on the HEVC video coding standard, a saliency-aware rate-distortion optimization algorithm, SRDO, is proposed using block-based saliency information to change the rate-distortion balance and guide more reasonable bit allocation. Furthermore, a more flexible QP selection method, SAQP, is developed using CU's saliency information adaptively to change the QP of CU to ensure the high quality of the high saliency areas. The final results are available in the following three configurations: SRDO, SRDO + SQP, and SRDO + SAQP. Experimental results show that our method achieves a very high video quality improvement while significantly reducing the video encoding time. Compared to HEVC, the BD-EWPSNR of the SRDO method improves 0.703 dB, and the BD-Rate based on EWPSNR saves 20.822%; the BD-EWPSNR of the SRDO + SAQP is improved up to 1.217 dB, and the BD-Rate based on EWPSNR further saves up to 32.41%. At the same time, in terms of compression time, the proposed method saves up to 29.06% compared to HEVC. Experimental results show the superiority of the proposed method in comparison to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]