Search

Mainstream image/video coding standards, exemplified by the state-of-the-art H.266/VVC, AVS3, and AV1, follow the block-based hybrid coding framework. Due to the block-based framework, encoders designed for these standards are easily optimized for peak signal-to-noise ratio (PSNR) but have difficulties optimizing for the metrics more aligned to perceptual quality, e.g., multi-scale structural similarity (MS-SSIM), since these metrics cannot be accurately evaluated at the small block level. We address this problem by leveraging inspiration from the end-to-end image compression built on deep networks, which is easily optimized through network training for any metric as long as the metric is differentiable. We compared the trained models using the same network structure but different metrics and observed that the models allocate rates in different ratios. We then propose a distillation method to obtain the rate allocation rule from end-to-end image compression models with different metrics and to utilize such a rule in the block-based encoders. We implement the proposed method on the VVC reference software—VTM and the AVS3 reference software—HPM, focusing on intraframe coding. Experimental results show that the proposed method on top of VTM achieves more than 10% BD-rate reduction than the anchor when evaluated with MS-SSIM or LPIPS, which leads to concrete perceptual quality improvement. [ABSTRACT FROM AUTHOR]

8K (7689×4320) ultra-high definition (UHD) videos are growing popular with the improvement of human visual experience demand. Therefore, the compression of 8K UHD videos has become a top priority in the third-generation audio video coding standard (AVS3). However, as an important part of the coding standard promotion, the real-time hardware implementation for AVS3-based 8K UHD video intra coding is severely hindered, especially in the coding unit (CU) partition stage. To break through the limitation, this article proposes a divide-and-conquer-based rate-distortion-optimization-free (RDO-free) CU partitioning algorithm for efficient hardware implementation. Aimed at the complex CU partition in AVS3, we separately design a lightweight optimization for original partitioning rules to improve division efficiency and a decision tree-based RDO-free CU decision framework to eliminate the latency caused by the waiting for rate-distortion cost calculation in RDO strategy. Afterward, a divide-and-conquer-based hardware-friendly gradient difference calculating approach is devised to accelerate the learning feature extracting speed. To ensure that the proposed algorithm is sufficient to support the real-time CU partition for 8K videos, we also develop a hardware architecture based on FPGA. Experimental results illustrate that the software coding performance of our algorithm is significantly ahead of the efficient implementation uAVS3e for AVS3 and the reference software HM-16.20 for High Efficiency Video Coding (HEVC), even though there is 9.96% loss on BD-Rate Y. Considering its importance for the hardware implementation of AVS3-based 8K real-time encoder, the coding loss is acceptable. Moreover, the hardware simulation results on VU440 FPGA with Vivado 2019 show that our algorithm can support 61.12 frames per second (fps) CU partition for 8K UHD videos with only 0.00%, 0.00%, 1.01%, and 7.78% consumption of BRAM_18K, DSP48E, FF, and LUT, respectively. Additionally, with dual-path parallelism, 122.24 fps also can be implemented with controllable resource utilization, which achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Objective: This study aims to evaluate the effect of in-vehicle audio warning at flashing-light-controlled grade crossings based on driving simulation and eye-tracking systems. Background: Collisions at flashing-light-controlled grade crossings have severe consequences. In-vehicle audio warning has the potential to regulate driver behavior. However, whether this improvement occurs through priming drivers' visual search patterns is not yet clear. Method: Drivers' visual activity and behaviors were recorded. The effect of a warning was tested with a series of flashing light trigger times (FLTTs) ranging from 2s to 6s with a 1s increment. Different driving conditions (i.e., clear and fog) and driver experience were considered in the experiment design. Results: Warnings could guide the allocation of both overt and covert attention, as well as raise drivers' situation awareness, manifesting as the enhanced perception of signs and better understanding of the flashing red light. Significant improvement in the stop-compliance rate was found in warning scenarios, particularly with a late FLTT. The decreased saccade duration and increased fixation duration on the signal implied a dilemma-zone effect when the FLTT was lower than 4s. Furthermore, reduced fixation duration on signs and signals was found in foggy conditions. Non-professional drivers had a wider search range than their counterparts. Conclusion: In-vehicle audio warning is an effective countermeasure for improving crossing safety by optimizing visual search strategy. Application: In-vehicle audio warnings warrant promotion at grade crossings based on the driver assistance system. [ABSTRACT FROM AUTHOR]

Compared with other traditional community discovery algorithms, density peak clustering algorithm is more efficient in getting network structures through clustering. However, DPC needs to contain the distance information between all nodes as sources, so it cannot directly processing the complex network represented by the adjacency matrix. DPC introduces truncation distance when calculating the local density of nodes, which is usually set as a fixed value according to experience, and lacks self-adaptability for different network structures. A feasible solution to those problems is to combined rough set theory and kernel fuzzy similarity measures. In this work, we present overlapping community detection algorithm based on improved rough entropy fusion density peak. The algorithm applied rough set theory to attribute reduction of massive high-dimensional data. Another algorithm defines the similarity of sample points by the inner product between two vectors on the basis of fuzzy partition matrix. Finally, a community detection algorithm based on rough entropy and kernel fuzzy density peaks clustering (CDRKD) has proposed by combining the two algorithms above, we perform an extensive set of experiments to verify the effectiveness and feasibility of the algorithm. [ABSTRACT FROM AUTHOR]

Recently, we have observed an exponential increase of user-generated content (UGC) videos. The distinguished characteristic of UGC videos originates from the video production and delivery chain, as they are usually acquired and processed by non-professional users before uploading to the hosting platforms for sharing. As such, these videos usually undergo multiple distortion stages that may affect visual quality before ultimately being viewed. Inspired by the increasing consensus that the optimization of the video coding and processing shall be fully driven by the perceptual quality, in this paper, we propose to study the quality of the UGC videos from both objective and subjective perspectives. We first construct a UGC video quality assessment (VQA) database, aiming to provide useful guidance for the UGC video coding and processing in the hosting platform. The database contains source UGC videos uploaded to the platform and their transcoded versions that are ultimately enjoyed by end-users, along with their subjective scores. Furthermore, we develop an objective quality assessment algorithm that automatically evaluates the quality of the transcoded videos based on the corrupted reference, which is in accordance with the application scenarios of UGC video sharing in the hosting platforms. The information from the corrupted reference is well leveraged and the quality is predicted based on the inferred quality maps with deep neural networks (DNN). Experimental results show that the proposed method yields superior performance. Both subjective and objective evaluations of the UGC videos also shed lights on the design of perceptual UGC video coding. [ABSTRACT FROM AUTHOR]

In Versatile Video Coding (VVC), reference picture resampling (RPR) is supported to change resolution without inserting an instantaneous decoder refresh (IDR) picture or intra random access picture (IRAP). The introduction of RPR enables VVC to adapt to unstable network conditions, which benefits real-time application scenarios a lot. However, the existing bitrate leap and the low quality of the upsampled frames are the bottlenecks of promoting its application. In this paper, we propose an efficient framework to alleviate bitrate surges and further improve the coding performance. The contributions of this paper mainly include the following three aspects. Firstly, it is proved theoretically and statistically that the temporal long-distance high-resolution (HR) reference pictures can provide better prediction than temporal short-distance low-resolution (LR) reference pictures when the temporal distance between them is less than a threshold. Secondly, motivated by the theoretical analysis, existing HR reference pictures in the reference pictures list (RPL) are used to replace LR reference pictures and an adaptive decision on the number of HR reference pictures kept in RPL is applied. Thirdly, an adaptive upsampling filter (AUF) is proposed to restore pictures with the original resolution more accurately. Simulation results show 15.09% BD-rate savings on average can be achieved for the 2:1 scaling ratio under RPR configurations. For the 1.5:1 scaling ratio, the BD-rate saving is 8.14%. Besides, the bitrate surge in RPR is reduced greatly. About 25% bits reductions on average at the leaping point can be observed and the bits decrease rate is up to 50% for sequences with little and slow motion. [ABSTRACT FROM AUTHOR]

Flexible block partition and newly adopted intra coding tools bring significant performance improvement for next generation video coding, and meanwhile introduce non-negligible encoding complexity increment. This paper presents a scalable intra coding optimization (SICO) scheme for the third generation of audio video coding standard (AVS3). The complexity distribution and inheriting relationship among different partitioning, as well as intra prediction modes, are systematically analyzed. Subsequently, low-complexity algorithms are proposed, which could early exclude unlikely coding modes safely. In particular, a data-driven binary classifier is elegantly trained for the determination of coding unit partitioning. Moreover, the preliminary coding information can be exhaustively utilized for the mode decision in a low-cost manner. The proposed method provides scalable intra coding optimizing solutions, which is eligible to cater to various application scenarios. Experimental results show that the proposed method could achieve a wide range of encoding complexity reduction from 18% to 76% with moderate compression performance loss. One implementation of the proposed method has been adopted to the AVS3 reference software. [ABSTRACT FROM AUTHOR]

Dynamic textures are time-varying motion patterns that exhibit certain temporal stationarity. The motion of such patterns is usually anisotropic, which is a great challenge for video coding. In this paper, we firstly analyze the predictive and temporal characteristics of dynamic textures and, based on the analysis, a pixel-level segmentation-synthesis framework for dynamic texture compression is proposed to improve predictive-coding efficiency. This framework consists of three sub-modules: dynamic texture segmentation, dynamic texture synthesis and fusion process. A deep-learning-based segmentation method is introduced to obtain a pixel-level mask. Subsequently, optical flow fields are used to generate flowlines and help analyze the motion periodicity of a given sequence. The flowlines and period values are utilized as inputs for flow-based dynamic texture synthesis. The segmented dynamic texture area is then reconstructed using the synthesized results at the decoder side at pixel-level, such that the overhead for transmitting these contents could be saved. The segmented mask is transmitted through an All-Zero-Run-Length coding algorithm and an effective fusion module is further proposed to reduce edge artifacts that occur at the boundary of dynamic texture areas. Coding blocks have been classified into three types and a weighting reconstruction scheme is performed. The proposed framework has been integrated into VTM-10.0. Experimental results on a collected dynamic texture test set demonstrate that 15.1% and 16.1% MOS BD-rate savings can be achieved on average for LDB and LDP configurations, respectively. [ABSTRACT FROM AUTHOR]

The recent trend in neural image compression (NIC) research could be generally grounded into two categories: analysis-synthesis transform network improvements and entropy estimation optimization. They promote the compression efficiency of NIC by leveraging more expressive network structures and advanced entropy models respectively. From a different but more systematic viewpoint, we extend the horizon of NIC from software- to hardware-based lossy compression using more resource-constrained platforms, such as field programmable gate array (FPGA) or deep-learning processor unit (DPU). In this paper, we propose a novel hardware-oriented NIC system for real-time edge-computing video services. We for the first time present FPX-NIC, an FPGA-accelerated NIC framework designed for hardware encoding, which consists of a novel NIC scheme and an energy-efficient neural network (NN) deployment method. The former contribution is a block-based adaptive NIC approach based on local content characteristics. Essential side-information is also signalled to realize adaptive patch representation. The critical advantage of our latter contribution lies in the network-reconfigurable framework plus fixed-precision weights quantization method that takes advantage of quantization-aware post training procedure to compensate the performance degradation caused by quantization error. Therefore it is able to improve both processing speed and energy efficiency. We finally establish an intelligent video coding system using the proposed scheme, enabling visual capturing, neural encoding, decoding, and display, realizing 4K ultra-high-definition (UHD) all intra neural video coding on edge-computing devices. [ABSTRACT FROM AUTHOR]

It is challenging to restore low-resolution (LR) images to super-resolution (SR) images with correct and clear details. Existing deep learning works almost neglect the inherent structural information of images, which acts as an important role for visual perception of SR results. In this paper, we design a hierarchical feature exploitation network to probe and preserve structural information in a multi-scale feature fusion manner. First, we propose a cross convolution upon traditional edge detectors to localize and represent edge features. Then, cross convolution blocks (CCBs) are designed with feature normalization and channel attention to consider the inherent correlations of features. Finally, we leverage multi-scale feature fusion group (MFFG) to embed the cross convolution blocks and develop the relations of structural features in different scales hierarchically, invoking a lightweight structure-preserving network named as Cross-SRN. Experimental results demonstrate the Cross-SRN achieves competitive or superior restoration performances against the state-of-the-art methods with accurate and clear structural details. Moreover, we set a criterion to select images with rich structural textures. The proposed Cross-SRN outperforms the state-of-the-art methods on the selected benchmark, which demonstrates that our network has a significant advantage in preserving edges. [ABSTRACT FROM AUTHOR]

In-loop filters have attracted increasing attention due to the remarkable noise-reduction capability in the hybrid video coding framework. However, the existing in-loop filters in Versatile Video Coding (VVC) mainly take advantage of the image local similarity. Although some non-local based in-loop filters can make up for this shortcoming, the widely-used unsupervised parameter estimation method by non-local filters limits the performance. In view of this, we propose a deformable Wiener Filter (DWF). It combines the local and non-local characteristics and supervisedly trains the filter coefficients based on the Wiener Filter theory. In the filtering process, local adjacent samples and non-local similar samples are first derived for each sample of interest. Then the to-be-filtered samples are classified into specific groups based on the patch-level noise and sample-level characteristics. Samples in each group share the same filter coefficients. After that, the local and non-local reference samples are adaptively fused based on the classification results. Finally, the filtering operation with outlier data constraints is conducted for each to-be-filtered sample. Moreover, the performance of the proposed DWF is analyzed with different reference sample derivation schemes in detail. Simulation results show that the proposed approach achieves 1.16%, 1.92%, and 2.67% bit-rate savings on average compared to the VTM-11.0 for All Intra, Random Access, and Low-Delay B configurations, respectively. [ABSTRACT FROM AUTHOR]

In the latest video coding standard, namely Versatile Video Coding (VVC), more directional intra modes and reference lines have been utilized to improve prediction efficiency. However, complex content still cannot be predicted well with only the adjacent reference samples. Although nonlocal prediction has been proposed to further improve the prediction efficiency in existing algorithms, explicit signalling or matching error potentially limits the coding efficiency. To address these issues, we propose a joint local and nonlocal progressive prediction scheme, targeting at improving nonlocal prediction accuracy without additional signalling. Specifically, template matching based prediction (TMP) is conducted firstly to derive an initial nonlocal predictor. Based on the first prediction and previously decoded reconstruction information, a local template, including inner textures and neighboring reconstruction, is carefully designed. With the local template involved in nonlocal matching process, a more accurate nonlocal predictor can be found progressively in the second prediction. Finally, the coefficients from the two predictions are fused and transmitted in bitstreams. In this way, more accurate nonlocal predictor can be derived implicitly with local information instead of being explicitly signalled. Experimental results on the reference software VTM-9.0 of VVC show that the method achieves 1.02% BD-Rate reduction for natural sequences and 2.31% BD-Rate reduction for screen content videos under all intra (AI) configuration. [ABSTRACT FROM AUTHOR]

Existing compression methods typically focus on the removal of signal-level redundancies, while the potential and versatility of decomposing visual data into compact conceptual components still lack further study. To this end, we propose a novel conceptual compression framework that encodes visual data into compact structure and texture representations, then decodes in a deep synthesis fashion, aiming to achieve better visual reconstruction quality, flexible content manipulation, and potential support for various vision tasks. In particular, we propose to compress images by a dual-layered model consisting of two complementary visual features: 1) structure layer represented by structural maps and 2) texture layer characterized by low-dimensional deep representations. At the encoder side, the structural maps and texture representations are individually extracted and compressed, generating the compact, interpretable, inter-operable bitstreams. During the decoding stage, a hierarchical fusion GAN (HF-GAN) is proposed to learn the synthesis paradigm where the textures are rendered into the decoded structural maps, leading to high-quality reconstruction with remarkable visual realism. Extensive experiments on diverse images have demonstrated the superiority of our framework with lower bitrates, higher reconstruction quality, and increased versatility towards visual analysis and content manipulation tasks. [ABSTRACT FROM AUTHOR]