Search

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for $Y'C_{B}C_{R}$ and of about 6% to 14% for $R'G'B'$ screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for $Y'C_{B}C_{R}$ and 43% to 66% for $R'G'B'$ screen content.

Recently, neural style transfer has drawn many attentions and significant progresses have been made, especially for image style transfer. However, flexible and consistent style transfer for videos remains a challenging problem. Existing training strategies, either using a significant amount of video data with optical flows or introducing single-frame regularizers, have limited performance on real videos. In this paper, we propose a novel interpretation of temporal consistency, based on which we analyze the drawbacks of existing training strategies; and then derive a new compound regularization. Experimental results show that the proposed regularization can better balance the spatial and temporal performance, which supports our modeling. Combining with the new cost formula, we design a zero-shot video style transfer framework. Moreover, for better feature migration, we introduce a new module to dynamically adjust inter-channel distributions. Quantitative and qualitative results demonstrate the superiority of our method over other state-of-the-art style transfer methods. Our project is publicly available at: https://daooshee.github.io/CompoundVST/.

Direct speech-to-image translation without text is an interesting and useful topic due to the potential applications in human-computer interaction, art creation, computer-aided design. etc. Not to mention that many languages have no writing form. However, as far as we know, it has not been well-studied how to translate the speech signals into images directly and how well they can be translated. In this paper, we attempt to translate the speech signals into the image signals without the transcription stage. Specifically, a speech encoder is designed to represent the input speech signals as an embedding feature, and it is trained with a pretrained image encoder using teacher-student learning to obtain better generalization ability on new classes. Subsequently, a stacked generative adversarial network is used to synthesize high-quality images conditioned on the embedding feature. Experimental results on both synthesized and real data show that our proposed method is effective to translate the raw speech signals into images without the middle text representation. Ablation study gives more insights about our method., Accepted by JSTSP

The quad-tree plus binary-tree (QTBT) partition structures was adopted into the versatile video coding (VVC) standard. In the QTBT partition structure, a dual tree structure can be applied on intra slices where the partitioning structures for luminance and chrominance components are separate. The dual tree structure is efficient for camera captured videos. However, it may be not the case for screen content videos. Therefore, adaptive dual tree structure is proposed in this paper wherein the coding structure of each coding tree unit is switched between separate and joint coding structure to adapt the textures adaptively. The proposed scheme is implemented into the reference software of the VVC standard-VTM. Simulation results report that up to 4.5% BD bitrate saving can achieve on the typical screen content videos when compared to VTM5.

Efficient compression of omnidirectional video is important for emerging virtual reality applications. To compress this kind of video, each frame is projected to a 2D plane [e.g., equirectangular projection (ERP) map] first, adapting to the input format of existing video coding systems. At the display side, an inverse projection is applied to the reconstructed video to restore signals in spherical domain. Such a projection, however, makes presentation and encoding in different domains. Thus, an encoder agnostic to the projection performs inefficiently. In this paper, we analyze how a projection influences the distortion measurements in different domains. Based on the analysis, we propose a scheme to optimize the encoding process based on signals’ distortion in spherical domain. With the proposed optimization, an average 4.31% (up to 9.67%) luma BD-rate reduction is achieved for ERP in random access configuration. The corresponding bit saving is averagely 10.84% (up to 34.44%) when considering the viewing field being $\pi /2$ . The proposed method also benefits other projections and viewport settings, with a marginal complexity increase.

Establishing an interactive screen sharing system that supports ultra high resolution (such as 4k) is challenging, with latency and frame rate playing important roles in user experience. The screen frame needs to be compressed efficiently without consuming extensive computational resources. We present a hardware-accelerated system for real-time screen sharing, which decreases encoding workload by exploiting content redundancies between successive screen frames. We propose a multiple codec approach that utilizes several encoders with H.264 Advanced Video Coding (H.264/AVC) of different input sizes, creating savings in encoding time by selecting the appropriate one for updated screen content. An optimized metadata processing method is proposed as well. Small but distant updates within a frame can be split into independent frames for more efficient compression, which is also beneficial for interactive latency. In the evaluation, the proposed system takes less encoding time than general single codec implementation in common screen sharing scenarios. Measurement for latency shows that the end-to-end latency for 4K resolution screen sharing is only about 17–25 ms, which makes the proposed system suitable for various applications in local wired and wireless connections.

Wireless virtual reality (VR) applications that provide users extraordinary viewing experience are now drawing great attentions. Transmitting VR video via wireless channel to users’ head-mounted display devices efficiently with low latency is very important for many emerging VR applications. In this paper, we propose a pseudo-analog transmission framework named OmniCast, which provides graceful quality degradation and competitive performance for unpredictably varying wireless channels while featuring low latency, low complexity, and low energy cost. In particular, we analyze the influence of projection between the spherical representation and the 2-D plane representation, and derive a power optimization scheme to minimize the distortion on the sphere. In addition, we develop an approach to measure the efficiency of decorrelation transform in the spherical domain. Based on that, an appropriate transform option can be determined. Experimental results show that the proposed framework improves the transmission efficiency of omnidirectional videos, while achieving elegant quality degradation for channel fluctuation in a wide channel SNR range.

In this paper, we concentrate on the super-resolution (SR) of compressed screen content video, in an effort to address the real-world challenges by considering the underlying characteristics of screen content. Firstly, we propose a new dataset for the SR of screen content video with different distortion levels. Meanwhile, we design an efficient SR structure that could capture the characteristics of compressed screen content video and manipulate the inner-connections in consecutive compressed low-resolution frames, facilitating the high-quality recovery of the high-resolution counter-part. Moreover, we design a new loss function for network training to better remedy the compression distortion and perceptual distortion. Experimental results demonstrate the effectiveness and superiority of the proposed method.

Versatile Video Coding (VVC) achieves promising compression performances compared with High Efficiency Video Coding (HEVC) while sacrificing the encoding speed. This paper focuses on predicting the partitioning structures with convolutional neural networks to speedup the VVC encoder. Specifically, we formulate the partitioning prediction problem into two alternatives: implicit partitioning prediction based on the split type of subblock boundaries and explicit partitioning prediction from the ensemble partitioning space. Then, we address both formulations using convolutional neural networks.

The third generation of the audio video coding standard (AVS3) is the latest standardization developed by Audio Video Coding Standard Working Group of China on the basis of IEEE 1857.4. It is well known that Intra Block Copy (IBC) significantly improves the coding efficiency of screen content videos which utilizes a block vector to identify a similar block in current reconstructed picture. However, how to better code the block vectors need to be explored. In this paper, we propose a history-based block vector prediction (HBVP) method for better coding block vectors associated with the IBC mode. Inspired by the history-based motion vector design, motion information of previously IBC-coded blocks is maintained in a table and used as BVP for the current block in IBC mode. Thanks to the HBVP, the encoder only needs to signal the HBVP index and block vector difference of an IBC coded block to the decoder, which significantly enhance the coding efficiency of IBC mode. Experimental results show that the proposed method achieves −9.43 % (TGM)/ −5.77 % (MC)/ −0.27% (G) BD rate reduction in IBC-BVD mode and −12.34% (TGM)/ −8.43% (MC)/ −0.57% (G) BD rate reduction in IBC-SKIP/DIRECT mode for SCC sequences in AI configurations with limited encoding-time increase, on the latest AVS reference software platform. The proposed methods were partially adopted by AVS3.

Due to the widespread deployment of fingerprint/face/speaker recognition systems, attacking deep learning based biometric systems has drawn more and more attention. Previous research mainly studied the attack to the vision-based system, such as fingerprint and face recognition. While the attack for speaker recognition has not been investigated yet, although it has been widely used in our daily life. In this paper, we attempt to fool the state-of-the-art speaker recognition model and present \textit{speaker recognition attacker}, a lightweight model to fool the deep speaker recognition model by adding imperceptible perturbations onto the raw speech waveform. We find that the speaker recognition system is also vulnerable to the attack, and we achieve a high success rate on the non-targeted attack. Besides, we also present an effective method to optimize the speaker recognition attacker to obtain a trade-off between the attack success rate with the perceptual quality. Experiments on the TIMIT dataset show that we can achieve a sentence error rate of $99.2\%$ with an average SNR $57.2\text{dB}$ and PESQ 4.2 with speed rather faster than real-time., Comment: Accepted by ICASSP2020

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

Palette mode is a powerful tool for screen content coding in the upcoming versatile video coding (VVC) standard. In the palette mode, escape samples are employed to handle an outlier case. In this paper, a quantization parameter (QP) dependent fixed-length binarization is proposed for escape samples coding to 1) simplify the design and 2) improve coding efficiency. The length is calculated according to the QP for the current block and correspondingly, the dequantization process can be also implemented by only left shifting. The proposed method is evaluated with VVC reference software VTM-6.0 on typical sequences containing "text and graphics with motion". Experimental results report that the proposed scheme can achieve up to 2.0% BD-rate savings compared to VTM-6.0 with a much simpler design.

In this paper, we focus on computationally modeling of the local texture correlations, in an effort to better explore the coding modes with higher priorities in the rate-distortion optimized intra coding. In particular, strong correlations and continuities of local texture with neighboring blocks have been revealed in our analysis, and empirical justifications provide us inspirations on the joint optimization of rate-distortion-complexity when angular modes become finer to adapt the local textures. We examine the philosophy with extensive experiments conducted for refining the intra full-RD list. The results show that better coding performance with on average 0.72% and 3.00% BD-Rate savings for the natural scene and screen content sequences can be achieved in AVS3 test model HPM-5.0 under all intra configuration, with negligible encoding and decoding time variations.

Attacking deep learning based biometric systems has drawn more and more attention with the wide deployment of fingerprint/face/speaker recognition systems, given the fact that the neural networks are vulnerable to the adversarial examples, which have been intentionally perturbed to remain almost imperceptible for human. In this paper, we demonstrated the existence of the universal adversarial perturbations~(UAPs) for the speaker recognition systems. We proposed a generative network to learn the mapping from the low-dimensional normal distribution to the UAPs subspace, then synthesize the UAPs to perturbe any input signals to spoof the well-trained speaker recognition model with high probability. Experimental results on TIMIT and LibriSpeech datasets demonstrate the effectiveness of our model., Comment: Accepted by ICME2020

Transform, as one of the most important modules of mainstream video coding systems, seems very stable over the past several decades. However, recent developments indicate that bringing more options for transform can lead to coding efficiency benefits. In this paper, we go further to investigate how the coding efficiency can be improved over the state-of-the-art method by adapting a transform for each block. We present a variable block-sized signal-dependent transforms (SDTs) design based on the High Efficiency Video Coding (HEVC) framework. For a coding block ranged from $4\times4$ to $32\times32$ , we collect a quantity of similar blocks from the reconstructed area and use them to derive the Karhunen–Loeve transform. We avoid sending overhead bits to denote the transform by performing the same procedure at the decoder. In this way, the transform for every block is tailored according to its statistics, to be signal-dependent. To make the large block-sized SDTs feasible, we present a fast algorithm for transform derivation. Experimental results show the effectiveness of the SDTs for different block sizes, which leads to up to 23.3% bit-saving. On average, we achieve BD-rate saving of 2.2%, 2.4%, 3.3%, and 7.1% under AI-Main10, RA-Main10, RA-Main10, and LP-Main10 configurations, respectively, compared with the test model HM-12 of HEVC. The proposed scheme has also been adopted into the joint exploration test model for the exploration of potential future video coding standard.