Search

One of the major challenges facing video object segmentation (VOS) is the gap between the training and test datasets due to unseen category in test set, as well as object appearance change over time in the video sequence. To overcome such challenges, an adaptive online framework for VOS is developed with bi-decoders mutual learning. We learn object representation per pixel with bi-level attention features in addition to CNN features, and then feed them into mutual learning bi-decoders whose outputs are further fused to obtain the final segmentation result. We design an adaptive online learning mechanism via a deviation correcting trigger such that bi-decoders online mutual learning will be activated when the previous frame is segmented well meanwhile the current frame is segmented relatively worse. Knowledge distillation from the well segmented previous frames, along with mutual learning between bi-decoders, improves generalization ability and robustness of VOS model. Thus, the proposed model adapts to the challenging scenarios including unseen categories, object deformation, and appearance variation during inference. We extensively evaluate our model on widely-used VOS benchmarks including DAVIS-2016, DAVIS-2017, YouTubeVOS-2018, YouTubeVOS-2019, and UVO. Experimental results demonstrate the superiority of the proposed model over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Abnormal crowd behavior detection has recently attracted increasing attention due to its wide applications in computer vision research areas. However, it is still an extremely challenging task due to the great variability of abnormal behavior coupled with huge ambiguity and uncertainty of video contents. To tackle these challenges, we propose a new probabilistic framework named variational abnormal behavior detection (VABD), which can detect abnormal crowd behavior in video sequences. We make three major contributions: (1) We develop a new probabilistic latent variable model that combines the strengths of the U-Net and conditional variational auto-encoder, which also are the backbone of our model; (2) We propose a motion loss based on an optical flow network to impose the motion consistency of generated video frames and input video frames; (3) We embed a Wasserstein generative adversarial network at the end of the backbone network to enhance the framework performance. VABD can accurately discriminate abnormal video frames from video sequences. Experimental results on UCSD, CUHK Avenue, IITB-Corridor, and ShanghaiTech datasets show that VABD outperforms the state-of-the-art algorithms on abnormal crowd behavior detection. Without data augmentation, our VABD achieves 72.24% in terms of AUC on IITB-Corridor, which surpasses the state-of-the-art methods by nearly 5%. [ABSTRACT FROM AUTHOR]

With the emergence of light field imaging in recent years, the compression of its elementary image array (EIA) has become a significant problem. Our coding framework includes modeling and reconstruction. For the modeling, the covariance-matrix form of the 4D Epanechnikov kernel (4D EK) and its correlated statistics were deduced to obtain the 4-D Epanechnikov mixture models (4-D EMMs). A 4D Epanechnikov mixture regression (4D EMR) was proposed based on this 4D EK, and a 4D adaptive model selection (4D AMLS) algorithm was designed to realize the optimal modeling for a pseudo video sequence (PVS) of the extracted key-EIA. A linear function based reconstruction (LFBR) was proposed based on the correlation between adjacent elementary images (EIs). The decoded images realized a clear outline reconstruction and superior coding efficiency compared to high-efficiency video coding (HEVC) and JPEG 2000 below approximately 0.05 bpp. This work realized an unprecedented theoretical application by (1) proposing the 4D Epanechnikov kernel theory, (2) exploiting the 4D Epanechnikov mixture regression and its application in the modeling of the pseudo video sequence of light field images, (3) using 4D adaptive model selection for the optimal number of models, and (4) employing a linear function-based reconstruction according to the content similarity. [ABSTRACT FROM AUTHOR]

Video coding algorithms attempt to minimize the significant commonality that exists within a video sequence. Each new video coding standard contains tools that can perform this task more efficiently compared to its predecessors. Modern video coding systems are block-based wherein commonality modeling is carried out only from the perspective of the block that need be coded next. In this work, we argue for a commonality modeling approach that can provide a seamless blending between global and local homogeneity information. For this purpose, at first the frame that need be coded, is recursively partitioned into rectangular regions based on the homogeneity information of the entire frame. After that each obtained rectangular region’s feature descriptor is taken to be the average value of all the pixels’ intensities encompassing the region. In this way, the proposed approach generates a coarse representation of the current frame by minimizing both global and local commonality. This coarse frame is computationally simple and has a compact representation. It attempts to preserve important structural properties of the current frame which can be viewed subjectively as well as from improved rate-distortion performance of a reference scalable HEVC coder that employs the coarse frame as a reference frame for encoding the current frame. [ABSTRACT FROM AUTHOR]

During the past few years, face videos, e.g., video conference, interviews and variety shows, have grown explosively with millions of users over social media networks. Unfortunately, the existing compression algorithms are applied to these videos for reducing bandwidth, which also bring annoying artifacts to face regions. This paper addresses the problem of face quality enhancement in compressed videos by reducing the artifacts of face regions. Specifically, we establish a compressed face video (CFV) database, which includes 196,337 faces in 214 high-quality video sequences and their corresponding 1,712 compressed sequences. We find that the faces of compressed videos exhibit tremendous scale variation and quality fluctuation. Motivated by scalable video coding, we propose a multi-scale recurrent scalable network (MRS-Net+) to enhance the quality of multi-scale faces in compressed videos. The MRS-Net+ is comprised by one base and two refined enhancement levels, corresponding to the quality enhancement of small-, medium- and large-scale faces, respectively. In the multi-level architecture of our MRS-Net+, small-/medium-scale face quality enhancement serves as the basis for facilitating the quality enhancement of medium-/large-scale faces. We further develop a landmark-assisted pyramid alignment (LPA) subnet to align faces across consecutive frames, and then apply the mask-guided quality enhancement (QE) subnet for enhancing multi-scale faces. Finally, experimental results show that our MRS-Net+ method achieves averagely 1.196 dB improvement of peak signal-to-noise ratio (PSNR) and 23.54% saving of Bjøntegaard distortion-rate (BD-rate), significantly outperforming other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The proliferation of low-cost DRAM chipsets now begins to allow for the consideration of substantially-increased decoded picture buffers in advanced video coding standards such as HEVC, VVC, and Google VP9. At the same time, the increasing demand for rapid scene changes and multiple scene repetitions in entertainment or broadcast content indicates that extending the frame referencing interval to tens of minutes or even the entire video sequence may offer coding gains, as long as one is able to identify frame similarity in a computationally- and memory-efficient manner. Motivated by these observations, we propose a “stitching” method that defines a reference buffer and a reference frame selection algorithm. Our proposal extends the referencing interval of inter-frame video coding to the entire length of video sequences. Our reference frame selection algorithm uses well-established feature descriptor methods that describe frame structural elements in a compact and semantically-rich manner. We propose to combine such compact descriptors with a similarity scoring mechanism in order to select the frames to be “stitched” to reference picture buffers of advanced inter-frame encoders like HEVC, VVC, and VP9 without breaking standard compliance. Our evaluation on synthetic and real-world video sequences with the HEVC and VVC reference encoders shows that our method offers significant rate gains, with complexity and memory requirements that remain manageable for practical encoders and decoders. [ABSTRACT FROM AUTHOR]

With the growing data consumption of emerging video applications and users’ requirement for higher resolutions, up to 8K, a huge effort has been made in video compression technologies. Recently, versatile video coding (VVC) has been standardized by the moving picture expert group (MPEG), providing a significant improvement in compression performance over its predecessor high efficiency video coding (HEVC). In this paper, we provide a comparative subjective quality evaluation between VVC and HEVC standards for 8K resolution videos. In addition, we evaluate the perceived quality improvement offered by 8K over UHD 4K resolution. The compression performance of both VVC and HEVC standards has been conducted in random access (RA) coding configuration, using their respective reference software, VVC test model (VTM-11) and HEVC test model (HM-16.20). Objective measurements, using PSNR, MS-SSIM and VMAF metrics have shown that the bitrate gains offered by VVC over HEVC for 8K video content are around 31%, 26% and 35%, respectively. Subjectively, VVC offers an average of around 41% of bitrate reduction over HEVC for the same visual quality. A compression gain of 50% has been reached for some tested video sequences regarding a Student’s t-test analysis. In addition, for most tested scenes, a significant visual difference between uncompressed 4K and 8K has been noticed. [ABSTRACT FROM AUTHOR]

In predictive transform video coding, optimal bit allocation and quantization parameter (QP) estimation are important to control the bit rate of blocks, frames and the whole sequence. Common solutions to this problem rely on trained models to approximate the rate-distortion (R-D) characteristics of the video content during coding. Moreover, these solutions are mainly targeted for natural content sequences, whose characteristics differ greatly from those of screen content (SC) sequences. In this article, we depart from such trained R-D models and propose a low-complexity RC method for SC sequences that leverages the availability of information about the R-D characteristics of previously coded blocks within a frame. Namely, our method first allocates bits at the frame- and block-levels based on their motion and texture characteristics. It then approximates the R-D and R-QP curves of each block by a set control points and random sample consensus (RANSAC). Finally, it computes the appropriate block-level QP values to attain a target bit rate with the minimum distortion possible. The proposed RC method is embedded into a standard High-Efficiency Video Coding (H.265/HEVC) encoder and evaluated on several SC sequences. Our results show that our method not only attains better R-D performance than that of H.265/HEVC and other methods designed for SC sequences but also attains a more constant and higher reconstruction quality on all frames. [ABSTRACT FROM AUTHOR]

To cope with rapid growth of video services, we propose a non-orthogonal multiple access (NOMA) based scalable video multicast (NOMA-SVM) framework for mobile networks, by exploiting NOMA's specific potential in scalable video multicast transmission. We consider statistical channels, instead of channels with perfect estimation, in the proposed NOMA-SVM framework in order to capture the realistic channel behaviors. As quality of experience (QoE) is a better metric than throughput for video transmission, QoE-driven power allocation is performed among multiple video layers in the proposed NOMA-SVM framework, in which users can decode video with quality proportional to their channel conditions. Specifically, we formulate the power allocation problem with the goal to maximize the average QoE over all users while guaranteeing the basic services of these users. To solve such a non-convex discrete problem, an optimal algorithm is developed based on the hidden monotonicity of the problem. A suboptimal algorithm is also proposed with much lower complexity in order to meet the practical needs. Simulation results show that the proposed algorithms outperform existing orthogonal multiple access (OMA) and NOMA based algorithms under various multicast scenarios in terms of QoE. [ABSTRACT FROM AUTHOR]

In this paper, the problem of describing visual contents of a video sequence with natural language is addressed. Unlike previous video captioning work mainly exploiting the cues of video contents to make a language description, we propose a reconstruction network (RecNet) in a novel encoder-decoder-reconstructor architecture, which leverages both forward (video to sentence) and backward (sentence to video) flows for video captioning. Specifically, the encoder-decoder component makes use of the forward flow to produce a sentence based on the encoded video semantic features. Two types of reconstructors are subsequently proposed to employ the backward flow and reproduce the video features from local and global perspectives, respectively, capitalizing on the hidden state sequence generated by the decoder. Moreover, in order to make a comprehensive reconstruction of the video features, we propose to fuse the two types of reconstructors together. The generation loss yielded by the encoder-decoder component and the reconstruction loss introduced by the reconstructor are jointly cast into training the proposed RecNet in an end-to-end fashion. Furthermore, the RecNet is fine-tuned by CIDEr optimization via reinforcement learning, which significantly boosts the captioning performance. Experimental results on benchmark datasets demonstrate that the proposed reconstructor can boost the performance of video captioning consistently. [ABSTRACT FROM AUTHOR]

Multilevel diversity coding is a classical coding model where multiple mutually independent information messages are encoded, such that different reliability requirements can be afforded to different messages. It is well known that superposition coding, namely separately encoding the independent messages, is optimal for symmetric multilevel diversity coding (SMDC) (Yeung-Zhang 1999). In the current paper, we consider weakly secure SMDC where security constraints are injected on each individual message, and provide a complete characterization of the conditions under which superposition coding is sum-rate optimal. Two joint coding strategies, which lead to rate savings compared to superposition coding, are proposed, where some coding components for one message can be used as the encryption key for another. By applying different variants of Han’s inequality, we show that the lack of opportunity to apply these two coding strategies directly implies the optimality of superposition coding. It is further shown that under a set of particular security constraints, one of the proposed joint coding strategies can be used to construct a code that achieves the optimal rate region. [ABSTRACT FROM AUTHOR]

Video-based person re-identification (re-ID) refers to matching people across camera views from arbitrary unaligned video footages. Existing methods rely on supervision signals to optimise a projected space under which the distances between inter/intra-videos are maximised/minimised. However, this demands exhaustively labelling people across camera views, rendering them unable to be scaled in large networked cameras. Also, it is noticed that learning effective video representations with view invariance is not explicitly addressed for which features exhibit different distributions otherwise. Thus, matching videos for person re-ID demands flexible models to capture the dynamics in time-series observations and learn view-invariant representations with access to limited labeled training samples. In this paper, we propose a novel few-shot deep learning approach to video-based person re-ID, to learn comparable representations that are discriminative and view-invariant. The proposed method is developed on the variational recurrent neural networks (VRNNs) and trained adversarially to produce latent variables with temporal dependencies that are highly discriminative yet view-invariant in matching persons. Through extensive experiments conducted on three benchmark datasets, we empirically show the capability of our method in creating view-invariant temporal features and state-of-the-art performance achieved by our method. [ABSTRACT FROM AUTHOR]

Shape is a popular way to define objects and shape coding is a key technique for object-based 3D video applications. In this paper, the issue of efficient shape coding for object-based 3D video applications is addressed, and a novel contour-based and chain-represented scheme is proposed. For a given 3D shape video, contour extraction and preprocessing are first implemented followed by chain-based representation. Then, to achieve high coding efficiency, a chain-based prediction and compensation technique is developed based on joint motion-compensated prediction and disparity-compensated prediction to effectively exploit the intra-view temporal correlation and the inter-view spatial correlation. Experiments are conducted, and the results demonstrate that the proposed scheme is more efficient than the existing methods, including state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper proposes a new metric to measure judder artifacts of video sequences. The judder artifacts appear as non-smooth motions in hold-type displays when the frame rate is low and object motion is fast. To analyze the judder artifacts in video sequences, the proposed judder metric is defined by analyzing the effects and cross-relations of judder features in the video sequences. The judder features include spatial features of image gradients and temporal features of motion vectors and the frame rate. In addition, sensitivity of the human visual system (HVS) is considered to determine the perceptual judder artifacts because it has special characteristics of sensitivity to image brightness and contrast. Therefore, a sensitivity map of the HVS is used to mask the judder artifacts. Then, the degree of perceptual judder artifacts is estimated using the judder features and a regression model. The experimental results demonstrate that the proposed judder metric is highly correlated with the subjective assessment results. [ABSTRACT FROM AUTHOR]

Eyeblinks are having the significance to analyze the attention, fatigue, behaviour and emotion of an individual. Eyeblink recognition is adopted by many medical and surveillance applications to identify the person's state. The eye blink recognition on videos requires tracking the eye region and to count the number of eye blinks. In this paper, a three-stage model is presented to detect the eye blinks accurately. In the first stage, the frame similarity analysis, background separation, positional and mathematical filters are applied collectively to identify the effective eye region on unique frames. In the second stage, the similarity analysis using wavelet decomposition and statistical filters are applied on the segmented eye region. The filtered evaluation is performed to identify the change on the eye region of continuous segmented frames. At the final stage, distance driven map on structural and statistical features is applied to remove the invalid frame changes and to obtain the accurate eye blink count. The proposed model is applied on real time, web-collected and the NRC-IIT dataset videos. These complex videos are associated to the indoor and outdoor environments. The news reading and other complex video sequences are analyzed in this research. The observations identified that the proposed model has reduced the possible generated errors and provided the accurate detection of eye blinks. [ABSTRACT FROM AUTHOR]

Video classification is highly important and has widespread applications, such as video search and intelligent surveillance. Video naturally contains both static and motion information, which can be represented by frames and optical flow, respectively. Recently, researchers have generally adopted deep networks to capture the static and motion information separately, which has two main limitations. First, the coexistence relationship between spatial and temporal attention is ignored, although they should be jointly modeled as the spatial and temporal evolutions of video to learn discriminative video features. Second, the strong complementarity between static and motion information is ignored, although they should be collaboratively learned to enhance each other. To address the above two limitations, this paper proposes the two-stream collaborative learning with spatial-temporal attention (TCLSTA) approach, which consists of two models. First, for the spatial-temporal attention model, the spatial-level attention emphasizes the salient regions in a frame, and the temporal-level attention exploits the discriminative frames in a video. They are mutually enhanced to jointly learn the discriminative static and motion features for better classification performance. Second, for the static-motion collaborative model, it not only achieves mutual guidance between static and motion information to enhance the feature learning but also adaptively learns the fusion weights of static and motion streams, thus exploiting the strong complementarity between static and motion information to improve video classification. Experiments on four widely used data sets show that our TCLSTA approach achieves the best performance compared with more than 10 state-of-the-art methods. [ABSTRACT FROM AUTHOR]

We introduce the term video co-saliency to denote the task of extracting the common noticeable, or salient, regions from multiple relevant videos. The proposed video co-saliency approach accounts for both inter-video foreground correspondences and intra-video saliency stimuli to emphasize the salient foreground regions of video frames and, at the same time, disregard irrelevant visual information of the background. Compared with image co-saliency, it is more reliable due to the utilization of temporal information of video sequence. Benefiting from the discriminability of video co-saliency, we present a unified framework for segmenting out the common salient regions of relevant videos, guided by video co-saliency prior. Unlike naive video co-segmentation approaches employing simple color differences and local motion features, the presented video co-saliency provides a more powerful indicator for the common salient regions, thus conducting video co-segmentation efficiently. Extensive experiments show that the proposed method successfully infers video co-saliency and extracts the common salient regions, outperforming the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Capturing a scene using multiple cameras from different angles is expected to provide the necessary interactivity in the 3-D space to satisfy end-users’ demands for observing objects and actions from different angles and depths. Existing multiview video coding (MVC) technologies face tradeoff among rate-distortion performance, random access frame delay, i.e., interactivity, and computational time. To address above mentioned tradeoffs, a novel cuboid MVC strategy is proposed with 3-D frame referencing structure to improve interactivity and computational time, an additional reference frame to improve rate-distortion performance for occluded areas, and visual attention-based bit allocation to provide better perceptual video quality. The experimental results reveal that the proposed scheme provides better interactivity, reduced computational time, and better perceptual quality compared to the 3D-HEVC implementation, HTM 15.0. [ABSTRACT FROM AUTHOR]

Scalable video coding (SVC) is a coding paradigm that allows once-encoded video content to be used in diverse scenarios. SVC-coded videos can be transmitted and rendered at specified bitrates according to network bandwidth and end device requirements. In this paper, an adaptive bit allocation algorithm is proposed for the emerging scalable High Efficiency Video Coding (SHVC) standard. The bit budget at the group-of-pictures level is allocated according to buffer occupancy. Picture complexity, measured using the predicted mean absolute difference (MAD), buffer occupancy, and hierarchical level, is proposed for regulating the bitrate at the picture level. The MAD of the current picture is predicted using a novel mean prediction error (MPE) model, which is obtained from the advanced motion vector prediction, and the test zone search specified in SHVC and the associated reference software of SHVC. Moreover, MPE is used to determine the number of assigned bits at the coding-tree-unit level. The bit budget of each level is incorporated with the $R-\lambda $ model for computing the required quantization parameter. Experimental results reveal that the proposed method achieves accurate bitrates with enhanced and consistent visual quality and more satisfactorily controls buffer occupancy compared with the state-of-the-art approaches reported in the literature. [ABSTRACT FROM PUBLISHER]

In High Efficiency Video Coding (HEVC) standard, higher video resolutions employ larger integer Discrete Cosine Transform (DCT)/inverse DCT (IDCT) block sizes. In this study, the authors propose two high‐throughput unified DCT/IDCT architectures. The proposed architectures can process variable DCT/IDCT block sizes according to the HEVC standard. The proposed architectures were prototyped on TSMC 65 nm CMOS technology. The prototyping results show that the two unified architectures have throughput of 15.24 and 16.03 Gsps, respectively, and they can encode video sequences with resolutions up to 8 K at 120 fps and decode the same resolution at 240 fps using only one circuit for both DCT and IDCT. [ABSTRACT FROM AUTHOR]

The random-access configuration is employed in H.265/HEVC video coding to make inter-frame prediction more efficient. The coding efficiency under the random-access configuration is closely related to the quantization parameter cascading (QPC) technique which determines the QP for encoding pictures in different temporal layers. The present QPC technique for the random-access configuration in H.265/HEVC is not optimized. In this paper, first, a rate-distortion-optimization-based technique for QPC, referred to as RDO-QPC, is proposed for the random-access configuration in H.265/HEVC. Based on the results from RDO-QPC, a simplified QPC technique, referred to as SRDO-QPC, is proposed. The experimental results verify the efficiency of the proposed two QPC techniques. Compared with the original configuration of QPC in H.265/HEVC, RDO-QPC achieves an average gain of 0.19 dB in APSNR and an average reduction of 4.87% in ABR, while SRDO-QPC achieves an average gain of 0.17 dB in APSNR and an average reduction of 4.32% in ABR. These two QPC techniques can both be used in practical situations, meeting different requirements of computational complexity. [ABSTRACT FROM AUTHOR]

This paper presents a scalable video coding (SVC)-based open-loop multi-input multioutput (MIMO) scheme for broadcast services such as LTE broadcast/evolved multimedia broadcast multicast service and terrestrial broadcast TV/ATSC 3.0. It intends to guarantee the minimum essential information (as a base layer) and obtain additional information for improving the video quality (as enhancement layers) when multiple transmit and receive antennas are employed. Two types of SVC-based open-loop MIMO broadcasting schemes are considered: 1) SVC-MIMO-time division multiplexing (TDM) and 2) SVC-MIMO-spatial multiplexing (SM). They provide differentiated video quality that depends on the average channel gain of individual users by employing different modulation and coding scheme (MCS) levels for unequal error protection of individual layers, i.e., a lower MCS level for the base layer and a higher MCS level for the enhancement layers. In particular, SVC-MIMO-SM extends the per-layer service coverage by performing a block-wise successive interference cancellation (SIC) from the base layer up to the highest enhancement layer in the receiver. As a result, the base layer can be decoded with the most robust MCS while the enhancement layers can be decoded with an SIC diversity gain. Furthermore, we formulate an optimal power allocation problem for the SVC-MIMO-SM scheme that can maximize the average system utility function while ensuring the target service coverage. Optimal solutions for different cases are derived by identifying the Karush–Kuhn–Tucker conditions for the given problem. Our numerical results demonstrate that the optimal power allocation provides more average system utility gain than SVC-MIMO-TDM and SVC-MIMO-SM without power control schemes, indicating the enhancement in the per-layer service coverage performance. [ABSTRACT FROM PUBLISHER]

Soft errors resulting from encoding video sequences on unreliable hardware can create significant artifacts in decoded video sequences, contributing to extreme video quality degradation. Modern systems are required to operate under increasingly challenging constraints, including smaller feature sizes and lower operating voltage, increasing the likelihood of soft errors in the video encoding hardware. These conditions are of particular concern for energy-limited battery-operated systems since they may be required to operate in nonideal environments and/or continue operating with a practically depleted energy source. The proposed parallel independent signature processing design performs error detection and mitigation in video encoding hardware, enabling a graceful degradation of quality when encoding by using unreliable hardware. The effects of soft errors are minimized by preventing the error propagation normally associated with errors in encoded video sequences. This allows for the recovery of quality when errors are present in the video encoding system. Conventional video encoding techniques are designed to handle worst case error rates by increasing gate sizes and/or increasing the operating voltage of the system. Such designs have error-rate limits, and when these limits are reached, the systems tend to fail catastrophically, resulting in an unrecoverable signal. The proposed design allows for single upset events to translate into single transient artifacts in a decoded video sequence. [ABSTRACT FROM PUBLISHER]

Light field (LF) technology has been popularly adopted by a wide range of conventional industries. However, one problem when dealing with LFs is the sheer size of data volume. There have been many multi-view video coding (MVC)-based LF video coding methods reported in the literature, aiming at finding the best prediction structure for LF video coding. It is clear that the number of possible prediction structures is unlimited, and it is also observed that the coding bit-rate can be reduced by increasing the number of bi-directionally encoded views in the prediction structure. However, none work has been conducted to analyze the relationship of the prediction structure with its coding performance. In light of this observation, we first design a new LF-MVC prediction structure by extending the inter-view prediction into a two-directional parallel structure. Analytical models for source coding rate and encoding time are developed to analyze their relationships with the prediction structure, and are proven to be well-matched to our experimental results. Experimental evaluation of two LF video sequences demonstrates that the proposed LF-MVC prediction structure can achieve a factor of 26% bit-rate reduction against the conventional MVC prediction structure for an LF video with $5\times 5$ views, and a further 34% bit-rate reduction for an LF video with a larger $10\times 10$ views. Compared with the state-of-the-art MVC-based LF video coding prediction structures in the literature, LF-MVC can achieve the best coding performance, and with its high encoding efficiency, is well suited for deployment in practical LF-based 3D systems. [ABSTRACT FROM PUBLISHER]

With the advance of coding and network technology, the multiview video (MVV) system has received growing attention in the 3-D TV market since it can provide consumers with more realistic scenes from different viewpoints. In the MVV system, each video sequence consists of one color-map video and one depth video to be encoded and the depth-image-based rendering technique is often used to create/synthesize virtual views for different viewpoints. To attain the same quality of virtual views, encoding depth videos in a fast and bitrate-saving manner is crucial. Based on the depth no-synthesis-error model developed by Zhao et al., this paper proposes a new intra-/inter-prediction scheme first and then a fast quadtree structure determination scheme for encoding depth videos in 3-D High Efficiency Video Coding (3D-HEVC). The first proposed scheme has bitrate-saving merit because for each coding unit, each depth value is modified to approach the predicted intra-/inter-depth value as closely as possible, but without causing any synthesis errors in the resultant virtual views. The second proposed scheme has the merit of low computational cost because it can terminate the quadtree structure determination for coding tree units as early as possible. Based on four typical test video sequences, the empirical results demonstrate that on average, our proposed encoding method for depth videos has 16.3% bitrate and 13.8% encoding-time improvement ratios when compared with the traditional method in 3D-HEVC test model, while preserving the quality of the rendered virtual views. In addition, our proposed method outperforms that of Lee et al. in terms of the required bitrate and encoding time. [ABSTRACT FROM AUTHOR]

Video content distributors, codec developers and researchers in related fields often rely on subjective assessments to ensure that their video processing procedures result in satisfactory quality. The current 10 s recommendation for the length of test sequences in subjective video quality assessment, however, has recently been questioned. Not only do sequences of this length depart from modern cinematic shooting styles, the use of shorter sequences would also enable substantial efficiency improvements to the data collection process. Our previous work, using a double-stimulus methodology, indicated that shortening test sequences had a limited impact upon rating behaviour. Here, using a larger database and additional opinion score measures, we also explore the same effect within the popular single-stimulus approach. Two groups of viewers assessed reference and distorted videos ranging in length from 1.5 s to 10 s. Analyses confirmed our previous findings using the DSCQS paradigm, and were replicated when using a similar single-stimulus paradigm: while viewers' DMOS for 1.5 s videos was significantly lower than for 10 s, no significant variation was found between the groups of 10 s, 7 s and 5 s videos. Together with our previous research, these data lead us to recommend the use of 5 s, temporally-consistent video clips in quality assessment studies that employ either DSCQS or its single-stimulus variant. The extension of our recommendation to further methodologies is also discussed. [ABSTRACT FROM AUTHOR]

Multi-view video coding (MVC) adopts variable size mode decision to achieve high coding efficiency. However, its high computational complexity is a bottleneck of enabling MVC into practical real-time applications. In this paper, an early termination strategy is proposed for DIRECT mode decision of MVC by exploiting mode homogeneity and rate distortion (RD) cost correlation. By comparing the RD cost between DIRECT mode and Inter $16\times 16$ mode, an adaptive threshold is defined based on the MB’s mode homogeneity and RD cost so as to early terminate the remaining inter and intra modes. Experimental results show that compared with the original JMVC model, the proposed approach can reduce the total encoding time from 65.08% to 91.45% (80.43% on average). Meanwhile, the Bjontegaard delta peak signal-to-noise ratio only decreases 0.031 dB and Bjontegaard delta bit rate increases 0.97% on average, which is a negligible loss of coding efficiency and superior to the performance of state-of-the-art methods. [ABSTRACT FROM AUTHOR]

High‐efficiency video coding (HEVC) is based on integer discrete cosine transforms (DCTs) of size 4 × 4, 8 × 8, 16 × 16 and 32 × 32 whose elements are coded on 8 bits. However, the algorithm requires that the output length at each processing stage should never exceed 16 bits. The conventional solution is the truncation of the least significant bits which leads to erroneous results and a waste of resources. In this Letter, different DCT kernels with reduced elements length are proposed. They are evaluated for compression efficiency and hardware implementation competence. An implementation on the Xilinx FPGA virtex‐6 circuit has given a maximal operating frequency increase of 4.81 and 93.41% for the DCT‐II 4 × 4 and the discrete sine transform, while reducing the energy consumption by 10.64 and 30.77% at 100 MHz, respectively. Using the HM 16.3 HEVC model and video sequences of different resolutions, the results show a quality degradation of 0.01 dB for a bit rate increase of 0.19% compared to the reference cores. [ABSTRACT FROM AUTHOR]

This paper presents a novel rate control framework for H.264/Advanced Video Coding-based video coding that improves the preservation of gradient-based features like scale-invariant feature transform or speeded up robust feature compared with the default rate control algorithm in the JM reference software. First, a criterion (matching score) for feature preservation on the basis of the bag-of-features concept is proposed. Then, the matching scores are collected as a function of the quantization parameters and analyzed for different feature types. With this analysis, macroblocks are categorized into different groups before encoding. Our rate control algorithm assigns different quantization parameters to each group according to the importance of the group for feature extraction. The experimental results show that our rate control algorithm achieves the desired target bit rate, and more features are preserved compared with videos encoded using the default rate control. The proposed approach not only improves feature preservation, but also leads to a noticeable performance improvement in a real image retrieval system. The rate control framework proposed in this paper is fully standard compatible. [ABSTRACT FROM PUBLISHER]

Recent works have shown how multiple description coding proves to be an effective solution for multimedia streaming over peer-to-peer (P2P) and content delivery networks (CDNs). However, the presence of losses and congestions throughout the network affects the visual quality of the reconstructed sequence at the end terminal. These inconveniences can be mitigated by specifying different levels of Quality of Service, but an optimal packet classification is hard to obtain since P2P and CDN protocols operate at higher protocol layers (ignoring network conditions of the lowest stages), the network can be quite distributed and little information can be available regarding other network segments involved in the transmission. The peculiarities of the transmission scenario require a distributed and robust packet classification strategy that grants both intra-stream and inter-stream diversities among the loss patterns for the different streams. The classification approach presented here is characterized via a noncooperative game, where the different uploading nodes are players/descriptions competing for the allocation of the available network resources. Each player may switch from a selfish strategy to a more cooperative strategy according to its convenience (opportunistic players). Experimental results show that the proposed solution proves to be quite effective under different network scenarios. [ABSTRACT FROM AUTHOR]

Generally, H.264/AVC video coding standard with hierarchical bipredictive picture (HBP) structure outperforms the classical prediction structures such as “IPPP \(\ldots \) ” and “IBBP \(\ldots \) ” through better exploitation of data correlation using reference frames and unequal quantization setting among frames. However, multiple reference frames (MRFs) techniques are not fully exploited in the HBP scheme because of the computational requirement for B-frames, unavailability of adjacent reference frames, and with no explicit sorting of the reference frames for foreground or background being used. To exploit MRFs fully and explicitly in background referencing, we observe that not a single frame of a video is appropriate to be the reference frame as no one covers adequate background of a video. To overcome the problems, we propose a new coding scheme with the HBP, which uses the most common frame in scene (McFIS), generated by background modeling, as a long-term reference (LTR) frame for the third unipredictive reference frame, so that foreground and background areas are expected to be referenced from the two frames in the HBP structure and the McFIS, respectively. There are two approaches to generate McFIS under the proposed methodology. In the first approach, we generate a McFIS using a number of original frames of a scene in a video and then encode it as an I-frame with a higher quality. For the rest of the scene, this generated I-frame is used as an LTR frame. In the second approach, we generate an McFIS from the decoded frames and then use it as an LTR frame, without the need to encode the McFIS. The first and the second approaches are suitable for a video with static background and dynamic background, respectively. In general, the second approach requires more computational time than that of the the first approach. The experiments confirm that the proposed scheme outperforms three state-of-the-art algorithms by improving the image quality significantly with reduced computational time. [ABSTRACT FROM AUTHOR]

The high efficiency video coding (HEVC) is the latest video coding standard, which adopts the quadtree structure based coding tree unit (CTU) to improve the coding efficiency. In the HEVC encoding process, the CTU is recursively split into the \(8\times 8\) size coding units (CUs) from the \(64\times 64\) size CU. Along with the increased number of the sizes of the CUs, the number of coding modes has been greatly increased, which results in high computational complexity in the HEVC encoder. In this paper, we propose an early MERGE mode decision algorithm to reduce the computational complexity of the HEVC encoder. Firstly, based on the all-zero block (AZB) and the motion estimation (ME) information of the INTER 2N \(\times\) 2N mode, an early MERGE mode decision is proposed for the root CUs (i.e., \(64\times 64\) size CUs). Then, an early MERGE mode decision is proposed for the children CUs (i.e., \(32\times 32\) , \(16\times 16\) , and \(8\times 8\) size CUs) by considering the mode selection correlation between the root CU and the children CUs. To maximize the computational complexity reduction, when the root CUs are encoded in the non-MERGE modes, the AZB and the ME information are also used for early termination of the children CUs. Experimental results demonstrate that compared to the state-of-the-art published method, the proposed algorithm can achieve about 35% encoding time on average saving while the rate distortion performance degradation is negligible. [ABSTRACT FROM PUBLISHER]

Video compression systems use prediction to reduce redundancies present in video sequences along the temporal and spatial dimensions. Standard video coding systems use either temporal or spatial prediction on a per block basis. If temporal prediction is used, spatial information is ignored. If spatial prediction is used, temporal information is ignored. This may be a computationally efficient approach, but it does not effectively combine temporal and spatial information. In this letter, we provide a framework where available temporal and spatial information can be combined effectively to perform joint spatial and temporal prediction in video coding. Experimental results obtained from one sample realization of this framework show its potential. [ABSTRACT FROM AUTHOR]

High Efficiency Video Coding (HEVC) uses a very flexible tree structure to organize coding units, which leads to a superior coding efficiency compared with previous video coding standards. However, such a flexible coding unit tree structure also places a great challenge for encoders. In order to fully exploit the coding efficiency brought by this structure, huge amount of computational complexity is needed for an encoder to decide the optimal coding unit tree for each image block. One way to achieve this is to use parallel computing enabled by many-core processors. In this paper, we analyze the challenge to use many-core processors to make coding unit tree decision. Through in-depth understanding of the dependency among different coding units, we propose a parallel framework to decide coding unit trees. Experimental results show that, on the Tile64 platform, our proposed method achieves averagely more than 11 and 16 times speedup for 1920x1080 and 2560x1600 video sequences, respectively, without any coding efficiency degradation. [ABSTRACT FROM AUTHOR]

In asymmetric resolution stereoscopic video coding (ARSVC), to reduce the bitrate required for bandwidth-limited channels, each downsampled right-view frame is a quarter the size of the corresponding left-view frame and will be upsampled to the original size by the decoder. In this paper, two upsampling methods for ARSVC are proposed. The first proposed method integrates the traditional Wiener filter-based method and the interview prediction scheme by incorporating the information from the similarity between the left-view and right-view frames. By compensating for the prediction errors, the second proposed method further improves the quality of the upsampled images obtained by the first proposed method, especially in the sequences with heavy irregular textures. [ABSTRACT FROM PUBLISHER]

This paper compares the quality of AVC/H.264 and HEVC encoded video in low bandwidth mobile environments. In this study, the focus within the mobile environment is smart phones. The key characteristics of a smart phone are smaller screen size, which is usually 3.5 inches diagonal to 5.0 inches diagonal for high end smart phones and typical cellular network bandwidth, which is 3G or faster. Subjective evaluations were conducted to evaluate the user experience on a mobile device with a small screen size and video coded at 200 and 400 Kbps. The studies showed compelling evidence that a user's experience in low bandwidth mobile environments is very similar between HEVC and AVC/H.264. The results suggest the benefits of HEVC over AVC/H.264 in a mobile environment with lower video bitrates and resolutions are not as clear. [ABSTRACT FROM AUTHOR]

We consider a point-to-point delay-stringent video communication system. To reduce the fluctuation of the coded data packet sizes for efficient wireless transmission, we adopt the intra-refreshment and adaptive slice partitioning for video encoding. We model the delay-constrained video transmission with the proposed video coding and slicing scheme using an MDP model, and compute the optimal coding and modulation scheme corresponding to each channel state and the encoding and transmission delay using the value iteration algorithm. Extensive simulations based on real videos show that the MDP-based transmission scheme significantly reduces the transmission cost and enhances the quality of the reconstructed video sequences compared with the non-adaptive transmission scheme, which transmits constant number of transmitted symbols for all slices. Simulation results also show that the intra-refreshment and adaptive slice partitioning significantly enhances the quality of the reconstructed video sequences compared with the video coding without intra-refreshment. [ABSTRACT FROM PUBLISHER]