Search

In this paper, a content adaptive coding method for plenoptic video is proposed to reduce both of the spatial and temporal redundancy. Based on the spatial correlations among subapertures, the plenoptic video is divided into two subaperture groups: the central view videos and the residual videos. Coding methods, multiview coding method based on spatial correlation(S-MVC) and multiview coding method based on temporal correlation(T-MVC), are adopted for residual videos according to correlation and residual energy analysis, while the central view videos are always encoded by S-MVC. Correlation analysis is performed first, and residual energy analysis is further performed if necessary. Coding the central view videos and the residual videos separately reduces bitstream, choosing different encoding methods for residual videos makes full use of the characteristics of different video content. Both the central view videos and the residual videos are compressed by MV-HEVC. The experimental results demonstrate that the proposed method outperforms a pseudo-sequence-based MVC method for plenoptic video by an average of 25.06% bitrate saving.

Content-adaptive steganography intends to hide data in the complex texture content of the image. Recently, some secure steganography methods have been proposed to identify the textural complexity of an image. However, most of the techniques do not take into account the information of pixel variation around the central pixel in all possible directions and therefore they are unable to accurately analyse the texture complexity. This work offers a quality-enhanced and secure method of content-adaptive image steganography. The proposed method is divided into three sequential steps: image segmentation, pixel complexity identification, and data embedding. An input cover image is initially divided into small local regions and the pixel-complexity is identified based on the proposed Complex Block Prior (CBP) criterion. In a local block, a high pass filter (HPF) bank is applied and eight residual responses are obtained. Following the CBP criterion, a complexity level out of nine levels is assigned to an individualized pixel block. The pixels are then arranged in the priority of complexity from highest to lowest. Data embedding for the corresponding complexity level then takes place using the proposed adaptive embedding algorithm. Experimental results verify the preservation of visual quality of stego images produced by the proposed method. Three image datasets: Standard test images, BOWS2 and BOSS-base are used for the experimentation and comparison with prior state-of-art methods. Highest values of the IQ (image quality) parameters e.g., SSIM and WPSNR show the effectiveness of the proposed method.

Video codecs have several dozen parameters that subtly affect the encoding rate, quality and size of the compressed video. Codec developers, as a rule, provide standard presets that on average yield acceptable performance for all videos, but for a given video, certain parameters may yield more efficient encoding. In this paper, we propose a new approach to predicting video codec presets to increase compression efficiency. Our effort involved collecting a new representative video-sequence dataset from Vimeo.com. An experimental evaluation showed relative bitrate decreases of 17.8% and 7.9%, respectively for the x264 and x265 codecs with standard options, all while maintaining quality and speed. Comparison with other methods revealed significantly faster automatic preset selection with a comparable improvement in results. Finally, our proposed content-adaptive method predicts presets that archive better performance than codec-developer presets from MSU Codec Comparison 2020 [1].

This "success story" panel illustrates how next generation video applications are enabled on PCs today using consumer integrated and discrete GPUs launched in 2020. Intel's XeLP graphics technology with dedicated media hardware powers the integrated graphics in Intel's latest client processor, Tiger Lake, and Intel's first entry-level mainstream discrete graphics card, DG1. XeLP graphics in Tiger Lake and DG1 has democratized access to high performance implementations of the latest emerging video codec standards. Three topics will be covered: (i) 8K HEVC/AV1 Playback with Content Adaptive Power Optimization; (ii) Ludicrous Speed HEVC Encoding with Integrated + Discrete GPU; and (iii) MPEG Immersive Video (MIV) Playback on DG1.

With the advent of modern smartphones, AR, VR services and advancement in display resolution of mobile devices coupled with real-time streaming services, the demand for highresolution video has boomed. To fulfill this requirement, a variety of Adaptive Bit-Rate Streaming methods for Video-on-Demand applications are employed. However, the use of multi-pass encoding in the aforementioned methods renders them obsolete when it comes to real-time video streaming due to latency restrictions. In this work, we bypass the conventional multiple-encoding used in Video-on-Demand applications and present a novel machinelearning-based approach that estimates the optimal video resolution for a given content at a particular bit-rate for ultra low latency applications. A new feature that captures temporal as well as spatial correlation in video sequence has been used to train the Deep Neural Network (DNN) model. A python-based testbed is designed to evaluate the proposed scheme. Experiment results corroborate the viability and effectiveness of the proposed method for real-time mobile video streaming applications.

In this paper, we introduce a modification of the popular U-Net neural network architecture for medical image segmentation. Our proposed architecture replaces the concatenation operations in the traditional U-Net's skip connections with content-adaptive convolution, thereby significantly reducing the number of parameters of the network. Our experiments on two segmentation tasks - cell nuclei segmentation, and pneumo-thorax segmentation - demonstrated that the modified architecture achieves higher segmentation accuracy compared to the original U-net.

Rate Control (RC) plays an important role in video encoding. Traditional solutions are using fixed rate or fixed quantization parameters as the unified rate-control targets for all videos in one given video application. However, unified ratecontrol targets tend to have some bad encoding cases because of applying wrong rate for the video content. In this paper, we propose one content-adaptive rate control solution. We employ one neural-network based model which can end-to-end learn the optimal rate-control target appropriate to the content characteristics. The experimental results show that the proposed model can predict the optimal rate-factor value with the accuracy up to 77.637%. With this model, the proposed video-encoding method can significantly decrease the encoding quality fluctuation.

This paper proposes a content adaptive fast CU size selection algorithm for HEVC intra-prediction using weighted support vector machines. The proposed algorithm demonstrates an average encoding time reduction of 52.38% with 1.19% average BDBR increase compared to HM16.1 reference encoder.

This paper proposes a framework of content-adaptive image compressed sensing using deep learning, which analyzes the image content and adaptively allocates samples for different image patches accordingly. Experimental results demonstrate that the proposed framework outperforms the state-of-the-arts both in subjective and objective quality, especially at low sampling rates. For example, when the sampling rate is 0.1, 1–6 dB improvement in peak signal to noise ratio (PSNR) is observed. Moreover, the proposed work reconstructs images with more details and less image blocking effects, leading to apparent visual improvement.

Aiming at improved rate-distortion (R-D) performance, this paper presents a machine-learning based solution for the run-time video resolution adaptation problem. The proposed approachutilizes neural networks that leverage a complexity feature extracted from the video frames topredict a quantization parameter (QP) for downscaled video targeting the same bitrate as the native video. The peak signal to noise ratio (PSNR) is also predicted for both the native and downscaled resolutions, and the one that leads to the highest PSNR is selected. Experimental results show that \quad the proposed adaptive approach achieves significant improvements in R-D performance compared to using a fixed resolution.

In this paper, a novel hybrid cubemap projection (HCP) is proposed to improve the 360-degree video coding efficiency. HCP allows adaptive sampling adjustments in the horizontal and vertical directions within each cube face. HCP parameters of each cube face can be adjusted based on the input 360-degree video content characteristics for a better sampling efficiency. The HCP parameters can be updated periodically to adapt to temporal content variation. An efficient HCP parameter estimation algorithm is proposed to reduce the computational complexity of parameter estimation. Experimental results demonstrate that HCP format achieves on average luma (Y) BD-rate reduction of 11.51%, 8.0%, and 0.54% compared to equirectangular projection format, cubemap projection format, and adjusted cubemap projection format, respectively, in terms of end-to-end WS-PSNR.

In high-efficiency coding (HEVC), the random-access structure (RAS) is employed due to its high coding efficiency and “random-access” performance. Pictures in RAS are assigned to different temporal layers. And how to select the QP for each temporal layer in quantization parameter cascading (QPC) technique is critical for improving the coding efficiency of RAS. In order to further improve the coding efficiency of RAS, a QPC technique considering the video content characteristics (denoted as VC-QPC for short) is proposed. In VC-QPC, motion and texture complexities of a video are used for predicting the optimal quantization parameter values of temporal layers. Compared with the method in the test model of HEVC, i.e., HM14.0, the BD-rate of proposed VC-QPC is −5.60% for RAS.

Near-eye multi-focal (light field) displays are known to be more advantageous than conventional stereoscopic 3D displays, because they can alleviate the vergence-accommodation conflict. While stereoscopic 3D displays are restricted to projecting images on a single focal plane, near-eye light field displays can form multiple focal planes and volumetrically render 3D data. Most existing near-eye light field displays use simplistic, uniformly spaced focal plane configuration (in dioptric space). In this paper, we present a novel technique that optimizes the focal plane configuration based on characteristics of the content to be rendered. We show that this technique can significantly improve the perceived visual quality of content visualized on such displays.

This paper provides a motion-based content-adaptive depth map enhancement algorithm to enhance the quality of the depth map and reduce the artifacts in the synthesized views. The proposed algorithm extracts depth cues from the motion distribution at the specific scenario of camera movement to align the distribution of depth and motion. In real world scenarios, when the camera is panning in horizontal direction, the nearer distance between the object and the camera, the larger motion will be, and vice versa; therefore, we could interpret the depth from motion in this. Moreover, in the scenario of fixed camera, the depth cue from motion could be derived in the same approach, and the depth variation within one moving object shall be small. Hence, the depth values of moving object should not be rapidly changing. In addition, this paper also employs the bi-directional motion-compensated infinite impulse response low-pass filter to stabilize the consistency of depth maps over time. As a consequence, the algorithm so introduced not only aligns the depth map to depth cues from motion but also enhance stability and consistency of depth maps in the spatial-temporal domain. Experiment results via enhanced depth maps show that the synthesized results would be better in both objective and subjective measurement in comparison with the results using original depth maps and the state-of-the-art depth enhancement algorithms.

In this abstract, we present preliminary evaluation of a full 4D reconstruction method aimed at cardiac gated emission tomography. The proposed method uses content adaptive mesh model (CAMM) to represent the 3D volumetric data. The nodes of the CAMM are then moved from a gate to gate, according to the motion obtained from a motion estimation algorithm, to form 4D deformable content adaptive mesh model (DCAMM).

This paper proposes an efficient content adaptive screen image scaling scheme for the real-time screen applications like remote desktop and screen sharing. In the proposed screen scaling scheme, a screen content classification step is first introduced to classify the screen image into text and pictorial regions. Afterward, we propose an adaptive shift linear interpolation algorithm to predict the new pixel values with the shift offset adapted to the content type of each pixel. The shift offset for each screen content type is offline optimized by minimizing the theoretical interpolation error based on the training samples respectively. The proposed content adaptive screen image scaling scheme can achieve good visual quality and also keep the low complexity for real-time applications., ICIP 2013

Integrating efficient visual perceptual cues into standardized video coding framework can improve performance significantly. In this paper we propose to enhance AVS encoder by using the latest just noticeable distortion (JND) model to adjust DCT coefficients of prediction residues in a content adaptive way. To better modeling JND profile in AVS integer DCT domain, we further derive the JND mapping from the classical DCT domain to AVS Integer DCT domain. The experiment shows that the proposed algorithm can reduce the bitrate by about 13% on average, compared to the AVS standard encoder at similar visual quality.

In this paper, a content adaptive in-loop depth map filter is proposed for HEVC based 3DV video coding to improve the quality of the synthesized views. The proposed depth map filtering scheme is block based by reusing the TU (transform unit) split structure in HEVC. Firstly, TU are classified into 3 categories: flat, directional, and textureless, according to the characteristics of the depth map. Then four kinds of filters are designed by considering the characteristics of the current TU and neighbor TUs jointly. The proposed scheme is incorporated into HTM3.0- the reference software of HEVC based 3D video coding. The experimental results show both subjective and objective quality improvement. The average BD-rate reduction is 0.65% for synthesis views in HD test sequences, and the maximum BD-rate reduction is up to 1.6%.

This paper proposes a content-adaptive blending method, smart compositing, for displaying two overlapped video frames on the same screen while preserving the readability of both. A pixel-wise adaptive blending factor map is generated according to the edge and saturation information of the content of only the overlay frame. Using this blending factor map, regions of the overlay frame with edges or saturated color are assigned to be more opaque and the remaining regions are assigned to be more transparent. A halo is also created around the edges of the overlay content which enhances the edges and disambiguates them from the underlying frame. The proposed method is suitable for overlaying many different types of content (e.g. drawings, slides, texts, and pictures) and does not require any information (e.g., an opacity mask) from the application which generates the content. This method has low computational complexity and has been implemented in real-time.

This paper presents a new self-adapting and content-sensitive optimization technique for the H.264/AVC in-loop deblocking filter [1], focussing on the visual enhancement of the perceived reconstruction quality. Performed frame-wisely at the encoder side, the proposed algorithm first identifies visually important image regions in the currently decoded and blocky frame, including natural and artificial edge areas, and then optimizes the filter inherent threshold decisions, finding the best trade-off between preserving natural and suppressing artificial edges. As fast optimization criterion, the low complex Edge-PSNR [2] is employed, which has proved a very good congruence to the human visual quality sensation, much better than standard PSNR. In this work, the optimization behavior of all thresholds is analyzed, showing their mostly good-natured curve shape for convex optimization, but high content dependency of the optimal values. The in-loop coding results for AVC evidence the approach's high capability for visual improvement, whose general design is easily portable to other AVC deblocking based architectures like HEVC.

This paper presents a content-adaptive coding scheme for immersive networked experience of sports events, in particular, soccer games. We assume that future sports events are captured by an array of fixed high-definition cameras which provide multiview image sequences for a free-viewpoint immersive networked experience in a home environment. We discuss a content-adaptive coding scheme for image sequences that exploits properties of such sequences and that permits efficient user interactions. In this work, we construct a rate distortion model for an image sequence to obtain the optimal bitrate allocation among static and dynamic content items. The optimal bitrate allocation results in a rate distortion performance of the coding scheme that outperforms that of conventional H.264/AVC coding significantly.

In order to restore the information loss in video communication, a content adaptive intra error concealment method is proposed in this paper. Different images and different packet loss rates of the video sequences are used in the experiment. The experimental results show that the improved method has the better subjective quality and the higher PSNR than the bilinear interpolation and the semi-adaptive error concealment method.

In prior work, we proposed an unequal error protection algorithm based on Wyner-Ziv coding for error resilient video transmission. In subsequentwork it was demonstrated that using either of content adaptive unequal error protection or feedback aided unequal error protection individually improved error resilience performance. In the current paper we propose to combine the use of a content adaptive function, implemented at the encoder, with the channel loss feedback provided by the decoder. The experimental results demonstrate improved rate distortion performance.

Realistic finite element (FE) head models for neuro-electromagnetic imaging are getting more attention due to their analytic advantages over conventional models. To improve the numerical efficiency, we have previously developed a novel mesh generation scheme that produces FE head models automatically that are content-adaptive to given MR images. MRI content-adaptive FE meshes (cMeshes) represent the electrically conducting domain more effectively with less number of nodes and elements, thus lessen the computational loads. In general, the cMesh generation is affected by the selection of feature maps derived from MRI. In this study, we have tested the effects of various feature maps on the generation of cMesh FE head models. Also we have evaluated the quality of cMesh FE head models to check their suitability for neuro-electromagnetic imaging using EEG and MEG. The results suggest that the cMesh FE head models with properly selected feature maps do show acceptable quality to be used in neuro-electromagnetic imaging.

Most artifact reduction techniques blur the details of the images while removing various coding artifacts. In this paper, we propose a novel and explicit approach for classifying blocks into detailed regions, intermediate regions and smooth regions. The classification is based on the information content of the underlying region, which is quantized by local entropy calculated on the PDF of the pixel intensity distribution. The entropy is used as an indicator of how much smoothing is needed for a certain region. Experimental results show that our proposed algorithm can preserve the details and reduce coding artifacts better than more expensive state of the art techniques.

This paper describes a new method of watermarking based on visual contents and using four spaces insertion of insertion to take advantage of properties of each of them. This is realized initially by segmenting image in unrefined regions and then adapting every space of insertion to these regions in order to maximize the invisibility of the marked image. Experimental results show that the watermark is robust to several distortions including geometric alterations, JPEG coding at different qualities, STIRMA4RK attack, filtering and cropping.

Multiple reference frame selection adopted by the state-of-art H.264 video compression standard offers substantial performance gain. The temporal search range control, as a consequence, is crucial for maintaining the coding performance with minimum complexity. In this paper, we investigate the relationships between the reference frame buffer utilization and the optimal search range. A content-adaptive algorithm is proposed to control the search range dynamically during the encoding process. Experimental results show that our algorithm can rapidly adapt to the video characteristics and effectively reduce the complexity with negligible coding performance penalty.

In this paper, we propose a low-complexity content-adaptive background skipping scheme for region-of-interest (ROI) video coding in real-time wireless video telephony applications. Based on the real-time content information of the current and previous frames, such as foreground shape deformation, foreground and background motion, and background texture complexity, the proposed algorithm dynamically decides whether to skip non-ROI macroblocks of current frame and reallocate saved bits to ROI and coded non-ROI. In this work, we proposed and compared three bit reallocation strategies, and the one predicting the future events using a Bayesian model is found the most efficient. Experimental results indicate that the proposed scheme significantly outperforms other methods by up to 2.5 dB

We propose a content-adaptive interpolation algorithm for digital images. The proposed algorithm makes use of a non-linear spatial varying filter (SVF), which is simple but efficient in enlarging images. First, edge orientation statistics in an image are exploited, and then the information is used to determine suitable weights of neighboring pixels when interpolating the current pixel. Using the proposed approach, we can generate pixel values to replace missing ones such that sharp edges can be accurately reconstructed. Simulation results show that the proposed algorithm obtains better results in both subjective and objective quality measurements over existing methods.

Conventional rate control schemes focus on the adjustment of quantization values to retain a certain buffer level, and buffer adaptive frame skippings are used to avoid buffer overflow at low bit-rates. In this paper, a content adaptive frame-skipping scheme is proposed to achieve the balance between spatial and temporal quality. The occurrence of frame skipping is jointly dependent on the temporal and spatial contents of the video, and on the fullness of the buffer. This helps to achieve a balanced spatial and temporal quality and to enhance the overall perceptual quality. Experimental results show that the new scheme is simple but very effective, with large average PSNR gains and consistently improved visual quality, and the improvement in perceptual quality is much significant than that in average PSNR's.

This paper presents a memory-saving long-term prediction algorithm. It takes advantage of the better image quality possible with this technique, but is characterized by an extreme reduction in the amount of memory space required. In reference picture selection mode (H.263 standard, Annex N), coder and decoder use not only the previous picture for prediction, but also information from less recent pictures. Contrary to Annex N, in the method that we propose only parts, e.g. individual blocks or areas of past pictures, are stored in a reference buffer. We present two new concepts, both for selecting picture areas for storing and for deleting superfluous blocks from the long-term memory. By suitable selection of the picture areas, our algorithm achieves a similar picture quality to H.263+, Annex N, but uses only a fraction of the picture buffer. In this way it is possible to address around 50 earlier pictures with the same amount of memory required for only 5 pictures. Moreover, our approach requires no signaling and produces no delay.

Unconstrained motion compensated temporal filtering (UMCTF) is a very general and flexible framework for temporal filtering. It allows the selection of many different filters as well as decomposition structures to allow easy adaptation to video content, bandwidth variations, complexity requirements, and in conjunction with embedded coding can provide spatio-temporal-SNR scalability. In this paper we demonstrate the content-adaptive filter selection provided within the UMCTF framework. We show improvements in coding efficiency as well as in decoded visual quality using content-adaptive filters, at different granularities.

We propose the use of a content-adaptive volumetric mesh model for fully three-dimensional (3D) tomographic image reconstruction. In the proposed framework, the image to be reconstructed is first modeled by an efficient mesh representation. The image is then obtained through estimation of the nodal values from the measured data. The use of a mesh representation can alleviate the ill-posed nature of the reconstruction problem, thereby leading to improved quality in the reconstructed images. In addition, it reduces the data storage requirement, resulting in efficient algorithms. The proposed methods are tested using gated cardiac-perfusion images. Initial results demonstrate that the proposed approach achieves good performance when compared to several commonly used methods for image reconstruction, and produces results very rapidly.