Search

Recently, most background modeling (BM) methods claim to achieve real-time efficiency for low-resolution and standard-definition surveillance videos. With the increasing resolutions of surveillance cameras, full high-definition (full HD) surveillance videos have become the main trend and thus processing high-resolution videos becomes a novel issue in intelligent video surveillance. In this article, we propose a novel content-adaptive resizing framework (CARF) to boost the computation speed of BM methods in high-resolution surveillance videos. For each frame, we apply superpixels to separate the content of the frame to homogeneous and boundary sets. Two novel downsampling and upsampling layers based on the homogeneous and boundary sets are proposed. The front one downsamples high-resolution frames to low-resolution frames for obtaining efficient foreground segmentation results based on BM methods. The later one upsamples the low-resolution foreground segmentation results to the original resolution frames based on the superpixels. By simultaneously coupling both layers, experimental results show that the proposed method can achieve better quantitative and qualitative results compared with state-of-the-art methods. Moreover, the computation speed of the proposed method without GPU accelerations is also significantly faster than that of the state-of-the-art methods. The source code is available at https://github.com/nchucvml/CARF.

Digital imaging has significantly influenced the outcome of research in various disciplines. For example, artificial intelligence and robotics, biometric security, multimedia and image processing, etc. Technically, image processing and the Human Visual System (HVS) relies heavily on image enhancement to improve the content of the image. One of the biggest challenges in image processing is detail enhancement due to halo artefacts and gradient inversion artefacts at edges. It has been used to enhance the visual quality of an image. Most algorithms that used to enhance the detail of an image essentially depend on edge-preserving decomposition techniques. in general, the image consist of two major elements are a base layer and a detail layer, which extracted by edge-preserving decomposition algorithms. The detail layer is enhanced to improve the details of the generated image. we propose in this paper, a new model to preserve the sharp edges and achieve better visual quality than the existing norm-based algorithm to enhance the details of the image. Experiments show that the proposed method reduces the distortion at the edges. It improves the details of the generated image significantly.

We address the problem of efficient content-adaptive superpixel segmentation. Instead of adapting the size and/or amount of superpixels to the image content, we propose a warpingtransform that makes the image content more suitable to be segmented into regular superpixels. Regular superpixels in the warped image induce content-adaptive superpixels in the original image with improved segmentation accuracy. To efficiently compute the warping transform, we develop an iterative coarse-to-fine optimization procedure and employ a parallelization strategy allowing for a speedy GPU-based implementation. This solution works as a simple ‘add-on’ framework over an underlying segmentation algorithm and requires no additional parameters. Compared to the state-of-the-art methods, our approach provides competitive quality results and achieves a better time-accuracy trade-off. We further demonstrate the effectiveness of our method with an application to disparity estimation.

Objects at different spatial positions in an image exhibit different scales. Adaptive receptive fields are expected to capture suitable ranges of context for accurate pixel level semantic prediction. Recently, atrous convolution with different dilation rates has been used to generate features of multi-scales through several branches which are then fused for prediction. However, there is a lack of explicit interaction among the branches of different scales to adaptively make full use of the contexts. In this paper, we propose a Content-Adaptive Scale Interaction Network (CASINet) to exploit the multi-scale features for scene parsing. We build CASINet based on the classic Atrous Spatial Pyramid Pooling (ASPP) module, followed by a proposed contextual scale interaction (CSI) module, and a scale adaptation (SA) module. Specifically, in the CSI module, for each spatial position of some scale, instead of being limited by a fixed set of convolutional filters that are shared across different spatial positions for feature learning, we promote the adaptivity of the convolutional filters to spatial positions. We achieve this by the context interaction among the features of different scales. The SA module explicitly and softly selects the suitable scale for each spatial position and each channel. Ablation studies demonstrate the effectiveness of the proposed modules. We achieve state-of-the-art performance on three scene parsing benchmarks Cityscapes, ADE20K and LIP.

In this paper, a content adaptive coding method for plenoptic video is proposed to reduce both of the spatial and temporal redundancy. Based on the spatial correlations among subapertures, the plenoptic video is divided into two subaperture groups: the central view videos and the residual videos. Coding methods, multiview coding method based on spatial correlation(S-MVC) and multiview coding method based on temporal correlation(T-MVC), are adopted for residual videos according to correlation and residual energy analysis, while the central view videos are always encoded by S-MVC. Correlation analysis is performed first, and residual energy analysis is further performed if necessary. Coding the central view videos and the residual videos separately reduces bitstream, choosing different encoding methods for residual videos makes full use of the characteristics of different video content. Both the central view videos and the residual videos are compressed by MV-HEVC. The experimental results demonstrate that the proposed method outperforms a pseudo-sequence-based MVC method for plenoptic video by an average of 25.06% bitrate saving.

Content-adaptive steganography intends to hide data in the complex texture content of the image. Recently, some secure steganography methods have been proposed to identify the textural complexity of an image. However, most of the techniques do not take into account the information of pixel variation around the central pixel in all possible directions and therefore they are unable to accurately analyse the texture complexity. This work offers a quality-enhanced and secure method of content-adaptive image steganography. The proposed method is divided into three sequential steps: image segmentation, pixel complexity identification, and data embedding. An input cover image is initially divided into small local regions and the pixel-complexity is identified based on the proposed Complex Block Prior (CBP) criterion. In a local block, a high pass filter (HPF) bank is applied and eight residual responses are obtained. Following the CBP criterion, a complexity level out of nine levels is assigned to an individualized pixel block. The pixels are then arranged in the priority of complexity from highest to lowest. Data embedding for the corresponding complexity level then takes place using the proposed adaptive embedding algorithm. Experimental results verify the preservation of visual quality of stego images produced by the proposed method. Three image datasets: Standard test images, BOWS2 and BOSS-base are used for the experimentation and comparison with prior state-of-art methods. Highest values of the IQ (image quality) parameters e.g., SSIM and WPSNR show the effectiveness of the proposed method.

This paper presents a novel fast transcoding framework to efficiently bridge the state-of-art high efficiency video coding (HEVC) standard and its screen content coding (SCC) extension to support the bitstream compatibility over legacy HEVC devices. By exploiting the side information from the SCC bitstream, fast mode and partition decisions are made to accurately translate the novel SCC modes to conventional HEVC modes based on statistical mode mapping techniques. Compared with the full-decoding-full-encoding (FDFE) solution, the proposed framework achieves on average 51% and 82% complexity reductions with 0.57% Bjontegaard-delta rate (BD-Rate) loss and 9.74% BD-Rate gain under all-intra (AI) and low-delay (LD) configurations, respectively. Compared with the direct transcoding reusing intra mode and inter motion, the proposed mode mapping framework introduces additional 23% and 6% complexity reductions for AI and LD encoding configurations with 0.43% BD-Rate loss and 1.10% BD-Rate saving, respectively. The proposed solution is extended to support the single-input-multiple-output screen content adaptive streaming at the edge clouds, where an SCC bitstream coded in high quality is transcoded into multiple HEVC bitstreams in reduced qualities. Our proposed solution achieves on average 49% and 76% complexity reductions with 0.78% BD-Rate loss and 7.40% BD-Rate gain under AI and LD configurations, respectively.

Source camera identification is still a hard task in forensics community, especially for the case of the query images with small size. In this paper, we propose a solution to identify the source camera of the small-size images: content-adaptive fusion residual networks. According to the differences of the image contents, firstly, the images are divided into three subsets: saturation, smoothness and others. Then, we train three fusion residual networks for saturated images, smooth images, and others, separately, through transform learning. The fusion residual networks is formed with three paralleled residual networks and the difference of three residual networks lies in the convolutional kernel size of preprocessing layer. The features learned from the last residual blocks of three residual networks are fused and fed into softmax classifier. In particular, the residual networks is designed to learn better feature representation from the input data. The convolutional operation is added in preprocessing stage and three residual blocks are used. The experiment results show that the proposed method has satisfactory performances at three levels of source camera identification: brand level, model level, and device level.

Video codecs have several dozen parameters that subtly affect the encoding rate, quality and size of the compressed video. Codec developers, as a rule, provide standard presets that on average yield acceptable performance for all videos, but for a given video, certain parameters may yield more efficient encoding. In this paper, we propose a new approach to predicting video codec presets to increase compression efficiency. Our effort involved collecting a new representative video-sequence dataset from Vimeo.com. An experimental evaluation showed relative bitrate decreases of 17.8% and 7.9%, respectively for the x264 and x265 codecs with standard options, all while maintaining quality and speed. Comparison with other methods revealed significantly faster automatic preset selection with a comparable improvement in results. Finally, our proposed content-adaptive method predicts presets that archive better performance than codec-developer presets from MSU Codec Comparison 2020 [1].

This "success story" panel illustrates how next generation video applications are enabled on PCs today using consumer integrated and discrete GPUs launched in 2020. Intel's XeLP graphics technology with dedicated media hardware powers the integrated graphics in Intel's latest client processor, Tiger Lake, and Intel's first entry-level mainstream discrete graphics card, DG1. XeLP graphics in Tiger Lake and DG1 has democratized access to high performance implementations of the latest emerging video codec standards. Three topics will be covered: (i) 8K HEVC/AV1 Playback with Content Adaptive Power Optimization; (ii) Ludicrous Speed HEVC Encoding with Integrated + Discrete GPU; and (iii) MPEG Immersive Video (MIV) Playback on DG1.

With the advent of modern smartphones, AR, VR services and advancement in display resolution of mobile devices coupled with real-time streaming services, the demand for highresolution video has boomed. To fulfill this requirement, a variety of Adaptive Bit-Rate Streaming methods for Video-on-Demand applications are employed. However, the use of multi-pass encoding in the aforementioned methods renders them obsolete when it comes to real-time video streaming due to latency restrictions. In this work, we bypass the conventional multiple-encoding used in Video-on-Demand applications and present a novel machinelearning-based approach that estimates the optimal video resolution for a given content at a particular bit-rate for ultra low latency applications. A new feature that captures temporal as well as spatial correlation in video sequence has been used to train the Deep Neural Network (DNN) model. A python-based testbed is designed to evaluate the proposed scheme. Experiment results corroborate the viability and effectiveness of the proposed method for real-time mobile video streaming applications.

For current steganalysis of the image content-adaptive steganography, there are multiple problems to be improved, such as high difficulty and low accuracy, when detecting images with various contents and textures. For this problem, an improved steganalytic method is proposed in this paper based on the pre-classification and feature selection. First, using the features extracted based on the dependency analysis of image adjacent data, the images with various content and texture complexities are pre-classified as multiple clusters by the K-means algorithms. Then, the performance of existing various steganalytic features are analyzed for different clusters of images, and the optimal features for each cluster are selected for final classification. The experimental results show that the detection accuracy could be improved by the proposed method, and the rationality and availability are also verified. At the same time, the analysis and experimental results in this paper also show that the images with rich content and complex texture should be paid more attention both in steganography and in steganalysis.

In this paper, we firstly analyze the statistical distribution of simultaneous sparse coding errors (SSCE), which reflects the local correlation and non-local correlation characteristics of natural images. Based on the observation, we establish the optimal denoising problem which uses L1 norm to constrain the image prior. According to the close-form solution of the proposed problem, we find that the denoising limit is only determined by the patch number, patch size and the variances of different bands in SSCE. Then we exploit the relationships between the patch complexity, patch size, patch number and the denoising limit. The study shows that a content-adaptive strategy may be useful to obtain better denoising performance. We then design a content-adaptive method for image denoising in which the groups are adaptively determined according to the structural complexities of reference patches. Experimental results show that the proposed scheme achieves competitive performance with several state-of-the-art methods in both subjective and objective aspects.

Universal Steganalysis rely on extracting higher order statistical features that gets disturbed when hiding the message in a clean image. Due to content adaptive steganographies like HUGO, WOW etc. which embed the data more in textured areas of the image rather than smooth areas by minimizing the distortion of the image itself, first order features are not sufficient to differentiate clean and stego images. Thus, rich models come into picture in which a large number of features are extracted based on higher order noise residuals of clean and stego images. Thus, Universal Steganalyser is essentially a supervised classifier built on high dimensional feature set. To work with such high dimensional features on a large dataset of images is a very challenging task due to curse of dimensionality as well as computationally very expensive. This paper aims at comparing performance of three techniques-Ensemble classifier, Logistic regression and K-Nearest Neighbors on Spatial Rich Model features extracted for benchmarked dataset BOSSbase_1.01, for the better discrimination of clean and stego images.

This article introduces a novel RNN unit inspired by GRU, namely the Content-Adaptive Recurrent Unit (CARU). The design of CARU contains all the features of GRU but requires fewer training parameters. We make use of the concept of weights in our design to analyze the transition of hidden states. At the same time, we also describe how the content adaptive gate handles the received words and alleviates the long-term dependence problem. As a result, the unit can improve the accuracy of the experiments, and the results show that CARU not only has better performance than GRU, but also produces faster training. Moreover, the proposed unit is general and can be applied to all RNN related neural network models.

In this paper, we introduce a modification of the popular U-Net neural network architecture for medical image segmentation. Our proposed architecture replaces the concatenation operations in the traditional U-Net's skip connections with content-adaptive convolution, thereby significantly reducing the number of parameters of the network. Our experiments on two segmentation tasks - cell nuclei segmentation, and pneumo-thorax segmentation - demonstrated that the modified architecture achieves higher segmentation accuracy compared to the original U-net.

Imprecision and noise in the time series data may result in series with similar overall behaviors being recognized as being dissimilar because of the accumulation of many small local differences in noisy observations. While smoothing techniques can be used for eliminating such noise, the degree of smoothing that needs to be performed may vary significantly at di erent parts of the given time series. In this paper, we propose a content-adaptive smoothing, CA-Smooth, technique to reduce the impact of non-informative details and noise in time series by means of a data-driven approach to smoothing. The proposed smoothing process treats different parts of the time series according to local information content. We show the impact of di erent adaptive smoothing criteria on a number of samples from different datasets, containing series with diverse characteristics.

Rate Control (RC) plays an important role in video encoding. Traditional solutions are using fixed rate or fixed quantization parameters as the unified rate-control targets for all videos in one given video application. However, unified ratecontrol targets tend to have some bad encoding cases because of applying wrong rate for the video content. In this paper, we propose one content-adaptive rate control solution. We employ one neural-network based model which can end-to-end learn the optimal rate-control target appropriate to the content characteristics. The experimental results show that the proposed model can predict the optimal rate-factor value with the accuracy up to 77.637%. With this model, the proposed video-encoding method can significantly decrease the encoding quality fluctuation.

Image Super resolution is used to upscale the low resolution Images. It is also known as image upscaling. This paper focuses on upscaling of compressed images with interpolation based Single Image Super Resolution technique. A content adaptive interpolation method of image upscaling has been proposed. This interpolation based scheme is useful for single image based Super Resolution methods. The presented method works on horizontal, vertical and diagonal directions of an image separately and it is adaptive to the local content of an image. This method relies only on a single image and uses the content of the original image only; therefore, the proposed method is more practical and realistic. The simulation results have been compared to other standard methods with the help of various performance matrices like PSNR, MSE, MSSIM etc. which indicates the preeminence of the proposed method.

This paper proposes a content adaptive fast CU size selection algorithm for HEVC intra-prediction using weighted support vector machines. The proposed algorithm demonstrates an average encoding time reduction of 52.38% with 1.19% average BDBR increase compared to HM16.1 reference encoder.

Aiming at the problem of choosing high-pass filters in image processing layer of deep CNN (Convolution Neural Network) for steganalysis, this paper studies the effect of different filter banks on detection performance for content-adaptive JPEG steganography, and proposes a steganalysis method based on multi-scale 2D Gabor filtering and ensemble of multiple deep CNNs. Firstly, the effect of high-pass filters of image processing layer on detection performance is studied and the detection errors corresponding to the different combinations of high-pass filters are shown, then the relevant experimental results are analyzed. Lastly, two typical content-adaptive JPEG steganography algorithms such as UED and J-UNIWARD are taken as examples, the proposed steganalysis method is compared with other methods to verify the effectiveness of the proposed method.

This paper proposes a framework of content-adaptive image compressed sensing using deep learning, which analyzes the image content and adaptively allocates samples for different image patches accordingly. Experimental results demonstrate that the proposed framework outperforms the state-of-the-arts both in subjective and objective quality, especially at low sampling rates. For example, when the sampling rate is 0.1, 1–6 dB improvement in peak signal to noise ratio (PSNR) is observed. Moreover, the proposed work reconstructs images with more details and less image blocking effects, leading to apparent visual improvement.

Aiming at improved rate-distortion (R-D) performance, this paper presents a machine-learning based solution for the run-time video resolution adaptation problem. The proposed approachutilizes neural networks that leverage a complexity feature extracted from the video frames topredict a quantization parameter (QP) for downscaled video targeting the same bitrate as the native video. The peak signal to noise ratio (PSNR) is also predicted for both the native and downscaled resolutions, and the one that leads to the highest PSNR is selected. Experimental results show that \quad the proposed adaptive approach achieves significant improvements in R-D performance compared to using a fixed resolution.

In this paper, a novel hybrid cubemap projection (HCP) is proposed to improve the 360-degree video coding efficiency. HCP allows adaptive sampling adjustments in the horizontal and vertical directions within each cube face. HCP parameters of each cube face can be adjusted based on the input 360-degree video content characteristics for a better sampling efficiency. The HCP parameters can be updated periodically to adapt to temporal content variation. An efficient HCP parameter estimation algorithm is proposed to reduce the computational complexity of parameter estimation. Experimental results demonstrate that HCP format achieves on average luma (Y) BD-rate reduction of 11.51%, 8.0%, and 0.54% compared to equirectangular projection format, cubemap projection format, and adjusted cubemap projection format, respectively, in terms of end-to-end WS-PSNR.

In video applications it is necessary to continuously measure the video quality perceived by the end-user. Thus it is desirable to know which parts of video frame, i.e. which contents, attract viewers’ attention. If this information is known, then it is possible to estimate perceived video quality in a meaningful way. However, automatic detection of viewers’ fixation points is time-consuming process and often is omitted in objective video quality assessment (VQA) metrics. Based on our previous work, in which we proposed Foveation-based content Adaptive Root Mean Squared Error (FARMSE) VQA metric, in this work we propose two new full-reference (FR) VQA metrics called Multi-Point FARMSE (MP-FARMSE), and Simple-FARMSE (S-FARMSE). Both new-proposed metrics are based on foveated-vision features of human visual system and spatio-temporal features of video signal. In MP-FARMSE, by using an engineering approach, we implemented the fact that viewer’s attention can be directed out of the center of the frame, thus covering use-cases when objects of interest are not located in the center of the frame. The main idea when creating the S-FARMSE metric was to reduce the computational complexity of the final algorithm and to make S-FARMSE metric capable of processing high-resolution video signals in real-time. Performances of the new-proposed metrics are compared to performances of seven existing VQA metrics on two different video quality databases. The results show that performances achieved by MP-FARMSE and S-FARMSE are quite close to those of state-of-the-art VQA metrics, whereas at the same time their computational complexity level is significantly lower.

When the video is compressed and transmitted over heterogeneous networks, it is necessary to ensure the satisfying quality for the end user. Since human observers are the end users of video applications, it is very important that the human visual system (HVS) characteristics are taken into account during the video quality evaluation. This paper deals with video quality assessment (VQA) based on HVS characteristics and proposes a novel full-reference (FR) VQA metric called the Foveation-based content Adaptive Root Mean Squared Error (FARMSE). FARMSE uses several HVS characteristics that significantly influence perception of distortions in a video. Primarily these are foveated vision, reduction of the spatial acuity due to motions as well as spatial masking. Foveated vision is related to variable resolution of HVS across the viewing field, where the highest resolution is at the point of fixation. The point of fixation is projected onto the fovea – the area of retina with the highest density of photoreceptors. The part of image that falls on fovea is perceived by the highest acuity, whereas the spatial acuity decreases as the distance of the image part from the fovea increases. Spatial acuity further decreases if eyes cannot track moving objects. Both mentioned mechanisms influence contrast sensitivity of the HVS. Contrast sensitivity is frequency dependent and FARMSE uses Haar filters to utilize this dependence. Furthermore, spatial masking is implemented in each frequency channel. The FARMSE performance is compared to this of nine state-of-the-art VQA metrics on two different databases, LIVE and ECVQ. Additionally, the metrics are compared in terms of calculation complexity. The performed experiments show that FARMSE achieves high performance when predicting the quality of videos with different resolutions, degradation types and content types. FARMSE results outperform the results of most of the analyzed metrics, whereas they are comparable to these of the best publicly available metrics, including the well-known MOtion-based Video Integrity Evaluation (MOVIE) index. Besides that, FARMSE calculation complexity is significantly lower than that of the metrics comparable thereto in terms of prediction accuracy.

Detail enhancement is required by many problems in the fields of image processing and computational photography. Existing detail enhancement algorithms first decompose a source image into a base layer and a detail layer via an edge-preserving smoothing algorithm, and then amplify the detail layer to produce a detail-enhanced image. In this letter, we propose a new L 0 norm based detail enhancement algorithm which generates the detail-enhanced image directly. The proposed algorithm preserves sharp edges better than an existing L 0 norm based algorithm. Experimental results show that the proposed algorithm reduces color distortion in the detail-enhanced image, especially around sharp edges.

In high-efficiency coding (HEVC), the random-access structure (RAS) is employed due to its high coding efficiency and “random-access” performance. Pictures in RAS are assigned to different temporal layers. And how to select the QP for each temporal layer in quantization parameter cascading (QPC) technique is critical for improving the coding efficiency of RAS. In order to further improve the coding efficiency of RAS, a QPC technique considering the video content characteristics (denoted as VC-QPC for short) is proposed. In VC-QPC, motion and texture complexities of a video are used for predicting the optimal quantization parameter values of temporal layers. Compared with the method in the test model of HEVC, i.e., HM14.0, the BD-rate of proposed VC-QPC is −5.60% for RAS.

This study proposes a subpixel-based image downsampling algorithm using content-adaptive two-dimensional (2D) finite impulse response (FIR) filters. The proposed algorithm consists of a learning stage and an inference stage. In the learning stage, using a sufficient number of low-resolution (LR) and high-resolution (HR) patch pairs, the authors compute optimal 2D FIR filters to synthesise LR patches of the highest quality from a specific HR patch and store the patch-adaptive 2D FIR filters in a dictionary. In the inference stage, they explore candidates that best match to each HR input patch in the dictionary and synthesise LR patches by using their corresponding 2D FIR filters on a subpixel basis. The experimental results show that the proposed algorithm produces higher-quality LR images on a patch basis than existing methods and entails no blur and aliasing artefacts.

This paper proposes a content-adaptive sharpening algorithm using two-dimensional (2D) FIR filters trained by pre-emphasis for various image pairs. In the learning stage, all low-quality (LQ) and high-quality (HQ) image pairs are first pre-emphasized, i.e., properly sharpened. Then selective 2D FIR filter coefficients for high-frequency synthesis are trained using the pre-emphasized LQ–HQ image pairs, and then are stored in a dictionary that resembles an LUT (look-up table). In the inference stage, each input image is pre-emphasized in the same manner as in the learning stage. The best-matched 2D filter for each LQ patch is then found in the dictionary, and an HQ patch corresponding to the input LQ patch is synthesized using the resultant 2D FIR filter. The experiment results show that the proposed algorithm visually outperforms existing ones and that the mean of absolute errors (MAEs) and MSSSIM (multi-scale structure similarity) of the proposed algorithm are about 10% to 60% lower and about 0.002–0.053 higher, respectively than those of the existing algorithms.

This paper presents a steganalytic approach against video steganography which modifies motion vector (MV) in content adaptive manner. Current video steganalytic schemes extract features from fixed-length frames of the whole video and do not take advantage of the content diversity. Consequently, the effectiveness of the steganalytic feature is influenced by video content and the problem of cover source mismatch also affects the steganalytic performance. The goal of this paper is to propose a steganalytic method which can suppress the differences of statistical characteristics caused by video content. The given video is segmented to subsequences according to block’s motion in every frame. The steganalytic features extracted from each category of subsequences with close motion intensity are used to build one classifier. The final steganalytic result can be obtained by fusing the results of weighted classifiers. The experimental results have demonstrated that our method can effectively improve the performance of video steganalysis, especially for videos of low bitrate and low embedding ratio.