Search

Objects at different spatial positions in an image exhibit different scales. Adaptive receptive fields are expected to capture suitable ranges of context for accurate pixel level semantic prediction. Recently, atrous convolution with different dilation rates has been used to generate features of multi-scales through several branches which are then fused for prediction. However, there is a lack of explicit interaction among the branches of different scales to adaptively make full use of the contexts. In this paper, we propose a Content-Adaptive Scale Interaction Network (CASINet) to exploit the multi-scale features for scene parsing. We build CASINet based on the classic Atrous Spatial Pyramid Pooling (ASPP) module, followed by a proposed contextual scale interaction (CSI) module, and a scale adaptation (SA) module. Specifically, in the CSI module, for each spatial position of some scale, instead of being limited by a fixed set of convolutional filters that are shared across different spatial positions for feature learning, we promote the adaptivity of the convolutional filters to spatial positions. We achieve this by the context interaction among the features of different scales. The SA module explicitly and softly selects the suitable scale for each spatial position and each channel. Ablation studies demonstrate the effectiveness of the proposed modules. We achieve state-of-the-art performance on three scene parsing benchmarks Cityscapes, ADE20K and LIP.

Content-adaptive steganography intends to hide data in the complex texture content of the image. Recently, some secure steganography methods have been proposed to identify the textural complexity of an image. However, most of the techniques do not take into account the information of pixel variation around the central pixel in all possible directions and therefore they are unable to accurately analyse the texture complexity. This work offers a quality-enhanced and secure method of content-adaptive image steganography. The proposed method is divided into three sequential steps: image segmentation, pixel complexity identification, and data embedding. An input cover image is initially divided into small local regions and the pixel-complexity is identified based on the proposed Complex Block Prior (CBP) criterion. In a local block, a high pass filter (HPF) bank is applied and eight residual responses are obtained. Following the CBP criterion, a complexity level out of nine levels is assigned to an individualized pixel block. The pixels are then arranged in the priority of complexity from highest to lowest. Data embedding for the corresponding complexity level then takes place using the proposed adaptive embedding algorithm. Experimental results verify the preservation of visual quality of stego images produced by the proposed method. Three image datasets: Standard test images, BOWS2 and BOSS-base are used for the experimentation and comparison with prior state-of-art methods. Highest values of the IQ (image quality) parameters e.g., SSIM and WPSNR show the effectiveness of the proposed method.

This paper presents a novel fast transcoding framework to efficiently bridge the state-of-art high efficiency video coding (HEVC) standard and its screen content coding (SCC) extension to support the bitstream compatibility over legacy HEVC devices. By exploiting the side information from the SCC bitstream, fast mode and partition decisions are made to accurately translate the novel SCC modes to conventional HEVC modes based on statistical mode mapping techniques. Compared with the full-decoding-full-encoding (FDFE) solution, the proposed framework achieves on average 51% and 82% complexity reductions with 0.57% Bjontegaard-delta rate (BD-Rate) loss and 9.74% BD-Rate gain under all-intra (AI) and low-delay (LD) configurations, respectively. Compared with the direct transcoding reusing intra mode and inter motion, the proposed mode mapping framework introduces additional 23% and 6% complexity reductions for AI and LD encoding configurations with 0.43% BD-Rate loss and 1.10% BD-Rate saving, respectively. The proposed solution is extended to support the single-input-multiple-output screen content adaptive streaming at the edge clouds, where an SCC bitstream coded in high quality is transcoded into multiple HEVC bitstreams in reduced qualities. Our proposed solution achieves on average 49% and 76% complexity reductions with 0.78% BD-Rate loss and 7.40% BD-Rate gain under AI and LD configurations, respectively.

Source camera identification is still a hard task in forensics community, especially for the case of the query images with small size. In this paper, we propose a solution to identify the source camera of the small-size images: content-adaptive fusion residual networks. According to the differences of the image contents, firstly, the images are divided into three subsets: saturation, smoothness and others. Then, we train three fusion residual networks for saturated images, smooth images, and others, separately, through transform learning. The fusion residual networks is formed with three paralleled residual networks and the difference of three residual networks lies in the convolutional kernel size of preprocessing layer. The features learned from the last residual blocks of three residual networks are fused and fed into softmax classifier. In particular, the residual networks is designed to learn better feature representation from the input data. The convolutional operation is added in preprocessing stage and three residual blocks are used. The experiment results show that the proposed method has satisfactory performances at three levels of source camera identification: brand level, model level, and device level.

With the advent of modern smartphones, AR, VR services and advancement in display resolution of mobile devices coupled with real-time streaming services, the demand for highresolution video has boomed. To fulfill this requirement, a variety of Adaptive Bit-Rate Streaming methods for Video-on-Demand applications are employed. However, the use of multi-pass encoding in the aforementioned methods renders them obsolete when it comes to real-time video streaming due to latency restrictions. In this work, we bypass the conventional multiple-encoding used in Video-on-Demand applications and present a novel machinelearning-based approach that estimates the optimal video resolution for a given content at a particular bit-rate for ultra low latency applications. A new feature that captures temporal as well as spatial correlation in video sequence has been used to train the Deep Neural Network (DNN) model. A python-based testbed is designed to evaluate the proposed scheme. Experiment results corroborate the viability and effectiveness of the proposed method for real-time mobile video streaming applications.

This article introduces a novel RNN unit inspired by GRU, namely the Content-Adaptive Recurrent Unit (CARU). The design of CARU contains all the features of GRU but requires fewer training parameters. We make use of the concept of weights in our design to analyze the transition of hidden states. At the same time, we also describe how the content adaptive gate handles the received words and alleviates the long-term dependence problem. As a result, the unit can improve the accuracy of the experiments, and the results show that CARU not only has better performance than GRU, but also produces faster training. Moreover, the proposed unit is general and can be applied to all RNN related neural network models.

In this paper, we introduce a modification of the popular U-Net neural network architecture for medical image segmentation. Our proposed architecture replaces the concatenation operations in the traditional U-Net's skip connections with content-adaptive convolution, thereby significantly reducing the number of parameters of the network. Our experiments on two segmentation tasks - cell nuclei segmentation, and pneumo-thorax segmentation - demonstrated that the modified architecture achieves higher segmentation accuracy compared to the original U-net.

Imprecision and noise in the time series data may result in series with similar overall behaviors being recognized as being dissimilar because of the accumulation of many small local differences in noisy observations. While smoothing techniques can be used for eliminating such noise, the degree of smoothing that needs to be performed may vary significantly at di erent parts of the given time series. In this paper, we propose a content-adaptive smoothing, CA-Smooth, technique to reduce the impact of non-informative details and noise in time series by means of a data-driven approach to smoothing. The proposed smoothing process treats different parts of the time series according to local information content. We show the impact of di erent adaptive smoothing criteria on a number of samples from different datasets, containing series with diverse characteristics.

Image Super resolution is used to upscale the low resolution Images. It is also known as image upscaling. This paper focuses on upscaling of compressed images with interpolation based Single Image Super Resolution technique. A content adaptive interpolation method of image upscaling has been proposed. This interpolation based scheme is useful for single image based Super Resolution methods. The presented method works on horizontal, vertical and diagonal directions of an image separately and it is adaptive to the local content of an image. This method relies only on a single image and uses the content of the original image only; therefore, the proposed method is more practical and realistic. The simulation results have been compared to other standard methods with the help of various performance matrices like PSNR, MSE, MSSIM etc. which indicates the preeminence of the proposed method.

This paper proposes a content adaptive fast CU size selection algorithm for HEVC intra-prediction using weighted support vector machines. The proposed algorithm demonstrates an average encoding time reduction of 52.38% with 1.19% average BDBR increase compared to HM16.1 reference encoder.

This paper proposes a framework of content-adaptive image compressed sensing using deep learning, which analyzes the image content and adaptively allocates samples for different image patches accordingly. Experimental results demonstrate that the proposed framework outperforms the state-of-the-arts both in subjective and objective quality, especially at low sampling rates. For example, when the sampling rate is 0.1, 1–6 dB improvement in peak signal to noise ratio (PSNR) is observed. Moreover, the proposed work reconstructs images with more details and less image blocking effects, leading to apparent visual improvement.

Aiming at improved rate-distortion (R-D) performance, this paper presents a machine-learning based solution for the run-time video resolution adaptation problem. The proposed approachutilizes neural networks that leverage a complexity feature extracted from the video frames topredict a quantization parameter (QP) for downscaled video targeting the same bitrate as the native video. The peak signal to noise ratio (PSNR) is also predicted for both the native and downscaled resolutions, and the one that leads to the highest PSNR is selected. Experimental results show that \quad the proposed adaptive approach achieves significant improvements in R-D performance compared to using a fixed resolution.

In this paper, a novel hybrid cubemap projection (HCP) is proposed to improve the 360-degree video coding efficiency. HCP allows adaptive sampling adjustments in the horizontal and vertical directions within each cube face. HCP parameters of each cube face can be adjusted based on the input 360-degree video content characteristics for a better sampling efficiency. The HCP parameters can be updated periodically to adapt to temporal content variation. An efficient HCP parameter estimation algorithm is proposed to reduce the computational complexity of parameter estimation. Experimental results demonstrate that HCP format achieves on average luma (Y) BD-rate reduction of 11.51%, 8.0%, and 0.54% compared to equirectangular projection format, cubemap projection format, and adjusted cubemap projection format, respectively, in terms of end-to-end WS-PSNR.

When the video is compressed and transmitted over heterogeneous networks, it is necessary to ensure the satisfying quality for the end user. Since human observers are the end users of video applications, it is very important that the human visual system (HVS) characteristics are taken into account during the video quality evaluation. This paper deals with video quality assessment (VQA) based on HVS characteristics and proposes a novel full-reference (FR) VQA metric called the Foveation-based content Adaptive Root Mean Squared Error (FARMSE). FARMSE uses several HVS characteristics that significantly influence perception of distortions in a video. Primarily these are foveated vision, reduction of the spatial acuity due to motions as well as spatial masking. Foveated vision is related to variable resolution of HVS across the viewing field, where the highest resolution is at the point of fixation. The point of fixation is projected onto the fovea – the area of retina with the highest density of photoreceptors. The part of image that falls on fovea is perceived by the highest acuity, whereas the spatial acuity decreases as the distance of the image part from the fovea increases. Spatial acuity further decreases if eyes cannot track moving objects. Both mentioned mechanisms influence contrast sensitivity of the HVS. Contrast sensitivity is frequency dependent and FARMSE uses Haar filters to utilize this dependence. Furthermore, spatial masking is implemented in each frequency channel. The FARMSE performance is compared to this of nine state-of-the-art VQA metrics on two different databases, LIVE and ECVQ. Additionally, the metrics are compared in terms of calculation complexity. The performed experiments show that FARMSE achieves high performance when predicting the quality of videos with different resolutions, degradation types and content types. FARMSE results outperform the results of most of the analyzed metrics, whereas they are comparable to these of the best publicly available metrics, including the well-known MOtion-based Video Integrity Evaluation (MOVIE) index. Besides that, FARMSE calculation complexity is significantly lower than that of the metrics comparable thereto in terms of prediction accuracy.

In high-efficiency coding (HEVC), the random-access structure (RAS) is employed due to its high coding efficiency and “random-access” performance. Pictures in RAS are assigned to different temporal layers. And how to select the QP for each temporal layer in quantization parameter cascading (QPC) technique is critical for improving the coding efficiency of RAS. In order to further improve the coding efficiency of RAS, a QPC technique considering the video content characteristics (denoted as VC-QPC for short) is proposed. In VC-QPC, motion and texture complexities of a video are used for predicting the optimal quantization parameter values of temporal layers. Compared with the method in the test model of HEVC, i.e., HM14.0, the BD-rate of proposed VC-QPC is −5.60% for RAS.

This paper presents a steganalytic approach against video steganography which modifies motion vector (MV) in content adaptive manner. Current video steganalytic schemes extract features from fixed-length frames of the whole video and do not take advantage of the content diversity. Consequently, the effectiveness of the steganalytic feature is influenced by video content and the problem of cover source mismatch also affects the steganalytic performance. The goal of this paper is to propose a steganalytic method which can suppress the differences of statistical characteristics caused by video content. The given video is segmented to subsequences according to block’s motion in every frame. The steganalytic features extracted from each category of subsequences with close motion intensity are used to build one classifier. The final steganalytic result can be obtained by fusing the results of weighted classifiers. The experimental results have demonstrated that our method can effectively improve the performance of video steganalysis, especially for videos of low bitrate and low embedding ratio.

Sketch portrait generation benefits a wide range of applications such as digital entertainment and law enforcement. Although plenty of efforts have been dedicated to this task, several issues still remain unsolved for generating vivid and detail-preserving personal sketch portraits. For example, quite a few artifacts may exist in synthesizing hairpins and glasses, and textural details may be lost in the regions of hair or mustache. Moreover, the generalization ability of current systems is somewhat limited since they usually require elaborately collecting a dictionary of examples or carefully tuning features/components. In this paper, we present a novel representation learning framework that generates an end-to-end photo-sketch mapping through structure and texture decomposition. In the training stage, we first decompose the input face photo into different components according to their representational contents (i.e., structural and textural parts) by using a pre-trained Convolutional Neural Network (CNN). Then, we utilize a Branched Fully Convolutional Neural Network (BFCN) for learning structural and textural representations, respectively. In addition, we design a Sorted Matching Mean Square Error (SM-MSE) metric to measure texture patterns in the loss function. In the stage of sketch rendering, our approach automatically generates structural and textural representations for the input photo and produces the final result via a probabilistic fusion scheme. Extensive experiments on several challenging benchmarks suggest that our approach outperforms example-based synthesis algorithms in terms of both perceptual and objective metrics. In addition, the proposed method also has better generalization ability across dataset without additional training., Comment: Published in TIP 2017

Near-eye multi-focal (light field) displays are known to be more advantageous than conventional stereoscopic 3D displays, because they can alleviate the vergence-accommodation conflict. While stereoscopic 3D displays are restricted to projecting images on a single focal plane, near-eye light field displays can form multiple focal planes and volumetrically render 3D data. Most existing near-eye light field displays use simplistic, uniformly spaced focal plane configuration (in dioptric space). In this paper, we present a novel technique that optimizes the focal plane configuration based on characteristics of the content to be rendered. We show that this technique can significantly improve the perceived visual quality of content visualized on such displays.

Source camera identification (SCI) is an important topic in image forensics. One of the most effective fingerprints for linking an image to its source camera is the sensor pattern noise, which is estimated as the difference between the content and its denoised version. It is widely believed that the performance of the sensor-based SCI heavily relies on the denoising filter used. This study proposes a novel sensor-based SCI method using content adaptive guided image filter (CAGIF). Thanks to the low complexity nature of the CAGIF, the proposed method is much faster than the state-of-the-art methods, which is a big advantage considering the potential real-time application of SCI. Despite the advantage of speed, experimental results also show that the proposed method can achieve comparable or better performance than the state-of-the-art methods in terms of accuracy.

This paper presents a content-adaptive coding scheme for immersive networked experience of sports events, in particular, soccer games. We assume that future sports events are captured by an array of fixed high-definition cameras which provide multiview image sequences for a free-viewpoint immersive networked experience in a home environment. We discuss a content-adaptive coding scheme for image sequences that exploits properties of such sequences and that permits efficient user interactions. In this work, we construct a rate distortion model for an image sequence to obtain the optimal bitrate allocation among static and dynamic content items. The optimal bitrate allocation results in a rate distortion performance of the coding scheme that outperforms that of conventional H.264/AVC coding significantly.

The design of computer-assisted decision (CAD) systems for different biomedical imaging scenarios is a challenging task in computer vision. Sometimes, this challenge can be attributed to the image acquisition mechanisms since the lack of control on the cameras can create different visualizations of the same imaging site under different rotation, scaling, and illumination parameters, with a requirement to get a consistent diagnosis by the CAD systems. Moreover, the images acquired from different sites have specific colors, making the use of standard color spaces highly redundant. In this paper, we propose to tackle these issues by introducing novel region-based texture, and color descriptors. The proposed texture features are based on the usage of analytic Gabor filters (for compensation of illumination variations) followed by the calculation of first- and second-order statistics of the filter responses and making them invariant using some trivial mathematical operators. The proposed color features are obtained by compensating for the illumination variations in the images using homomorphic filtering followed by a bag-of-words approach to obtain the most typical colors in the images. The proposed features are used for the identification of cancer in images from two distinct imaging modalities, i.e., gastroenterology and dermoscopy . Experiments demonstrate that the proposed descriptors compares favorably to several other state-of-the-art methods, elucidating on the effectiveness of adapted features for image characterization.