Search

Recently, most background modeling (BM) methods claim to achieve real-time efficiency for low-resolution and standard-definition surveillance videos. With the increasing resolutions of surveillance cameras, full high-definition (full HD) surveillance videos have become the main trend and thus processing high-resolution videos becomes a novel issue in intelligent video surveillance. In this article, we propose a novel content-adaptive resizing framework (CARF) to boost the computation speed of BM methods in high-resolution surveillance videos. For each frame, we apply superpixels to separate the content of the frame to homogeneous and boundary sets. Two novel downsampling and upsampling layers based on the homogeneous and boundary sets are proposed. The front one downsamples high-resolution frames to low-resolution frames for obtaining efficient foreground segmentation results based on BM methods. The later one upsamples the low-resolution foreground segmentation results to the original resolution frames based on the superpixels. By simultaneously coupling both layers, experimental results show that the proposed method can achieve better quantitative and qualitative results compared with state-of-the-art methods. Moreover, the computation speed of the proposed method without GPU accelerations is also significantly faster than that of the state-of-the-art methods. The source code is available at https://github.com/nchucvml/CARF.

We address the problem of efficient content-adaptive superpixel segmentation. Instead of adapting the size and/or amount of superpixels to the image content, we propose a warpingtransform that makes the image content more suitable to be segmented into regular superpixels. Regular superpixels in the warped image induce content-adaptive superpixels in the original image with improved segmentation accuracy. To efficiently compute the warping transform, we develop an iterative coarse-to-fine optimization procedure and employ a parallelization strategy allowing for a speedy GPU-based implementation. This solution works as a simple ‘add-on’ framework over an underlying segmentation algorithm and requires no additional parameters. Compared to the state-of-the-art methods, our approach provides competitive quality results and achieves a better time-accuracy trade-off. We further demonstrate the effectiveness of our method with an application to disparity estimation.

Objects at different spatial positions in an image exhibit different scales. Adaptive receptive fields are expected to capture suitable ranges of context for accurate pixel level semantic prediction. Recently, atrous convolution with different dilation rates has been used to generate features of multi-scales through several branches which are then fused for prediction. However, there is a lack of explicit interaction among the branches of different scales to adaptively make full use of the contexts. In this paper, we propose a Content-Adaptive Scale Interaction Network (CASINet) to exploit the multi-scale features for scene parsing. We build CASINet based on the classic Atrous Spatial Pyramid Pooling (ASPP) module, followed by a proposed contextual scale interaction (CSI) module, and a scale adaptation (SA) module. Specifically, in the CSI module, for each spatial position of some scale, instead of being limited by a fixed set of convolutional filters that are shared across different spatial positions for feature learning, we promote the adaptivity of the convolutional filters to spatial positions. We achieve this by the context interaction among the features of different scales. The SA module explicitly and softly selects the suitable scale for each spatial position and each channel. Ablation studies demonstrate the effectiveness of the proposed modules. We achieve state-of-the-art performance on three scene parsing benchmarks Cityscapes, ADE20K and LIP.

This paper presents a novel fast transcoding framework to efficiently bridge the state-of-art high efficiency video coding (HEVC) standard and its screen content coding (SCC) extension to support the bitstream compatibility over legacy HEVC devices. By exploiting the side information from the SCC bitstream, fast mode and partition decisions are made to accurately translate the novel SCC modes to conventional HEVC modes based on statistical mode mapping techniques. Compared with the full-decoding-full-encoding (FDFE) solution, the proposed framework achieves on average 51% and 82% complexity reductions with 0.57% Bjontegaard-delta rate (BD-Rate) loss and 9.74% BD-Rate gain under all-intra (AI) and low-delay (LD) configurations, respectively. Compared with the direct transcoding reusing intra mode and inter motion, the proposed mode mapping framework introduces additional 23% and 6% complexity reductions for AI and LD encoding configurations with 0.43% BD-Rate loss and 1.10% BD-Rate saving, respectively. The proposed solution is extended to support the single-input-multiple-output screen content adaptive streaming at the edge clouds, where an SCC bitstream coded in high quality is transcoded into multiple HEVC bitstreams in reduced qualities. Our proposed solution achieves on average 49% and 76% complexity reductions with 0.78% BD-Rate loss and 7.40% BD-Rate gain under AI and LD configurations, respectively.

Source camera identification is still a hard task in forensics community, especially for the case of the query images with small size. In this paper, we propose a solution to identify the source camera of the small-size images: content-adaptive fusion residual networks. According to the differences of the image contents, firstly, the images are divided into three subsets: saturation, smoothness and others. Then, we train three fusion residual networks for saturated images, smooth images, and others, separately, through transform learning. The fusion residual networks is formed with three paralleled residual networks and the difference of three residual networks lies in the convolutional kernel size of preprocessing layer. The features learned from the last residual blocks of three residual networks are fused and fed into softmax classifier. In particular, the residual networks is designed to learn better feature representation from the input data. The convolutional operation is added in preprocessing stage and three residual blocks are used. The experiment results show that the proposed method has satisfactory performances at three levels of source camera identification: brand level, model level, and device level.

Image Super resolution is used to upscale the low resolution Images. It is also known as image upscaling. This paper focuses on upscaling of compressed images with interpolation based Single Image Super Resolution technique. A content adaptive interpolation method of image upscaling has been proposed. This interpolation based scheme is useful for single image based Super Resolution methods. The presented method works on horizontal, vertical and diagonal directions of an image separately and it is adaptive to the local content of an image. This method relies only on a single image and uses the content of the original image only; therefore, the proposed method is more practical and realistic. The simulation results have been compared to other standard methods with the help of various performance matrices like PSNR, MSE, MSSIM etc. which indicates the preeminence of the proposed method.

Detail enhancement is required by many problems in the fields of image processing and computational photography. Existing detail enhancement algorithms first decompose a source image into a base layer and a detail layer via an edge-preserving smoothing algorithm, and then amplify the detail layer to produce a detail-enhanced image. In this letter, we propose a new L 0 norm based detail enhancement algorithm which generates the detail-enhanced image directly. The proposed algorithm preserves sharp edges better than an existing L 0 norm based algorithm. Experimental results show that the proposed algorithm reduces color distortion in the detail-enhanced image, especially around sharp edges.

This study proposes a subpixel-based image downsampling algorithm using content-adaptive two-dimensional (2D) finite impulse response (FIR) filters. The proposed algorithm consists of a learning stage and an inference stage. In the learning stage, using a sufficient number of low-resolution (LR) and high-resolution (HR) patch pairs, the authors compute optimal 2D FIR filters to synthesise LR patches of the highest quality from a specific HR patch and store the patch-adaptive 2D FIR filters in a dictionary. In the inference stage, they explore candidates that best match to each HR input patch in the dictionary and synthesise LR patches by using their corresponding 2D FIR filters on a subpixel basis. The experimental results show that the proposed algorithm produces higher-quality LR images on a patch basis than existing methods and entails no blur and aliasing artefacts.

This paper proposes a content-adaptive sharpening algorithm using two-dimensional (2D) FIR filters trained by pre-emphasis for various image pairs. In the learning stage, all low-quality (LQ) and high-quality (HQ) image pairs are first pre-emphasized, i.e., properly sharpened. Then selective 2D FIR filter coefficients for high-frequency synthesis are trained using the pre-emphasized LQ–HQ image pairs, and then are stored in a dictionary that resembles an LUT (look-up table). In the inference stage, each input image is pre-emphasized in the same manner as in the learning stage. The best-matched 2D filter for each LQ patch is then found in the dictionary, and an HQ patch corresponding to the input LQ patch is synthesized using the resultant 2D FIR filter. The experiment results show that the proposed algorithm visually outperforms existing ones and that the mean of absolute errors (MAEs) and MSSSIM (multi-scale structure similarity) of the proposed algorithm are about 10% to 60% lower and about 0.002–0.053 higher, respectively than those of the existing algorithms.

In this paper, a content-adaptive deblocking method is proposed to improve the visual quality of block-based DCT compressed videos. We find that the edge information obtained through the global orientation energy edge detection (OEED) on an initially deblocked image provides a robust partition of local directional features (LDFs). Based on this partition, for the directional featured region, we design corresponding filter orientation and thresholds to best preserve image details while reducing blocking artifacts; for the consecutive non-featured region, we impose extra smoothing to suppress the visually severe blocking artifacts. Experimental results demonstrate that our method effectively improves the visual quality while well maintaining the objective fidelity of block-based DCT compressed videos, compared with the H.264/AVC deblocking method and other existing directional deblocking methods.

We optimize automultiscopic displays built by stacking a pair of modified LCD panels. To date, such dual-stacked LCDs have used heuristic parallax barriers for view-dependent imagery: the front LCD shows a fixed array of slits or pinholes, independent of the multi-view content. While prior works adapt the spacing between slits or pinholes, depending on viewer position, we show both layers can also be adapted to the multi-view content, increasing brightness and refresh rate. Unlike conventional barriers, both masks are allowed to exhibit non-binary opacities. It is shown that any 4D light field emitted by a dual-stacked LCD is the tensor product of two 2D masks. Thus, any pair of 1D masks only achieves a rank-1 approximation of a 2D light field. Temporal multiplexing of masks is shown to achieve higher-rank approximations. Non-negative matrix factorization (NMF) minimizes the weighted Euclidean distance between a target light field and that emitted by the display. Simulations and experiments characterize the resulting content-adaptive parallax barriers for low-rank light field approximation.

Portable and mobile multimedia production faces special challenges posed by transmission bandwidth limitations. This paper describes experimental studies of the multimodal quality of compressed television materials for mobile and portable devices, discusses required optimizations and proposes a mathematical model for content adaptive bit rate allocation. Study results are used to discuss the characteristics of multimodal quality and support the proposition that, if multimodal audiovisual perceptual quality is taken into account, it will improve the quality of new portable and mobile multimedia applications.

In this paper, a method for Lagrange multiplier selection is proposed in the context of rate-distortion optimisation for wavelet-based scalable video coding targeting quality scalability. Despite the prevalence of the conventional method for Lagrange multiplier selection in hybrid video coding, the underlying formulation is not applicable to wavelet-based scalable video coding. To address the inherent challenges, a thorough analysis of the rate-distortion models for transform video coding is provided with regard to low and middle-to-high bit-rates, respectively. Based on the analysis, the models are consolidated according to experimental observations and the consolidated rate-distortion models serve as the basis for the derivation of the Lagrange multiplier. Considering the influence of the open-loop prediction structure on the rate-distortion performance, the Lagrange multiplier is initially derived for a single-targeted bit-rate. Moreover, the method for Lagrange multiplier selection in scalable video coding aiming at multiple-targeted bit-rates is proposed in a general sense of bit-rate range, varying from low to high bit-rates, building on the initially derived Lagrange multiplier for a single-targeted bit-rate. The proposed Lagrange multiplier is content adaptive and well suited for wavelet-based scalable video coding where quantisation steps are unavailable. Detailed performance evaluation of the proposed method for wavelet-based scalable video coding is provided with regard to a given targeted bit-rate and multiple-targeted bit-rates, respectively. The experimental results have demonstrated the effectiveness of the proposed Lagrange multiplier for rate-distortion optimisation considering quality scalability in wavelet-based scalable video coding.

Skin detection plays an important role in applications such as face detection and tracking, person detection and pornography detection. While previous studies focus on pixel-based skin color detection techniques that individually classify each pixel as skin color or non-skin color, this study presents a region-based algorithm for detecting skin color. The proposed algorithm uses a special region, called key skin region, as the basis to classify skin color. A performance comparison with conventional skin classifiers, including the Bayesian classifier, the unimodal Gaussian classifier and the Gaussian mixture classifier, is made in this study. Experimental results show that the proposed algorithm outperforms other tested skin classifiers. Furthermore, the skin regions detected by the proposed algorithm, especially facial regions, are nearly complete with no hollow holes in these regions. This property can simplify the complexity of implementing applications that use skin color as their basis, such as face detection and face tracking.

We propose a new pre-roll delay-distortion optimization (DDO) framework that allows determination of the minimum pre-roll delay and distortion while ensuring continuous playback for on-demand content-adaptive video streaming over limited bitrate networks. The input video is first divided into temporal segments, which are assigned a relevance weight and a maximum distortion level, called relevance-distortion policy, which may be specified by the user. The system then encodes the input video according to the specified relevance-distortion policy, whereby the optimal spatial and temporal resolutions and quantization parameters, also called encoding parameters, are selected for each temporal segment. The optimal encoding parameters are computed using a novel, multi-objective optimization formulation, where a relevance weighted distortion measure and pre-roll delay are jointly minimized under maximum allowable buffer size, continuous playback, and maximum allowable distortion constraints. The performance of the system has been demonstrated for on-demand streaming of soccer videos with substantial improvement in the weighted distortion without any increase in pre-roll delay over a very low-bitrate network using AVC/H.264 encoding.

A content adaptive, network aware approach for joint optimization of wireless video transmission is presented in this article. The joint system optimization approach has been developed in the framework of the IST PHOENIX project (www.ist-phoenix.org). After a description of the considered cross-layer approach and of the information to be exchanged among the system component blocks, the concept of "JSCC/D controllers" is introduced. Results obtained through the demonstrator realized as proof-of-concept for the described paradigm are then shown, confirming the validity of the described joint optimization approach

Fourier ptychography (FP) is a recently reported technique, for large field-of-view and high-resolution imaging. Specifically, FP captures a set of low-resolution images, under angularly varying illuminations, and stitches them together in the Fourier domain. One of FP's main disadvantages is its long capturing process, due to the requisite large number of incident illumination angles. In this Letter, utilizing the sparsity of natural images in the Fourier domain, we propose a highly efficient method, termed adaptive Fourier ptychography (AFP), which applies content adaptive illumination for FP, to capture the most informative parts of the scene's spatial spectrum. We validate the effectiveness and efficiency of the reported framework, with both simulated and real experiments. Results show that the proposed AFP could shorten the acquisition time of conventional FP, by around 30%-60%.

A novel content-adaptive music watermarking technique is proposed in this article. To optimally balance inaudibility and robustness when embedding and extracting watermarks, the embedding scheme is highly related to the music structure and human auditory system (HAS). A note-based segmentation method is proposed and used for music vocal/instrumental boundary detection. A multiple bit hopping and hiding scheme with different embedding parameters is applied to vocal and instrumental frames of the music. The experimental results in inaudibility and robustness are provided to support all novel features in the proposed watermarking scheme.