Search

Local and non-local attention-based methods have been well studied in various image restoration tasks while leading to promising performance. However, most of the existing methods solely focus on one type of attention mechanism (local or non-local). Furthermore, by exploiting the self-similarity of natural images, existing pixel-wise non-local attention operations tend to give rise to deviations in the process of characterizing long-range dependence due to image degeneration. To overcome these problems, in this paper we propose a novel collaborative attention network (COLA-Net) for image restoration, as the first attempt to combine local and non-local attention mechanisms to restore image content in the areas with complex textures and with highly repetitive details respectively. In addition, an effective and robust patch-wise non-local attention model is developed to capture long-range feature correspondences through 3D patches. Extensive experiments on synthetic image denoising, real image denoising and compression artifact reduction tasks demonstrate that our proposed COLA-Net is able to achieve state-of-the-art performance in both peak signal-to-noise ratio and visual perception, while maintaining an attractive computational complexity. The source code is available on https://github.com/MC-E/COLA-Net., 11 pages, 6 tables, 9 figures, to be published in IEEE Transactions on Multimedia

Images taken in extremely low light suffer from various problems such as heavy noise, blur, and color distortion. Assuming the low-light images contain a good representation of the scene content, current enhancement methods focus on finding a suitable illumination adjustment but often fail to deal with heavy noise and color distortion. Recently, some works try to suppress noise and reconstruct low-light images from raw data. But these works apply a network instead of an image signal processing pipeline (ISP) to map the raw data to enhanced results which leads to heavy learning burden for the network and get unsatisfactory results. In order to remove heavy noise, correct color bias and enhance details more effectively, we propose a two-stage Low Light Image Signal Processing Network named LLISP. The design of our network is inspired by the traditional ISP: processing the images in multiple stages according to the attributes of different tasks. In the first stage, a simple denoising module is introduced to reduce heavy noise. In the second stage, we propose a two-branch network to reconstruct the low-light images and enhance texture details. One branch aims at correcting color distortion and restoring image content, while another branch focuses on recovering realistic texture. Experimental results demonstrate that the proposed method can reconstruct high-quality images from low-light raw data and replace the traditional ISP.

The major task of traditional motion deblurring methods is to estimate the blur kernel and restore the latent image. In low-light conditions, the pointolite is likely to produce saturated light streaks in captured blurred images. The light streaks are usually double-edged swords—outliers to the deconvolution, but a cue to kernel estimation. In this paper, we propose a novel blind motion deblurring method for blurred images including light streaks. The main idea is to model the non-linear blur caused by outliers as the Huber's M-estimation in blind deconvolution and take the shape of the light streak as a cue to estimate the blur kernel. Specifically, the optimal light streak patch is selected automatically according to the characteristics of light streaks and the blur kernel. This simple yet effective selection strategy solves the problems of false detection of candidate light streaks and optimal light streak in existing methods. Then, the optimal light streak patch is parameterized as a prior and is combined with other regularizers to estimate the blur kernel. Compared with the state-of-the-art kernel estimation methods, the proposed algorithm reduces the influence of outliers on deconvolution and utilizes more information. Thus, the restored image is more accurate. Experimental results on both synthetic and real images demonstrate the high accuracy of our algorithm.

The existing learning-based multi-view stereo (MVS) approaches achieve impressive results compared with traditional methods. However, most of them rely on ground-truth 3D data as supervision, and the acquisition of high-quality ground truth for various scenes is a challenging problem. In this paper, we propose a novel real-time unsupervised multi-view depth estimation network for virtual view synthesis tasks and take multi-view images as supervision. To improve the completeness and accuracy of the virtual viewpoint, we propose a novel shared occlusion mask to deal with the artifacts caused by occlusion in the reconstructed image, and filter out the unreliable points in the depth map. Besides, we also design a mask-based photometric loss to guide our network to generate more reasonable masks and high-quality depth maps. Experimental results on the IEEE1857.9 virtual viewpoint synthesis dataset demonstrate that our proposed method outperforms other recent MVS methods and achieves more excellent real-time performance.

Video frame interpolation aims at synthesizing new video frames in-between existing frames to generate higher frame rate video. Current methods usually use two adjacent frames to generate intermediate frames, but sometimes fail to handle challenges like large motion, occlusion, and motion blur. This paper proposes a multi-frame pyramid refinement network to effectively use spatio-temporal information contained in multiple frames (more than two). There are three technical contributions in the proposed network. First, a special coarse-to-fine framework is proposed to refine optical flows in-between multiple frames with residual flows at each pyramid level. Therefore, large motion and occlusion can be effectively estimated. Second, a 3D U-net feature extractor is used to excavate spatio-temporal context and restore texture, which tend to disappear at course pyramid levels. Third, a multi-step perceptual loss is adopted to preserve more details in intermediate frame. It is worth mentioning that our approach can be easily extended to multi-frame interpolation. Our network is trained end-to-end using more than 80K collected frame groups (25 frames per group). Experimental results on several independent datasets show that our approach can effectively handle challenging cases, and perform consistently better than other state-of-the-art methods.

Converting a single low dynamic range (LDR) image into a high dynamic range (HDR) image, which is the so-called inverse tone mapping (ITM), is a challenging ill-posed problem since a lot of information is lost during compression and storage. Traditional ITM techniques mainly focus on high-quality LDR images without compression artifacts. However, in practice, LDR images are usually stored as a lossy compression format for the convenience of transmission, which will cause artifacts, i.e., blocking and ringing. Hence, these ITM methods suffer from severe performance drop for some real-world applications. In this paper, we propose a novel decomposition-based network to reconstruct HDR images from compressed LDR images; in other words, the proposed network can simultaneously remove blocking/ringing artifacts and recover high-quality HDR information. Considering compression artifacts mainly embedded in high frequency part, we thus decompose the input image into the high-frequency component (also known as detail layer) and low-frequency component (also known as a base layer). The detail layer mainly contains the information of texture, noise, and artifacts, while the base layer contains the information of structure and large object. Based on this, we design two subnetworks, i.e., detail layer recovery subnetwork and base layer recovery subnetwork, to restore the two parts separately. The detail layer recovery subnetwork is responsible for the artifacts removal with texture preserving, while the base layer recovery subnetwork is designed for tone expansion and overexposed/underexposed region restoration. Finally, in order to further reconstruct the serious overexposed regions, we adapt a merge subnetwork to fuse the result from the previous two subnetworks. The experimental results on compressed images demonstrate that the proposed method significantly outperforms other state-of-the-art methods.

Lossy video compression is widely applied in the process of video transmission and storage. With the pursuit of higher compression ratio, the demand for compressed video-enhancement is derived from the annoying degradation. Recently some deep-neural-network-based approaches have achieved impressive performance in this task. Unfortunately, most of them are plainly designed for images without exploiting the high temporal redundancy for video restoration. To this end, we present a two-stage Video compression Artifacts Removal Neural Network (VARNN) with deformable convolutional kernels and modified non-local blocks. Conventional non-local block is incapable of capturing dependencies among channels and suffers from high computation and memory cost. We thus introduce a Divided Non-Local Block (DNLB), in which global dependencies can be captured in a fast and low space-cost way. Finally, experiments show that the proposed VARNN outperforms some state-of-the-art methods.

With the rapid development of image restoration techniques, high-definition reconstruction of early videos has achieved impressive results. However, there are few studies about the interlacing artifacts that often appear in early videos and significantly affect visual perception. Traditional deinterlacing approaches are mainly focused on early interlacing scanning systems and thus cannot handle the complex and complicated artifacts in real-world early interlaced videos. Hence, this paper proposes a specific deinterlacing network (DIN), which is motivated by the traditional deinterlacing strategy. The proposed DIN consists of two stages, i.e., a cooperative vertical interpolation stage for split fields, and a merging stage that is applied to perceive movements and remove ghost artifacts. Experimental results demonstrate that the proposed method can effectively remove complex artifacts in early interlaced videos., 6 pages, 5 figures

3D movie has become the mainstream for in cinema, and the growing of 3D television and other digital devices is rapid, 3D contents are more demanded in recent years. Producing stereoscopic effect from many existing 2D videos is the most cost-effective under many circumstances. In this paper, we aim to integrate the pipeline into an end-to-end framework, fusing the unsupervised depth estimation with DIBR, the synthesized view generated by the network directly to get the frame for the virtual view angle, and the segmentation model. The fusion makes it possible to exploit both contextual and geometric information better.

Low-light image enhancement is still a challenging task nowadays. On one hand, sensitive methods tend to reproduce light images with severe noise and color deviation. On the other hand, insensitive methods can recover clear and natural results but with much lower brightness. Hence, this paper analyzes several basic mathematical models and then proposes a low-light image enhancement network (LLIENet) based on a basic mathematical model, which contains several modules. First, a MaskNet is proposed to estimate the global illumination prior. Second, BaseNet and CoefficientNet are used to decompose the low-light image into a lightened base image and a subtle coefficient map. Finally, a RefineNet is added to further refine high-frequency details and suppress noise and color deviation. Extensive experiments are evaluated to demonstrate the superiority of the proposed method over several state-of-the-art methods.

Video frame interpolation (VFI) aims at synthesizing new video frames in-between existing frames to generate smoother high frame rate videos. Current methods usually use the fixed pre-trained networks to generate interpolated-frames for different resolutions and scenes. However, the fixed pre-trained networks are difficult to be tailored for a variety of cases. Inspired by classical pyramid energy minimization optical flow algorithms, this paper proposes a recurrent residual pyramid network (RRPN) for video frame interpolation. In the proposed network, different pyramid levels share the same weights and base-network, named recurrent residual layer (RRL). In RRL, residual displacements between warped images are detected to gradually refine optical flows rather than directly predict the flows or frames. Owing to the flexible recurrent residual pyramid architecture, we can customize the number of pyramid levels, and make trade-offs between calculations and quality based on the application scenarios. Moreover, occlusion masks are also generated in this recurrent residual way to solve occlusion better. Finally, a refinement network is added to enhance the details for final output with contextual and edge information. Experimental results demonstrate that the RRPN is more flexible and efficient than current VFI networks but has fewer parameters. In particular, the RRPN, which avoid over-reliance on datasets and network structures, shows superior performance for large motion cases.

This paper proposes a stable method of generating a stereoscopic panoramic video with the omnidirectional stereo (ODS) format. Different from the traditional image stitching method which can only be applied to generate a monocular panorama, we adopt an optical flow-based blending method to create two panoramas for binocular vision. In addition, traditional image stitching methods based on seam-finding tend to cause the problem of temporal flicker. We address this problem by restricting the optical flow field of the new frame with its previous frame's optical flow field. Thus, the generated video is stable and out of temporal flicker. There are four key operations in our approach. First, we adopt the ODS format which is the basis of stereoscopic panorama. Second, we do effective exposure compensation, making the brightness of the two eyes' panoramas consistent. Third, we employ an optical flow-based blending method to synthesis the final panorama effectively. Fourth, we take the previous frame's optical flow field as the restriction of the present frame's optical flow field to acquire a stable video. The final output videos can deliver a pleasant and impressive stereoscopic viewing experience to the audience when the audience watches the videos in the virtual reality headset.

A new methodology to measure coded image/video quality using the just-noticeable-difference (JND) idea was proposed in Lin et al. (2015). Several small JND-based image/video quality datasets were released by the Media Communications Lab at the University of Southern California in Jin et al. (2016) and Wang et al. (2016) [3]. In this work, we present an effort to build a large-scale JND-based coded video quality dataset. The dataset consists of 220 5-s sequences in four resolutions (i.e., 1920 × 1080 , 1280 × 720 , 960 × 540 and 640 × 360 ). For each of the 880 video clips, we encode it using the H.264/AVC codec with QP = 1 , … , 51 and measure the first three JND points with 30 + subjects. The dataset is called the “VideoSet”, which is an acronym for “Video Subject Evaluation Test (SET)”. This work describes the subjective test procedure, detection and removal of outlying measured data, and the properties of collected JND data. Finally, the significance and implications of the VideoSet to future video coding research and standardization efforts are pointed out. All source/coded video clips as well as measured JND data included in the VideoSet are available to the public in the IEEE DataPort (Wang et al., 2016 [4]).

Cost computation, cost aggregation, disparity optimization, and disparity refinement are the four main steps for stereo matching. While the first three steps have been widely investigated, few efforts have been taken on disparity refinement. In this paper, we propose a color image-guided disparity refinement method to further remove the boundary-inconsistent regions on disparity map. First, the origins of boundary-inconsistent regions are analyzed. Then, these regions are detected with the proposed hybrid-superpixel-based strategy. Finally, the detected boundary-inconsistent regions are refined by a modified weighted median filtering method. Experimental results on various stereo matching conditions validate the effectiveness of the proposed method. Furthermore, depth maps obtained by active depth acquisition devices like Kinect can also be well refined with our proposed method.

With the development of ultra-high-resolution display devices, the visual perception of fine texture details is becoming increasingly important. Traditional image upsampling methods suffer from either loss of high-frequency texture details or very high time cost. In this paper, we propose an iterative projection reconstruction (IPR) method for fast and efficient image upsampling. The proposed method refines high-frequency texture details with an iterative projection process, and utilizes the pre-computed projection matrix to accelerate the example-based image reconstruction. As a result, the proposed method can reproduce fine texture details with low time cost. Experimental results demonstrate that the proposed method outperforms some state-of-the-art methods.

Tone mapping operators can convert high dynamic range (HDR) images to low dynamic range (LDR) images so that we can enjoy the informative contents of HDR images with LDR devices. However, current state-of-the-art tone mapping algorithms mainly focus on the luminance mapping while neglecting the color component. Meanwhile, they often suffer from halo artifacts and over-enhancement. In this paper, we propose a tone mapping network (TMNet) in Hue-Saturation-Value (HSV) color space to obtain better luminance and color mapping. We adopt the improved Wasserstein generative adversarial network (WGAN-GP) as the basic architecture and further introduce several improvements. A meticulously designed loss function is adopted to push tone mapped image to the natural image manifold. What’s more, we create a tone mapped image dataset in which the label images are manually adjusted by photographers. Compared with some state-of-the-art tone mapping methods, the proposed method can achieve better performance in both subjective and objective evaluations.

One of the most challenging problems of novel view synthesis is to predict the scene in complex geometric environments. Existing methods depend on either homography optimization or deep neural networks. In this paper, we provide a framework of view synthesis, which includes grid-based warp, depth refinement and ghost artifacts removal. The depth refinement method is our main contribution, which can be combined with any other warp operation to generate high quality images. To achieve it, the depth refinement method is combined with a shape-preserving warp operation together based on reliable, half-reliable and unreliable superpixel discrimination. We remove outliers in half-reliable superpixels by considering their neighboring reliable superpixels and distinguish half-reliable ones into reliable and unreliable parts. This step helps us to get more accurate depth information. Experimental results show that our view synthesis system has nearly 0.7dB gains in PSNR and 0.03 gains in SSIM compared with the state-of-the-art view synthesis algorithm.

A blur map estimation method using edge information is proposed.The blur map is segmented into multiple superpixels according to the image contours.Ringing artifacts and noise are detected and removed after deconvolution. In this paper, we propose a single image deblurring algorithm to remove spatially variant defocus blur based on the estimated blur map. Firstly, we estimate the blur map from a single image by utilizing the edge information and K nearest neighbors (KNN) matting interpolation. Secondly, the local kernels are derived by segmenting the blur map according to the blur amount of local regions and image contours. Thirdly, we adopt a BM3D-based non-blind deconvolution algorithm to restore the latent image. Finally, ringing artifacts and noise are detected and removed, to obtain a high quality in-focus image. Experimental results on real defocus blurred images demonstrate that our proposed algorithm outperforms some state-of-the-art approaches.

The auto-stereoscopic 3D display technology is the most widely used 3D display technology nowadays, but its problem of vergence-accommodation conflict (VAC) remains to be solved. This paper proposes a novel method to solve this problem via monocular multi-view, aiming to alleviate the discomfort caused by VAC. By utilizing the light splitting of the parallax barrier and a specific rendering algorithm, we generate multiple images from different angles in a viewing zone. In this way, each eye can receive more than two images, thereby easing VAC through the automatic blurring adjustment of human eyes.

3D reconstruction using Struct-from-Motion(SFM) and Multi-View Stereo(MVS), along with image-based rendering (IBR), has been applied to the synthesis of the virtual-viewpoint images. We have investigated the latest works, and we find that the underlying 3D reconstruction methods are quite efficient for buildings and regular structure. However, for scenes with weakly textured or discontinuous-depth regions, these methods do not work well. Such regions do not contain reliable or dense 3D information, and as a result, biases will occur in the procedure of synthesizing point clouds. In this paper, we propose a new method using the image pyramid in the depth-map merging based MVS framework. In this method, we use multi-scale image information to estimate depth. Our method can capture more 3D information in weakly textured or discontinuous-depth regions, and finally obtain the better visible effect in the virtual-viewpoint images.

In this paper, we present a parallax-robust hexahedral panoramic video stitching method. An efficient three-stage stitching procedure is proposed. In the preprocessing stage, layered feature points matching strategy extracts feature matches lying in different depth layers. In the rough alignment stage, based on the first layer of feature matches, global projective warping estimates and refines camera parameters by considering the constraints among all cameras to avoid accumulated errors. With camera parameters, image pixels are roughly mapped onto a spherical surface. In the refined alignment stage, based on multiple layers of feature matches, layered content-preserving warping further aligns abundant feature pairs exploited from multiple depth layers, so as to alleviate ghosting caused by large parallax. Experimental results show that our method can effectively stitch panoramic video without noticeable parallax errors.

Convolutional neural network (CNN) based super-resolution (SR) has achieved superior performance compared with traditional methods for uncompressed images/videos, but its performance degenerates dramatically for compressed content especially at low bit-rate scenario due to the mixture distortions during sampling and compressing. This is critical because images/videos are always compressed with degraded quality in practical scenarios. In this paper, we propose a novel dual-network structure to improve the CNN-based SR performance for compressed high definition video especially at low bit-rate. To alleviate the impact of compression, an enhancement network is proposed to remove the compression artifacts which is located ahead of the SR network. The two networks, enhancement network and SR network, are optimized stepwise for different tasks of compression artifact reduction and SR respectively. Moreover, an improved geometric self-ensemble strategy is proposed to further improve the SR performance. Extensive experimental results demonstrate that the dual-network scheme can significantly improve the quality of super-resolved images/videos compared with those reconstructed from single SR network for compressed content. It achieves around 31.5% bit-rate saving for 4 K video compression compared with HEVC when applying the proposed method in a SR-based video coding framework, which proves the potential of our method in practical scenarios, e.g., video coding and SR.

The 3D panoramic video provides an immersive stereo visual experience by presenting 3D omnidirectional videos of real-world scenes. The key challenge is to develop an efficient representation of 3D panoramic video in order to maximize coding efficiency. In this paper, we propose an asymmetric representation based on the binocular suppression theory that in a stereo sequence, where the sharpness of left-eye and right-eye view differ, the perceived binocular quality of a stereoscopic sequence is rated close to the sharper view. According to the theory, in our method, one view is down-sampled to half size of original sequence in horizontal and vertical direction, and the other keeps the original resolution. To improve the quality of the reconstructed 3D panoramic video, we propose an anti-aliasing method based on detail blurring. Experimental results show that our representation can get significant coding gains (26.13%) over side-by-side and top-and-bottom method with full resolution.

Camera pose tracking is one of the major problems of Augmented Reality (AR) system. While tracking technologies based on fiducial markers have dominated the development of AR applications for almost a decade, various real-time approaches to Natural-Image-Marker tracking have recently been presented. However, most existing approaches do not yet achieve sufficient frame rates for AR on mobile phones or at least require an extensive training phase in advance. In this paper, we propose a novel method for fast camera pose tracking using visual complementary filtering. Compared with existing methods, our method improves the accuracy at the same time guarantee the running speed. Our method can be used in Augmented Reality System with a stored natural image. The implementation is very fast and robust, allowing running even on mobile phones at highly interactive rates and accuracy. Our implementation can immediately track features captured from a photo without any training.

With the development of ultra-high-resolution display devices, the visual perception of fine texture details is becoming more and more important. A method of high-quality image upsampling with a low cost is greatly needed. In this paper, we propose a fast and efficient image upsampling method that makes use of high-resolution local structure constraints. The average local difference is used to divide a bicubic-interpolated image into a sharp edge area and a texture area, and these two areas are reconstructed separately with specific constraints. For reconstruction of the sharp edge area, a high-resolution gradient map is estimated as an extra constraint for the recovery of sharp and natural edges; for the reconstruction of the texture area, a high-resolution local texture structure map is estimated as an extra constraint to recover fine texture details. These two reconstructed areas are then combined to obtain the final high-resolution image. The experimental results demonstrated that the proposed method recovered finer pixel-level texture details and obtained top-level objective performance with a low time cost compared with state-of-the-art methods.

Panoramic video provides an immersive experience by presenting a 360° spherical video content. Due to the limitations of coding and storage technology, the spherical panoramic video needs to be projected onto the two-dimensional plane for storage and encoding. In this paper, we propose a polar square projection scheme. We project the area near the poles of the sphere into two square planes and a latitude circle on sphere is projected to a square circle on squares plane, in addition, the rest of area on sphere is projected into a rectangle by means of equal area projection. Experimental results show our proposed projection can obtain a gain of 11.63% BD-rate compared to the equirectangular projection.

Recently,saliency detection is an active topic in the multimedia field. Several algorithms have been proposed in this field. However,for some complex situations which contain multiple objects or complex background,they are not robust and their performances are not satisfied. In this paper,we introduce an innovative framework to improve the current saliency detection algorithms,in which we propose a new algorithm based on depth cue and an adaptive fusion mechanism. The proposed framework shows a good performance and maintains the robustness in complex situations. Experiments' results show that the proposed framework is superior to other existing saliency approaches. Finally,we show a good application example by this framework to construct an image montage.

Low-light images are not conducive to human observation and computer vision algorithms due to their low visibility. To solve this problem, many image enhancement techniques have been proposed. However, existing techniques inevitably introduce color and lightness distortion when increasing visibility. To lower the distortion, we propose a novel enhancement method using the response characteristics of cameras. First, we investigate the relationship between two images with different exposures to obtain an accurate camera response model. Then we borrow the illumination estimation techniques to estimate the exposure ratio map. Finally, we use our camera response model to adjust each pixel to its desired exposure according to the estimated exposure ratio map. Experiments show that our method can obtain enhancement results with less color and lightness distortion compared to several state-of-the-art methods.

Saliency detection aims to detect the most attractive objects in images, which has been widely used as a foundation for various multimedia applications. In this paper, we propose a novel salient object detection algorithm for RGB-D images using center-dark channel prior. First, we generate an initial saliency map based on a color saliency map and a depth saliency map of a given RGB-D image. Then, we generate a center-dark channel map based on a center saliency prior and a dark channel prior. Finally, we fuse the initial saliency map with the center dark channel map to generate the final saliency map. The proposed algorithm is evaluated on two public RGB-D datasets, and the experimental results show that our method outperforms the state-of-the-art methods.

To provide full panoramic video content, the transmission bandwidth and playing complexity of virtual reality (VR) video are much higher than that of traditional video. Dynamic stream switching is an efficient scheme which stores multiple asymmetric projections of panoramic video in server side, and transmits only one asymmetric projection (with much lower resolution than full panoramic video) according to the direction of user's head. In this paper, we propose an asymmetric circular (ASC) projection method to generate the asymmetric projection of panoramic video. ASC uses equal area projection in the main viewpoint area to ensure the high and uniform quality of this area, and a projection with decreasing sampling density in other areas. Experimental results show that the bit rate and playing complexity of VR video are greatly reduced with the proposed ASC method. ASC is superior to state-of-the-art method and has been adopted in the standard of IEEE 1857.9-Immersive Visual Content Coding.

Detecting salient objects with complex backgrounds is still a challenging problem. Under the background having similar colors with complex patterns of salient objects, existing methods' performance is not satisfied, especially for multiple salient objects detection. In this paper, we propose a framework based on cognitive neuroscience to tackle with these challenges. According to cognitive neuroscience, human visual system is sensitive to depth of field, conspicuous color, moving objects and central object of scene. In the proposed framework, we imitate these human visual characteristics with following approaches: (1) using depth to represent the depth of field in the real world, (2) using luminance which imitates the light changing to represent the relative motions among objects, (3) using the center-bias to enhance object around the center. Experimental results on two challenging RGB-D datasets demonstrate that our method is superior to the existing methods in terms of effectiveness.

Saliency detection is an active topic in multimedia field. Several algorithms have been proposed in this field. Most previous works on saliency detection focus on 2D images. However, for some complex situations which contain multiple objects or complex background, they are not robust and their performances are not satisfied. Recently, 3D visual information supplies a powerful cue for saliency detection. In this paper, we propose a multilayer backpropagation saliency detection algorithm based on depth mining by which we exploit depth cue from four different layers of images. The evaluation of the proposed algorithm on two challenging datasets shows that our algorithm outperforms state-of-the-art.

In this paper, an innovated method is proposed for video denoising. The method consists of two major procedures. First, a new adaptive superpixel video texture metric is proposed. This video texture metric is calculated, which relates to different parts of a video stream. Then a new adaptive structure variance is estimated by adopting fine and coarse structures. Finally, a noise filter based on the estimated weights of different structures in a video stream is applied. By comparison, the proposed method outperforms traditional state-of-art methods, especially in block artifacts reduction.

Road detection is one of the most important research areas in driver assistance and automated driving field. However, the performance of existing methods is still unsatisfactory, especially in severe shadow conditions. To overcome those difficulties, first we propose a novel shadow-free feature extractor based on the color distribution of road surface pixels. Then we present a road detection framework based on the extractor, whose performance is more accurate and robust than that of existing extractors. Also, the proposed framework has much low-complexity, which is suitable for usage in practical systems.

Object retrieval in the large-scale image corpus is an appealing, yet challenging task. Most of existing frameworks are based on bag-of-visual-words (BoVW) model. However, BoVW has an obvious drawback, i.e. lack of spatial information. In this paper, we propose weighted transformable spatial pyramid and scalable query for object retrieval. We first break the whole image into sub-images and then make these sub-images up in a new order in the final representation. Our method has two contributions: 1) relative spatial relationships of local features instead of absolute geometric layouts of features are encoded so that translation invariance is guaranteed, 2) scaling invariance in the image representation is ensured by scalable query. The experimental results show that our approach outperforms BoVW and traditional spatial pyramid matching.

In this paper, a novel united low-light image enhancement framework for both contrast enhancement and denoising is proposed. First, the low-light image is segmented into superpixels, and the ratio between the local standard deviation and the local gradients is utilized to estimate the noise-texture level of each superpixel. Then the image is inverted to be processed in the following steps. Based on the noise-texture level, a smooth base layer is adaptively extracted by the BM3D filter, and another detail layer is extracted by the first order differential of the inverted image and smoothed with the structural filter. These two layers are adaptively combined to get a noise-free and detail-preserved image. At last, an adaptive enhancement parameter is adopt into the dark channel prior dehazing process to enlarge contrast and prevent over/under enhancement. Experimental results demonstrate that our proposed method outperforms traditional methods in both subjective and objective assessments.

In this paper, we propose a robust method to remove motion blur from a single photograph. We find that an inaccurate kernel and an unreliable final latent image reconstruction method are two main factors leading to low-quality restored images. To improve image quality, we do the following technical contributions. For robust blur kernel estimation, first, an edge mask and a smooth constraint are used to provide reliable intermediate latent images for salient structure extraction; second, we adopt an effective salient structure selection method to remove detrimental edges for kernel estimation; third, we use a gradient sparsity prior to remove kernel noise and ensure the continuity of blur kernels. For final latent image reconstruction, we combine the merits of both the TV-l 2 model and the hyper-Laplacian model to preserve tiny details and eliminate noise. Experimental results on synthetically blurred images and real photographs demonstrate that the proposed algorithm performs better than state-of-the-art approaches.

This paper presents a compact shot representation for video semantic indexing (SIN). The proposed representation consists of visual cues from only two frames, i.e., key frame (KF) and difference frame (DF), which are both constructed with spatial pyramid. The KF describes static information while the generated DF captures non-static information. Each region of DF is derived from the same location in a selected frame, which has the most salient difference compared with the key frame in that region. We introduce a variation of DF to further enhance our model. Experimental results on TRECVID SIN demonstrate that our method obtains better accuracy than the state-of-the-art, while requiring less storage space and consuming time.

This paper presents a fast integer-pixel motion estimation (ME) algorithm for High Efficiency Video Coding (HEVC), which provides a strategy to speed up the search process significantly, while yielding the same quality performance as the Test Zone Search (TZSearch) scheme in HEVC Test Model (HM). Compared with the H.264/AVC, the ME process employs a more complex hybrid coding architecture and a larger size search window in HEVC, leading to great computational complexity. We utilize limited pixels of certain position in the current search range to build a quadratic model, and then shrink the search range repeatedly by analyzing the sum of absolute difference (SAD) distribution until the best motion vector (MV) is obtained. The proposed algorithm can be applied to various encoding conditions. Experimental results show that our method can save 56% of computations compared with the TZSearch scheme, with negligible decrease of coding quality.

In recent years, the local feature descriptor called SHOT (Signature of Histograms of OrienTations) has achieved excellent performance in 3D vision. However, there still exists room for improvement: on the one hand, the convexity and concavity information is still ambiguous; on the other hand, it is not consistent and unique enough for constructing histograms with just angle information between normals. In this paper, we propose a new robust descriptor called Signature of Unique Angles Histograms (SUAH) with consideration of the unique angles, which are used to construct histograms, between local reference frames. And the convexity sign is embedded to disambiguate the convexity and concavity. In addition, power normalization is used to address the point density problem. A series of experiments are conducted to show that the performance of our method is better than state-of-the-art methods on synthesized dataset and real scenes dataset.

High Efficient Video Coding (HEVC) is the latest coding standard with superior compression efficiency while its encoding complexity is much higher compared with H.264/AVC. Motion estimation is one of the most time-consuming parts in video coding. In the reference software of HEVC, TZ (Test Zone) search method is adopted as the fast motion estimation method. However, its complexity is still high. There are many other fast motion estimation methods, for example, the hexagon search method, but their performance loss is larger than TZ search. In order to balance coding speed and performance, a new context-adaptive fast motion estimation algorithm is proposed in this paper. In this the proposed, motion intensity is defined in block-level, motion vectors and motion vector differences of neighbor blocks are utilized to measure the motion intensity. When motion intensity is large, TZ search method is used; otherwise, hexagon search method is used. Experimental results show that the proposed method can save 39% ∼ 60% of motion estimation time with average 0.5% of BD-rate loss.

Stereo matching is a fundamental topic in computer vision. Usually, stereo matching is mainly composed of four stages: cost computation, cost aggregation, disparity optimization and disparity refinement. In this paper, we propose a novel stereo matching method with space-constrained cost aggregation and segmentation-based disparity refinement. Stateof- the-art methods are used for cost aggregation and disparity optimization stages. Three technical contributions are given in this paper. First, applying space-constrained cross-region in cost aggregation stage; second, utilizing both color and disparity information in image segmentation; third, using image segmentation and occlusion region detection to aid disparity refinement. The performance of our platform ranks second in the Middlebury evaluation.

In this paper, an innovative method of HEVC video pre-processing is proposed. The method applies a simple linear iterative clustering (SLIC), which adapts a k-means clustering to group pixels into perceptually meaningful atomic regions of superpixels. By calculating the average of weighted average of luminance differences around each pixel in the superpixel, a suitable parameter of Gaussian filter for the superpixel is determined. Experimental results show that bit rate can be reduced up to 29% without loss in visual quality.

Accuracy of prediction block (PB) plays a very important role in improving the coding performance. In this paper, we propose tagged multi-hypothesis motion compensation scheme (TMHMC) for inter frames to improve the accuracy of PB. TMHMC not only makes use of temporal correlation between frames but also the spatial correlation as motion vectors of adjacent blocks are used to derive the PB. For entropy coding process, only one motion vector and a tag indicating which adjacent block is used are coded in bit-stream. Adding TMHMC scheme as an additional mode in MPEG internet video coding (TVC) platform, the bitrate saving is up to 12% at the same objective quality compared with anchor. Average bitrate saving is close to 6% over all test sequences. In addition, we also implement the conventional multi-hypothesis motion compensation (MHMC) scheme. 3% bitrate is further saved on average by TMHMC compared with conventional MHMC.

In this paper, a new frame interpolation method with pixel-level motion vector field (MVF) is proposed. Given that existing methods cannot handle occlusions and blocking artifacts well, there are three contributions in our method: (i) applying the pixel-level motion vectors (MVs) estimated by optical flow algorithm to eliminate blocking artifacts (ii) motion post-processing to keep spatial consistency (iii) robust warping method to address collisions and holes caused by occlusions. The method could remove blocking artifacts and alleviate the artifacts caused by occlusions. Experimental results show that the proposed method outperforms existing methods both in terms of objective and subjective performances, especially for sequences with complex motions.

Recent cost-volume filtering-based local stereo methods have achieved comparable accuracy with global methods. However, there are still some significant outliers existing in the final disparity map. In this paper, we propose a cost-volume filtering-based local stereo matching method that employs a new combined cost and a novel secondary disparity refinement mechanism. The combined cost is formulated by a modified color census transform, truncated absolute differences of color and gradients. Symmetric guided filter is used for the cost aggregation. Different from traditional stereo matching, a novel secondary disparity refinement is proposed to further remove remaining outliers. Experimental results on Mid-dlebury benchmark show that our method ranks the 5 th out of the 144 submitted methods, and is the best cost-volume filtering-based local method. Furthermore, experiments on real world sequences also validate the effectiveness of our proposed method.

Video stabilization enhances video quality by stabilizing unstable motion. This paper proposes a new video stabilization method that simultaneously factors and smooths motion trajectories. We model the trajectories with a time-variant local subspace constraint. Every column of the trajectory matrix is factored and smoothed in separate local subspace. This model makes our method more flexible and accurate than subspace video stabilization. In addition, we design a novel outlier detection technique that utilizes the relationship between consecutive local subspaces. Experiments on synthetic data validate the numerical performance of our factorization. Quantitative comparisons on real videos show that our local method is better than subspace video stabilization. Moreover, our stabilized videos are comparable with the public results from some other state-of-the-art methods.

Compared with H.264, the High Efficiency Video Coding standard has higher compression efficiency and better transmission performance, and will be used in wider field. TZ search method is adopted to perform integer pixel search in HEVC. Its performance is very perfect but very time-consuming. In this paper, we use hexagon search method in place of TZ search method to speed up motion estimation process. Simulation results show that this scheme accelerates 2-6 times and the performance loss is negligible in sequences with slow movements.

The fraction-pel interpolation filter varies in the video coding standards such as H.264/AVC, AVS and HEVC. Since fractional-pel motion compensation plays an important role in the video encoder, the interpolation of fractional-pel pixels can be refined and designed better to enhance the coding efficiency. In this paper, we firstly propose the generation algorithm of interpolation filter coefficients, and four different tap filters, namely 4tap, 6 tap, 8 tap and 10tap, are tested. A resolution-adaptive interpolation filter for different resolution videos is then introduced based on this algorithm to achieve the maximum bitrate saving. In the proposed scheme, 4 tap filter is applied for the UHD (2560×1600 and above) videos, 6 tap filter and 10 tap filter are performed in the videos whose resolution ranging from 720P (1280×720) to 1080P (1920×1080) and the videos with the resolution below 720P, respectively. When 4 tap filter and 6 tap filter are used in high-definition video, the coding efficiency can increase and the computational complexity will reduce greatly, which is actually beneficial to make hardware optimization more effectively especially SIMD (Single Instruction Multiple Data) and VLSI design. Experiments show that the average BD-rate gains on luma Y, chroma U and V are 1.4%, 0.7% and 0.7% for LP-Main configuration, when conducted in HEVC reference software HM11.0. The coding efficiency gains are significant for some video sequences and can reach up to 6.1%.

This paper presents a new method for calculating the low-rank approximation of a highly incomplete trajectory matrix for subspace video stabilization. We extend moving factorization proposed in [1], which is a streamable method based on least squares. By utilizing sparse representation of trajectories, the proposed factorization method is more accurate while still streamable. We test our sparse moving factorization on synthetic data as well as real videos. Experiments on synthetic sequence demonstrate the numerical properties of our method, and stabilized videos show that our method outperforms moving factorization for subspace video stabilization. In addition, our results are also better than the ones from some other state-of-the-art video stabilization methods.